2024-05-27T00:00:43.113 INFO:root:teuthology version: 0.0.1.dev291+ge7ae758 2024-05-27T00:00:43.114 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174 --verbose --owner scheduled_teuthology@teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --name teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174/orig.config.yaml 2024-05-27T00:00:43.165 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T00:00:43.252 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174 branch: squid description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7727174' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: cephfs: session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - Replacing daemon mds sha1: 0ef46a246e1118e5adafa1857b4189e7d780c816 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0ef46a246e1118e5adafa1857b4189e7d780c816 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 7d7cb32d1d130135a98121a564d709025c2154b6 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 1555 sha1: 0ef46a246e1118e5adafa1857b4189e7d780c816 sleep_before_teardown: 0 subset: 10/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_7d7cb32d1d130135a98121a564d709025c2154b6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7d7cb32d1d130135a98121a564d709025c2154b6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9za9Kcz6urGNPKFQx9cMZxEB85f2qJjQnWlSb2kKHd8vMx5iWcIKGPxR5jZS/OJovet3O2cVUb4NJCurn5HGA= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJK5fupUbTcvOxwVRSb9OBYO5yahwCCHNCt1CPD2Z40ka7gsBnHd+cfgmxqx6HDJiC7pxd1KDBLKvVozEtBX8K0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e7ae7580c417d6ab27fda98fecadca5d9f85d63e timestamp: 2024-05-26_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-27T00:00:43.252 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7d7cb32d1d130135a98121a564d709025c2154b6/qa; will attempt to use it 2024-05-27T00:00:43.253 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7d7cb32d1d130135a98121a564d709025c2154b6/qa/tasks 2024-05-27T00:00:43.253 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-27T00:00:43.255 INFO:teuthology.task.internal:Checking packages... 2024-05-27T00:00:43.276 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '0ef46a246e1118e5adafa1857b4189e7d780c816' 2024-05-27T00:00:43.276 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-27T00:00:43.276 INFO:teuthology.packaging:ref: None 2024-05-27T00:00:43.276 INFO:teuthology.packaging:tag: None 2024-05-27T00:00:43.276 INFO:teuthology.packaging:branch: squid 2024-05-27T00:00:43.276 INFO:teuthology.packaging:sha1: 0ef46a246e1118e5adafa1857b4189e7d780c816 2024-05-27T00:00:43.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-05-27T00:00:43.561 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2859-g7d7cb32d-1jammy 2024-05-27T00:00:43.562 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-27T00:00:43.576 INFO:teuthology.task.internal:no buildpackages task found 2024-05-27T00:00:43.576 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-27T00:00:43.582 INFO:teuthology.task.internal:Saving configuration 2024-05-27T00:00:43.600 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-27T00:00:43.608 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-27T00:00:43.632 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-26 23:54:53.365849', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINEt5GimRhv7Ntr6MHQ84Gf9dgDA5Gwb/A3jJ/T1Zrpl'} 2024-05-27T00:00:43.653 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-26 23:54:53.367367', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOo+dPCK1mKU18QTcpBJca9Yh0LrLSStZhVESWhoJBH2'} 2024-05-27T00:00:43.653 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-27T00:00:43.667 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-05-27T00:00:43.668 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-05-27T00:00:43.668 INFO:teuthology.run_tasks:Running task console_log... 2024-05-27T00:00:43.735 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f070c6011b0>, signals=[15]) 2024-05-27T00:00:43.735 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-27T00:00:43.748 INFO:teuthology.task.internal:Opening connections... 2024-05-27T00:00:43.748 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:00:43.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T00:00:43.835 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:00:43.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T00:00:43.915 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-27T00:00:43.945 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-05-27T00:00:43.950 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-05-27T00:00:43.951 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-05-27T00:00:43.997 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-27T00:00:43.997 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="Ubuntu" 2024-05-27T00:00:43.997 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="22.04" 2024-05-27T00:00:43.997 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-27T00:00:43.997 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_CODENAME=jammy 2024-05-27T00:00:43.997 INFO:teuthology.orchestra.run.smithi098.stdout:ID=ubuntu 2024-05-27T00:00:43.997 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE=debian 2024-05-27T00:00:43.998 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-27T00:00:43.998 INFO:teuthology.orchestra.run.smithi098.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-27T00:00:43.998 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-27T00:00:43.998 INFO:teuthology.orchestra.run.smithi098.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-27T00:00:43.998 INFO:teuthology.orchestra.run.smithi098.stdout:UBUNTU_CODENAME=jammy 2024-05-27T00:00:43.999 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-05-27T00:00:44.031 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-05-27T00:00:44.036 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-05-27T00:00:44.036 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-05-27T00:00:44.085 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-27T00:00:44.085 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu" 2024-05-27T00:00:44.085 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="22.04" 2024-05-27T00:00:44.085 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-27T00:00:44.085 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=jammy 2024-05-27T00:00:44.085 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu 2024-05-27T00:00:44.085 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian 2024-05-27T00:00:44.086 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-27T00:00:44.086 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-27T00:00:44.086 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-27T00:00:44.086 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-27T00:00:44.086 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=jammy 2024-05-27T00:00:44.086 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-05-27T00:00:44.106 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-27T00:00:44.126 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-27T00:00:44.135 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-27T00:00:44.135 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-05-27T00:00:44.137 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-05-27T00:00:44.141 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-27T00:00:44.149 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-27T00:00:44.149 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-05-27T00:00:44.186 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-05-27T00:00:44.195 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-27T00:00:44.285 INFO:teuthology.run_tasks:Running task kernel... 2024-05-27T00:00:44.305 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-05-27T00:00:44.306 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-05-27T00:00:44.306 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-05-27T00:00:44.306 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T00:00:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:00:44.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2024-05-27T00:00:44.500 DEBUG:teuthology.task.kernel:sha1 for client.0 is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T00:00:44.500 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-27T00:00:44.500 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-05-27T00:00:44.505 INFO:teuthology.orchestra.run.smithi098.stdout:5.15.0-56-generic 2024-05-27T00:00:44.506 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T00:00:44.506 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-27T00:00:44.506 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2024-05-27T00:00:44.507 DEBUG:teuthology.orchestra.run.smithi098:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-05-27T00:00:44.567 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-05-27T00:00:44.705 INFO:teuthology.orchestra.run.smithi098.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-05-27T00:00:44.705 INFO:teuthology.orchestra.run.smithi098.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-05-27T00:00:44.705 INFO:teuthology.orchestra.run.smithi098.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-05-27T00:00:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:hint: 2024-05-27T00:00:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:hint: git config --global init.defaultBranch 2024-05-27T00:00:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:hint: 2024-05-27T00:00:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-05-27T00:00:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-05-27T00:00:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:hint: 2024-05-27T00:00:44.706 INFO:teuthology.orchestra.run.smithi098.stderr:hint: git branch -m 2024-05-27T00:00:44.707 INFO:teuthology.orchestra.run.smithi098.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-05-27T00:00:44.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-05-27T00:00:44.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-05-27T00:00:44.846 DEBUG:teuthology.orchestra.run.smithi098:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-05-27T00:01:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-05-27T00:01:12.383 INFO:teuthology.orchestra.run.smithi098.stderr: * [new branch] main -> origin/main 2024-05-27T00:01:12.418 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20190312 -> 20190312 2024-05-27T00:01:12.418 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20190416 -> 20190416 2024-05-27T00:01:12.418 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20190514 -> 20190514 2024-05-27T00:01:12.418 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20190618 -> 20190618 2024-05-27T00:01:12.418 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20190717 -> 20190717 2024-05-27T00:01:12.420 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20190815 -> 20190815 2024-05-27T00:01:12.420 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20190923 -> 20190923 2024-05-27T00:01:12.420 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20191022 -> 20191022 2024-05-27T00:01:12.420 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20191215 -> 20191215 2024-05-27T00:01:12.420 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20200122 -> 20200122 2024-05-27T00:01:12.420 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20200316 -> 20200316 2024-05-27T00:01:12.420 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20200421 -> 20200421 2024-05-27T00:01:12.420 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20200519 -> 20200519 2024-05-27T00:01:12.420 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20200619 -> 20200619 2024-05-27T00:01:12.423 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20200721 -> 20200721 2024-05-27T00:01:12.423 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20200817 -> 20200817 2024-05-27T00:01:12.423 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20200918 -> 20200918 2024-05-27T00:01:12.423 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20201022 -> 20201022 2024-05-27T00:01:12.423 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20201118 -> 20201118 2024-05-27T00:01:12.423 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20201218 -> 20201218 2024-05-27T00:01:12.423 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20210208 -> 20210208 2024-05-27T00:01:12.423 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20210315 -> 20210315 2024-05-27T00:01:12.423 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20210511 -> 20210511 2024-05-27T00:01:12.423 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20210716 -> 20210716 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20210818 -> 20210818 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20210919 -> 20210919 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20211027 -> 20211027 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20211216 -> 20211216 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20220209 -> 20220209 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20220310 -> 20220310 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20220411 -> 20220411 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20220509 -> 20220509 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20220610 -> 20220610 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20220708 -> 20220708 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20220815 -> 20220815 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20220913 -> 20220913 2024-05-27T00:01:12.424 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20221012 -> 20221012 2024-05-27T00:01:12.427 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20221109 -> 20221109 2024-05-27T00:01:12.427 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20221214 -> 20221214 2024-05-27T00:01:12.427 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20230117 -> 20230117 2024-05-27T00:01:12.427 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20230210 -> 20230210 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20230310 -> 20230310 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20230404 -> 20230404 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20230515 -> 20230515 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20230625 -> 20230625 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20230804 -> 20230804 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20230919 -> 20230919 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20231030 -> 20231030 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20231111 -> 20231111 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20231211 -> 20231211 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20240115 -> 20240115 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20240220 -> 20240220 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20240312 -> 20240312 2024-05-27T00:01:12.428 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20240410 -> 20240410 2024-05-27T00:01:12.429 INFO:teuthology.orchestra.run.smithi098.stderr: * [new tag] 20240513 -> 20240513 2024-05-27T00:01:16.724 INFO:teuthology.orchestra.run.smithi098.stderr:Updating files: 45% (1552/3421) Updating files: 46% (1574/3421) Updating files: 47% (1608/3421) Updating files: 48% (1643/3421) Updating files: 49% (1677/3421) Updating files: 50% (1711/3421) Updating files: 51% (1745/3421) Updating files: 52% (1779/3421) Updating files: 53% (1814/3421) Updating files: 54% (1848/3421) Updating files: 55% (1882/3421) Updating files: 56% (1916/3421) Updating files: 57% (1950/3421) Updating files: 58% (1985/3421) Updating files: 59% (2019/3421) Updating files: 59% (2043/3421) Updating files: 60% (2053/3421) Updating files: 61% (2087/3421) Updating files: 62% (2122/3421) Updating files: 63% (2156/3421) Updating files: 64% (2190/3421) Updating files: 65% (2224/3421) Updating files: 66% (2258/3421) Updating files: 67% (2293/3421) Updating files: 68% (2327/3421) Updating files: 69% (2361/3421) Updating files: 70% (2395/3421) Updating files: 70% (2402/3421) Updating files: 71% (2429/3421) Updating files: 72% (2464/3421) Updating files: 73% (2498/3421) Updating files: 74% (2532/3421) Updating files: 75% (2566/3421) Updating files: 76% (2600/3421) Updating files: 77% (2635/3421) Updating files: 78% (2669/3421) Updating files: 79% (2703/3421) Updating files: 80% (2737/3421) Updating files: 81% (2772/3421) Updating files: 82% (2806/3421) Updating files: 83% (2840/3421) Updating files: 83% (2847/3421) Updating files: 84% (2874/3421) Updating files: 85% (2908/3421) Updating files: 86% (2943/3421) Updating files: 87% (2977/3421) Updating files: 88% (3011/3421) Updating files: 89% (3045/3421) Updating files: 90% (3079/3421) Updating files: 91% (3114/3421) Updating files: 92% (3148/3421) Updating files: 93% (3182/3421) Updating files: 94% (3216/3421) Updating files: 95% (3250/3421) Updating files: 96% (3285/3421) Updating files: 97% (3319/3421) Updating files: 98% (3353/3421) Updating files: 99% (3387/3421) Updating files: 100% (3421/3421) Updating files: 100% (3421/3421), done. 2024-05-27T00:01:16.729 INFO:teuthology.orchestra.run.smithi098.stdout:HEAD is now at ec8627ef Merge branch 'amd-isp-staging' into 'main' 2024-05-27T00:01:16.751 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on client.0... 2024-05-27T00:01:16.751 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T00:01:17.004 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-gb9459a26b088/ 2024-05-27T00:01:17.004 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /tmp/linux-image.deb && echo linux-image-6.9.0-gb9459a26b088_6.9.0-gb9459a26b088-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://2.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-gb9459a26b088/ --input-file=- 2024-05-27T00:01:17.007 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi098.front.sepia.ceph.com to complete... 2024-05-27T00:01:18.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27 00:01:19 URL:https://2.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-gb9459a26b088/linux-image-6.9.0-gb9459a26b088_6.9.0-gb9459a26b088-1_amd64.deb [44988888/44988888] -> "/tmp/linux-image.deb" [1] 2024-05-27T00:01:18.710 INFO:teuthology.orchestra.run.smithi098.stderr:FINISHED --2024-05-27 00:01:19-- 2024-05-27T00:01:18.710 INFO:teuthology.orchestra.run.smithi098.stderr:Total wall clock time: 1.7s 2024-05-27T00:01:18.710 INFO:teuthology.orchestra.run.smithi098.stderr:Downloaded: 1 files, 43M in 1.6s (27.6 MB/s) 2024-05-27T00:01:18.714 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on client.0... 2024-05-27T00:01:18.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo dpkg -i /tmp/linux-image.deb 2024-05-27T00:01:18.960 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-image-6.9.0-gb9459a26b088. 2024-05-27T00:01:20.997 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-05-27T00:01:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-05-27T00:01:21.154 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-6.9.0-gb9459a26b088 (6.9.0-gb9459a26b088-1) ... 2024-05-27T00:01:25.185 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-6.9.0-gb9459a26b088 (6.9.0-gb9459a26b088-1) ... 2024-05-27T00:01:25.281 INFO:teuthology.orchestra.run.smithi098.stderr:update-initramfs: Generating /boot/initrd.img-6.9.0-gb9459a26b088 2024-05-27T00:01:47.947 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub' 2024-05-27T00:01:47.955 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-27T00:01:47.983 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-05-27T00:01:48.278 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T00:01:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-6.9.0-gb9459a26b088 2024-05-27T00:01:48.607 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-27T00:01:48.620 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-27T00:01:48.684 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-27T00:01:48.689 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-27T00:01:48.846 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-27T00:01:48.847 INFO:teuthology.orchestra.run.smithi098.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-27T00:01:48.847 INFO:teuthology.orchestra.run.smithi098.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-27T00:01:48.893 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-05-27T00:01:49.112 DEBUG:teuthology.orchestra.run.smithi098:> dpkg-deb -c /tmp/linux-image.deb 2024-05-27T00:01:49.163 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./ 2024-05-27T00:01:49.164 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./boot/ 2024-05-27T00:01:49.164 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7004139 2024-05-23 11:07 ./boot/System.map-6.9.0-gb9459a26b088 2024-05-27T00:01:49.181 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 172251 2024-05-23 11:07 ./boot/config-6.9.0-gb9459a26b088 2024-05-27T00:01:49.182 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13136896 2024-05-23 11:07 ./boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T00:01:49.201 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/postinst.d/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/postrm.d/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/preinst.d/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/prerm.d/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/ 2024-05-27T00:01:49.202 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ 2024-05-27T00:01:49.203 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 507497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/aesni-intel.ko 2024-05-27T00:01:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-27T00:01:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-27T00:01:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-27T00:01:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 97945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-27T00:01:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 109897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-27T00:01:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 155241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-27T00:01:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-27T00:01:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-27T00:01:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 87001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-27T00:01:49.205 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 54505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 48041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 62865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 53561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/ 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/ 2024-05-27T00:01:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/ 2024-05-27T00:01:49.207 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-27T00:01:49.207 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpuid.ko 2024-05-27T00:01:49.207 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/msr.ko 2024-05-27T00:01:49.207 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/ 2024-05-27T00:01:49.207 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 443289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-27T00:01:49.207 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1037081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-27T00:01:49.207 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 2850153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-27T00:01:49.220 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/af_alg.ko 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/algif_hash.ko 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/algif_skcipher.ko 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/asymmetric_keys/ 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/ 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-27T00:01:49.221 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-27T00:01:49.222 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cryptd.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 48105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_simd.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lrw.ko 2024-05-27T00:01:49.223 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 169057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-27T00:01:49.224 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xts.ko 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_extlog.ko 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/apei/ 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/apei/einj.ko 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-27T00:01:49.225 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 119017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 99177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 107729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-27T00:01:49.226 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-27T00:01:49.227 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 60249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_legacy.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-27T00:01:49.228 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_radisys.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rz1000.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-27T00:01:49.229 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 102113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 53345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ 2024-05-27T00:01:49.230 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-27T00:01:49.231 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/ 2024-05-27T00:01:49.231 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 163833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-27T00:01:49.231 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/ 2024-05-27T00:01:49.231 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/ 2024-05-27T00:01:49.231 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 140833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-27T00:01:49.231 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/ 2024-05-27T00:01:49.231 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1241585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-27T00:01:49.260 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 203281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/ 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 115257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 143425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/ 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 126609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 119865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 313209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/ 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 117217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/ 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 70369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ 2024-05-27T00:01:49.261 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/applicom.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/ 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 186913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 179441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-27T00:01:49.262 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/ 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 103385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/nvram.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/ 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-27T00:01:49.263 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/ 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/xillybus_class.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 62537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/xillybus_core.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/ 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-palmas.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 57281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-si5351.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-twl6040.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-wm831x.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/ 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-27T00:01:49.264 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 89209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 333593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/ 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/ 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/ 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 751433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/ 2024-05-27T00:01:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 90465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/ 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/ 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 146745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 96137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 51097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-27T00:01:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 64153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 68689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/ 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-gpio.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-max14577.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-max77693.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-max8997.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-palmas.ko 2024-05-27T00:01:49.267 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-rt8973a.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-sm5502.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/ 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/dmi-sysfs.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/efi/ 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/efi/efi-pstore.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/ 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-adp5520.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-amd8111.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-arizona.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-da9052.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-da9055.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-dln2.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 81065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-f7188x.ko 2024-05-27T00:01:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-generic.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-kempld.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-lp3943.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-max7300.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-max730x.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-max732x.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-pca953x.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-pcf857x.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-rdc321x.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-sch.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-sch311x.ko 2024-05-27T00:01:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-tps65912.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-twl4030.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-twl6040.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-vx855.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-wm831x.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-wm8350.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-wm8994.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/ 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-a4tech.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-apple.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-27T00:01:49.270 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-belkin.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-cherry.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-chicony.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 74385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-cp2112.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-cypress.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ezkey.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-generic.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-27T00:01:49.271 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kensington.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 54209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lg-g15.ko 2024-05-27T00:01:49.272 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 67961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 121361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 97929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-magicmouse.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-microsoft.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-monterey.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 58993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ntrig.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 91217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-27T00:01:49.273 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-27T00:01:49.274 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 62121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-27T00:01:49.275 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 81097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 184729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 319897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/ 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 153617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/usbhid.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/usbkbd.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/usbmouse.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 261929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-27T00:01:49.276 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/ 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 75057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 110657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 381625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 77145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 67001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 115049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-27T00:01:49.277 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 89017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 81465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 48169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 68857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 60185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-27T00:01:49.278 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 69001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 47969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/da9052-hwmon.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/da9055-hwmon.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 113129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc2103.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 78913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-27T00:01:49.279 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 108953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 56305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hih6130.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-27T00:01:49.280 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 156457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/jc42.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 57761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-27T00:01:49.281 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 86489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 76481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 131497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-27T00:01:49.282 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mc13783-adc.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-05-27T00:01:49.283 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 56497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 141345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 72729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 67769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 97769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-27T00:01:49.284 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 101225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-27T00:01:49.285 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 48705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 46465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-27T00:01:49.286 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 62729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 71161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 96569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 75121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 83417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 116769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 134433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/ 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/ 2024-05-27T00:01:49.287 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/ 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-05-27T00:01:49.288 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 64217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 76961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 52929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-27T00:01:49.289 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 57209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-27T00:01:49.290 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/ 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 329681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1119785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-27T00:01:49.291 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 83897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 408329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 120905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 347201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 84937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/ 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/ 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 590873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/ 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 514577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/ 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 911649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 375449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/ocrdma/ 2024-05-27T00:01:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 197097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/usnic/ 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 212497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 334353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 120649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 132577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 157929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ 2024-05-27T00:01:49.293 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 156665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-27T00:01:49.317 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/ 2024-05-27T00:01:49.317 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/evbug.ko 2024-05-27T00:01:49.317 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/ff-memless.ko 2024-05-27T00:01:49.317 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/ 2024-05-27T00:01:49.317 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-27T00:01:49.317 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-27T00:01:49.317 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-27T00:01:49.317 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/ 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/lkkbd.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/lm8323.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/lm8333.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-05-27T00:01:49.318 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/newtonkbd.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/qt1070.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/qt2160.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/stowaway.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/sunkbd.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/xtkbd.ko 2024-05-27T00:01:49.319 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/matrix-keymap.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/ 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 159993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 114193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/gpio_mouse.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 428121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/psmouse.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/ 2024-05-27T00:01:49.320 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 140921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/ 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/ct82c710.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/parkbd.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/pcips2.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/ps2mult.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serport.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/ 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-88pm860x.ko 2024-05-27T00:01:49.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-adp5520.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 77193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-bd2802.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-da903x.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-da9052.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-gpio.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3533.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm355x.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3642.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp8788.ko 2024-05-27T00:01:49.322 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-max8997.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-mc13783.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-menf21bmc.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-pca9532.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-pca955x.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-pca963x.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-tca6507.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-wm831x-status.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-wm8350.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-27T00:01:49.323 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/ 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/ 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 790665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 116817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bufio.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 58273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 199017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 134937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-27T00:01:49.324 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 67489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-era.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 60969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-mirror.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 97193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-region-hash.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 139793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-snapshot.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-27T00:01:49.325 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 234441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 66417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-zero.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/ 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 264273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 124705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 157673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 444649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/ 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/ 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 214833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 77977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 54777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-27T00:01:49.326 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 150753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 97209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 63865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/ 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/88pm800.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/88pm805.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/88pm80x.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/arizona-i2c.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 148545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/arizona.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/bcm590xx.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/cros_ec_dev.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/dln2.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/janz-cmodio.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/kempld-core.ko 2024-05-27T00:01:49.327 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lm3533-core.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lp3943.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 60489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/max8907.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mc13xxx-core.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/menf21bmc.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/pcf50633-adc.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/pcf50633-gpio.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/pcf50633.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/retu-mfd.ko 2024-05-27T00:01:49.328 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 99409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/si476x-core.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 58049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 60673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/tps65010.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/tps6507x.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 52345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wm8994.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 64257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ad525x_dpot.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/ 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 57497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-27T00:01:49.329 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 47681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 58625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/c2port/ 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 47441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/c2port/core.ko 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/ 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ds1682.ko 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/dummy-irq.ko 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/echo/ 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/echo/echo.ko 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/ 2024-05-27T00:01:49.330 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/genwqe/ 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 191617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hmc6352.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ibmasm/ 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 92289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ics932s401.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/ 2024-05-27T00:01:49.331 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 54857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/ 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 154257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 69793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 388977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/phantom.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ti-st/ 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 91129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ti-st/st_drv.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/ 2024-05-27T00:01:49.332 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 234849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/ 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 620849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/ 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/ 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 139745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/ 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 589537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/ 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 53745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-27T00:01:49.333 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 158641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 73857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 109513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/ 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 64697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/ 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 84321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/ 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 70729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-27T00:01:49.334 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/ 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 52681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/ 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 56049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 115873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/ 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/ 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 124481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 128393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 110321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-27T00:01:49.335 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/ 2024-05-27T00:01:49.336 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 101849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-27T00:01:49.336 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 88945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-27T00:01:49.336 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/ 2024-05-27T00:01:49.336 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 105625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-27T00:01:49.336 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 228977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-27T00:01:49.336 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-05-27T00:01:49.336 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1733297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-27T00:01:49.336 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 167177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-27T00:01:49.336 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/ 2024-05-27T00:01:49.336 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 164249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 440553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/ 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/ 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 432145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/ 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 257145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 446057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1311193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-05-27T00:01:49.337 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 206673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/ 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/ 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 269249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/ 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/ 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 77449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 169041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 76201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-27T00:01:49.338 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/ 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 75937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 78697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 52305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ec_bhf.ko 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/ 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/ 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 356353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 63177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fujitsu/ 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 51505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 115561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/ 2024-05-27T00:01:49.339 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 366193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/ 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 755673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/ 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 324513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/ 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1196201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/ 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 362609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/ 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 672313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/ 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 121513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 966601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-27T00:01:49.340 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 193465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 108625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/ 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 126873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 150769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/ 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 904929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 344089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1413529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ 2024-05-27T00:01:49.341 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 54217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 117345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/ 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 131353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 91457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 56681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/ 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 181201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/ 2024-05-27T00:01:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 171409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/ 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 58241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/ 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 300793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 105681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 885057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/ 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 52297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/ 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 81633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 100681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-27T00:01:49.343 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-27T00:01:49.344 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 264145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-27T00:01:49.344 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/samsung/ 2024-05-27T00:01:49.344 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-05-27T00:01:49.344 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 142401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-05-27T00:01:49.344 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/ 2024-05-27T00:01:49.344 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1350201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-27T00:01:49.378 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/ 2024-05-27T00:01:49.378 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 57945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-27T00:01:49.378 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/ 2024-05-27T00:01:49.378 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 66497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 80057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/ 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 74281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 79449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 100777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 83433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/ 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 46977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 644329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-27T00:01:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/ 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 128489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 207985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 82625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 91345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/ 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/ 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 110721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/ 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 98681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 102505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/ 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-27T00:01:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/ 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 69905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/ 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/defxx.ko 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/skfp/ 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 275713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/skfp/skfp.ko 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 91665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/ 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 290441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/ 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-27T00:01:49.381 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 108313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 78057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/ 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-gpio.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 62673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-27T00:01:49.382 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/ 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 48233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 52857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 53401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-27T00:01:49.383 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 103369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 96017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 121097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 62105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/plip/ 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/plip/plip.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-27T00:01:49.384 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 53225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sb1000.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/ 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 52065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 80153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/ 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 137313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-27T00:01:49.385 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 108081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/ 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 176481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/ 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 384329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/ 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 175513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-27T00:01:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/ 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/ 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 335489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 127497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/ 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 168113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 103881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/ 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/ 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-05-27T00:01:49.387 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pci-stub.ko 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/ 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 179937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pcmcia.ko 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 78145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pcmcia_core.ko 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 133745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/ 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/broadcom/ 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/samsung/ 2024-05-27T00:01:49.388 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/ 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/ 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 78185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/ 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/ 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec.ko 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-05-27T00:01:49.389 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/ 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 87929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 48249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 73857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 164761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 46177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 62065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 51873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-27T00:01:49.390 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/ibm_rtl.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/ 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 53993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 51889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-27T00:01:49.391 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/ 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/88pm860x_battery.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/88pm860x_charger.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 68825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/bq2415x_charger.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 73065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/bq24190_charger.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/bq24735-charger.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/da9030_battery.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/da9052-battery.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2760_battery.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2780_battery.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2781_battery.ko 2024-05-27T00:01:49.392 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2782_battery.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/gpio-charger.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/isp1704_charger.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/lp8727_charger.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max14577_charger.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max17042_battery.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max8903_charger.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max8925_power.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/pcf50633-charger.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/sbs-battery.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/test_power.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/tps65090-charger.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/wm831x_backup.ko 2024-05-27T00:01:49.393 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/wm831x_power.ko 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/wm8350_power.ko 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/ 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 73289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/ 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/ 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 91369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-27T00:01:49.394 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/ 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/ 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/ 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 67473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 47129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/ 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-88pm80x.ko 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-88pm860x.ko 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-da9052.ko 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-da9055.ko 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-da9063.ko 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-27T00:01:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 82217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-lp8788.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 46905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-27T00:01:49.396 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t86.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8907.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8925.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8997.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8998.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-palmas.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 56769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf50633.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-27T00:01:49.397 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rc5t583.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-s35390a.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-tps6586x.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-tps65910.ko 2024-05-27T00:01:49.398 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-wm831x.ko 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-wm8350.ko 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 90369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 69585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 164473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/ 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 340689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 120393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/ 2024-05-27T00:01:49.399 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 468049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 380633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic94xx/ 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 243649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/ 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 135601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/ 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 304969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/ 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1527017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/ 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 297769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/ 2024-05-27T00:01:49.400 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 144377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 47449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/ 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 390657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/ 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 88873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 117473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 173009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 91497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/ 2024-05-27T00:01:49.401 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 267521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 78961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/ 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 95321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 194001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fdomain.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/ 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 366457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 251497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 57089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/imm.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 327833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-27T00:01:49.402 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 129777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/ 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 397657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 63113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/ 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 386801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 178169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 64801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/ 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 273953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/ 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 2735913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-27T00:01:49.403 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/ 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 88521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 389089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 104761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/ 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 836897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/ 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 158705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 96529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/ 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 101609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-05-27T00:01:49.404 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/ 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 443713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 152025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ppa.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 66969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/ 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 2324713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 103593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/ 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 697281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qlogicfas408.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-27T00:01:49.405 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 322857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 219849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 357145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 121177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 93473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 138113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 57393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/ 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 205001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 48769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-27T00:01:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/xen-scsifront.ko 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 174865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/ 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/ 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 920073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/ 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 60313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 48121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 54225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1170873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 51889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 149777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/ 2024-05-27T00:01:49.407 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 90033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/ 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/ 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/ 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-27T00:01:49.408 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 99289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 69721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/moxa.ko 2024-05-27T00:01:49.409 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/mxser.ko 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 134329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 78961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/ 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/ 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/altera_uart.ko 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 91593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/ 2024-05-27T00:01:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 120553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/rp2.ko 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/sc16is7xx.ko 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/ 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 693057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/ 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-27T00:01:49.411 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_dmem_genirq.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_mf624.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_netx.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/ 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/c67x00/ 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 90297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/c67x00/c67x00.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 189169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-05-27T00:01:49.412 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/ 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 101537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 63017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 56713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 107409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/common/ 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/common/ulpi.ko 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc2/ 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/ 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 575217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/dwc3.ko 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/ 2024-05-27T00:01:49.413 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/ 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 72241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/u_ether.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/u_serial.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 131513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 62313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 146601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 70569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-05-27T00:01:49.414 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-05-27T00:01:49.415 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 70009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-05-27T00:01:49.415 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 101233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-05-27T00:01:49.415 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-05-27T00:01:49.415 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 71241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-05-27T00:01:49.415 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 84929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-05-27T00:01:49.444 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/ 2024-05-27T00:01:49.444 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-05-27T00:01:49.444 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-05-27T00:01:49.444 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-05-27T00:01:49.444 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 78065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 204009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/libcomposite.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/ 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-05-27T00:01:49.445 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/bdc/ 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 182537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 66505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 69305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 90305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/net2272.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 90353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/net2280.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 94833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 80513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 87457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 95433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 195953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/ 2024-05-27T00:01:49.446 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/bcma-hcd.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 108313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/isp116x-hcd.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 104433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 102265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 60097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811_cs.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/ssb-hcd.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/ 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 47345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/cytherm.ko 2024-05-27T00:01:49.447 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ehset.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/lvstest.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/ 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-27T00:01:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 84257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbtest.ko 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 52057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/mon/ 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 91505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/mon/usbmon.ko 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/musb/ 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 396169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/musb/musb_hdrc.ko 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/ 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-isp1301.ko 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-tahvo.ko 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ 2024-05-27T00:01:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 79673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 66865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 58321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 53001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/f81232.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 188601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 152297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 142825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 69209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-27T00:01:49.450 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 66353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 122321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/metro-usb.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 79673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 89961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mxuport.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-27T00:01:49.451 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 197201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 70489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-27T00:01:49.452 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 151121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usbserial.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/wishbone-serial.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 64041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-27T00:01:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 201145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/ 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 76025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 122233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/ 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/ 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 206897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-27T00:01:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 134241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 93665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/ 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 138641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 71273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 107897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/ 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/ 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/aat2870_bl.ko 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/adp5520_bl.ko 2024-05-27T00:01:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/adp8860_bl.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 51737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/adp8870_bl.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/as3711_bl.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/bd6107.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/da903x_bl.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/da9052_bl.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/gpio_backlight.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/kb3886_bl.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lm3533_bl.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lm3639_bl.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lv5207lp.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/max8925_bl.ko 2024-05-27T00:01:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/pandora_bl.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/wm831x_bl.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/ 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 46793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/matrox_w1.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/w1-gpio.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/ 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-27T00:01:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 70361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 129409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/acquirewdt.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/advantechwdt.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-27T00:01:49.458 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/cpu5wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/da9052_wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/da9055_wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/da9063_wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/dw_wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/eurotechwdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-27T00:01:49.459 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/kempld_wdt.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/mena21_wdt.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pc87413_wdt.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 46945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/retu_wdt.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sc1200wdt.ko 2024-05-27T00:01:49.460 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/twl4030_wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wafer5823wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wm831x_wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wm8350_wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/ 2024-05-27T00:01:49.461 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/ 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 213865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 47673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-scsiback.ko 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/ 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/ 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 194449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/adfs/ 2024-05-27T00:01:49.462 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 78049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/adfs/adfs.ko 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/ 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 197801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/afs/ 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1260937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/afs/kafs.ko 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/autofs/ 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 127337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/autofs/autofs4.ko 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/ 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 80481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/bfs/ 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/bfs/bfs.ko 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-27T00:01:49.463 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/ 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 4491713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/ 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 376801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1586617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/ 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 130017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/configfs/ 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 139049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/configfs/configfs.ko 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/ 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/ 2024-05-27T00:01:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 813137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/efs/ 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 46377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/efs/efs.ko 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/ 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1906713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/ 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/freevxfs/ 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 52265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/freevxfs/freevxfs.ko 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/ 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/ 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1486393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/ 2024-05-27T00:01:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 183217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/ 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 281993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hpfs/ 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 248945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hpfs/hpfs.ko 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/ 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 92985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/ 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 532953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/ 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 361281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/ 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 112569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-27T00:01:49.466 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/ 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 591625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/ 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/ 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 102537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/ 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 74337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/ 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 147673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1320377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 82089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv2.ko 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 151617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 2518385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/ 2024-05-27T00:01:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/ 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 2021969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/ 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 733969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/ 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-27T00:01:49.468 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ascii.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-27T00:01:49.469 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 95617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 144201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 167129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 119505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-27T00:01:49.470 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-27T00:01:49.471 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ntfs3/ 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 634457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ntfs3/ntfs3.ko 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 302033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 548057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 66345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 3245449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 22105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-27T00:01:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/omfs/ 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 67465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/omfs/omfs.ko 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/ 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 462817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 67961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx4/ 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx4/qnx4.ko 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx6/ 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 56073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx6/qnx6.ko 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/ 2024-05-27T00:01:49.473 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/quota_tree.ko 2024-05-27T00:01:49.474 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/quota_v1.ko 2024-05-27T00:01:49.474 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/quota_v2.ko 2024-05-27T00:01:49.474 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/ 2024-05-27T00:01:49.474 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 684713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-27T00:01:49.474 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/ 2024-05-27T00:01:49.474 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-27T00:01:49.474 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/ 2024-05-27T00:01:49.474 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/ 2024-05-27T00:01:49.474 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 3350841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-27T00:01:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/ 2024-05-27T00:01:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-27T00:01:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-27T00:01:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/ 2024-05-27T00:01:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 164145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-27T00:01:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/ 2024-05-27T00:01:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 115865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-27T00:01:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/ 2024-05-27T00:01:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 309937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-27T00:01:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 198185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/ 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 5192193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/ 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/locking/ 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 97617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/locking/locktorture.ko 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/time/ 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/time/test_udelay.ko 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 72657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc7.ko 2024-05-27T00:01:49.508 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/ 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/interval_tree_test.ko 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/ 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/ 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/memory-notifier-error-inject.ko 2024-05-27T00:01:49.509 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/notifier-error-inject.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/percpu_test.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/pm-notifier-error-inject.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/ 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 151945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/rbtree_test.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/ 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test-kstrtox.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 4912433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_bpf.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 61161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_firmware.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_module.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_user_copy.ko 2024-05-27T00:01:49.510 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/xz/ 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/xz/xz_dec_test.ko 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/mm/ 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/ 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 66945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-27T00:01:49.511 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/ 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/ 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 106697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/ 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 236769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-27T00:01:49.512 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/ 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 879593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/ 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1031177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-27T00:01:49.513 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-27T00:01:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 70521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/ 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1243369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/core/ 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 153793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/ 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 244825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 71025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 74985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-27T00:01:49.515 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/ 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 507649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/hsr/ 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 128577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/hsr/hsr.ko 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 357369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 59385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ 2024-05-27T00:01:49.516 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 46409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 60217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 75481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 77713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-27T00:01:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ip_tables.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-27T00:01:49.518 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-27T00:01:49.519 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 54425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-27T00:01:49.520 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 47425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 103169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_gre.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 118873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 71081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/mip6.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-27T00:01:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 46297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-27T00:01:49.522 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 29809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 94081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 30289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/key/ 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 106361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/ 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 153209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-27T00:01:49.523 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 66233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/lapb/ 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 63321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/lapb/lapb.ko 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/ 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 332817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc2.ko 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/ 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 388817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/ 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ 2024-05-27T00:01:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 147729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 124033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 124337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 126673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 126465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 137825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 72345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 133673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 139065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 138257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-27T00:01:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 136169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 139889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 534033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-27T00:01:49.526 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 55209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 496713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 134361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 140793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-27T00:01:49.527 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 79865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 117385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 46097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 118337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-27T00:01:49.528 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 53385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 811705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 53041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 65137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 50649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-27T00:01:49.529 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_osf.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-27T00:01:49.530 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 24985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_synproxy.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tproxy.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 108385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/x_tables.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 21857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-27T00:01:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-27T00:01:49.532 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-27T00:01:49.533 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 52809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 16153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-27T00:01:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 11873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 19769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-27T00:01:49.535 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpudp.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/ 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 17225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/ 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 14241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/ 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 488457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 10865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-27T00:01:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/ 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/phonet/ 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 141993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/phonet/phonet.ko 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 68801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/phonet/pn_pep.ko 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/ 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 318537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 202249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 85073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rxrpc/ 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1028129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rxrpc/rxrpc.ko 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/ 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-27T00:01:49.537 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 28713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 27585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 53457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 45665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 41449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_cgroup.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 37297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-27T00:01:49.538 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 49313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 79137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 15377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 48777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 8305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 25089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 43145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 39169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq_codel.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 42881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 62809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-27T00:01:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 81825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 36617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 44785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 32825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 13497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 31793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 60241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 40881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 33425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 38321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 35769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-27T00:01:49.540 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 34153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/ 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1187153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/ 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/ 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 432225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 70081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1940753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/ 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 889729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/ 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 1122273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/ 2024-05-27T00:01:49.541 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 18793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/ 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 105601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 90249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 123745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 12897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 20393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/x25/ 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 165777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/x25/x25.ko 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/ 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 26721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_algo.ko 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 131857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_user.ko 2024-05-27T00:01:49.542 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 646387 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 630543 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 7706 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 23609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 9941 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 70990 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 146985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 232892 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 210 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 72627 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 879 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 258450 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 313019 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/ 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/share/ 2024-05-27T00:01:49.543 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/share/doc/ 2024-05-27T00:01:49.544 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/share/doc/linux-image-6.9.0-gb9459a26b088/ 2024-05-27T00:01:49.544 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 171 2024-05-23 11:07 ./usr/share/doc/linux-image-6.9.0-gb9459a26b088/changelog.Debian.gz 2024-05-27T00:01:49.544 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r-- root/root 691 2024-05-23 11:07 ./usr/share/doc/linux-image-6.9.0-gb9459a26b088/copyright 2024-05-27T00:01:49.549 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-27T00:01:49.549 INFO:teuthology.task.kernel:searching for kernel 6.9.0-gb9459a26b088 2024-05-27T00:01:49.549 DEBUG:teuthology.orchestra.run.smithi098:> egrep '(submenu|menuentry.*6.9.0-gb9459a26b088).*{' /boot/grub/grub.cfg 2024-05-27T00:01:49.555 INFO:teuthology.orchestra.run.smithi098.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-05-27T00:01:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: menuentry 'Ubuntu, with Linux 6.9.0-gb9459a26b088' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-gb9459a26b088-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-05-27T00:01:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: menuentry 'Ubuntu, with Linux 6.9.0-gb9459a26b088 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-gb9459a26b088-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-05-27T00:01:49.555 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-05-27T00:01:49.555 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.9.0-gb9459a26b088 2024-05-27T00:01:49.556 DEBUG:teuthology.orchestra.run.smithi098:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-05-27T00:01:49.597 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi098.front.sepia.ceph.com to complete... 2024-05-27T00:01:50.023 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub' 2024-05-27T00:01:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-27T00:01:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-05-27T00:01:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T00:01:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-6.9.0-gb9459a26b088 2024-05-27T00:01:50.499 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-27T00:01:50.505 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-27T00:01:50.546 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-27T00:01:50.552 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-27T00:01:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-27T00:01:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-27T00:01:50.698 INFO:teuthology.orchestra.run.smithi098.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-27T00:01:50.708 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-05-27T00:02:21.854 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-27T00:02:21.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-27T00:02:21.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T00:02:40.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-05-27T00:02:49.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-27T00:02:49.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T00:02:52.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-05-27T00:03:04.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-27T00:03:04.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T00:03:36.349 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-05-27T00:03:36.961 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-27T00:03:36.961 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-gb9459a26b088"... 2024-05-27T00:03:36.961 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-05-27T00:03:37.009 INFO:teuthology.orchestra.run.smithi098.stdout:6.9.0-gb9459a26b088 2024-05-27T00:03:37.009 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-27T00:03:37.009 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T00:03:37.009 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-27T00:03:38.010 DEBUG:teuthology.parallel:result is None 2024-05-27T00:03:38.011 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-27T00:03:38.044 INFO:teuthology.task.internal:Creating test directory... 2024-05-27T00:03:38.045 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T00:03:38.048 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T00:03:38.055 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-27T00:03:38.115 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-27T00:03:38.122 INFO:teuthology.task.internal:Creating archive directory... 2024-05-27T00:03:38.122 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T00:03:38.125 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T00:03:38.147 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-27T00:03:38.154 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-27T00:03:38.154 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T00:03:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:03:38.184 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T00:03:38.188 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T00:03:38.251 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T00:03:38.265 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T00:03:38.287 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T00:03:38.318 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T00:03:38.320 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-27T00:03:38.361 INFO:teuthology.task.internal:Configuring sudo... 2024-05-27T00:03:38.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T00:03:38.364 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T00:03:38.382 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-27T00:03:38.393 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-27T00:03:38.393 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T00:03:38.433 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T00:03:38.439 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T00:03:38.494 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T00:03:38.546 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:03:38.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T00:03:38.622 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T00:03:38.627 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T00:03:38.676 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:03:38.677 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T00:03:38.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-05-27T00:03:38.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-05-27T00:03:38.843 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-27T00:03:38.853 INFO:teuthology.task.internal:Starting timer... 2024-05-27T00:03:38.854 INFO:teuthology.run_tasks:Running task pcp... 2024-05-27T00:03:38.867 INFO:teuthology.run_tasks:Running task selinux... 2024-05-27T00:03:38.877 DEBUG:teuthology.task.selinux:Excluding smithi098: OS 'ubuntu' does not support SELinux 2024-05-27T00:03:38.878 DEBUG:teuthology.task.selinux:Excluding smithi204: OS 'ubuntu' does not support SELinux 2024-05-27T00:03:38.878 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-27T00:03:38.878 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-05-27T00:03:38.878 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-27T00:03:38.878 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-27T00:03:38.899 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-27T00:03:38.910 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-27T00:03:38.911 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-27T00:11:15.143 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-05-27T00:11:15.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-27T00:11:15.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T00:11:15.230 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-05-27T00:11:15.353 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-05-27T00:11:15.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-05-27T00:11:15.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T00:11:15.433 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-05-27T00:11:15.546 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-05-27T00:11:15.546 INFO:teuthology.run_tasks:Running task clock... 2024-05-27T00:11:15.556 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-27T00:11:15.557 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T00:11:15.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T00:11:15.560 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T00:11:15.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T00:11:15.595 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-27T00:11:15.595 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: Command line: ntpd -gq 2024-05-27T00:11:15.595 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: ---------------------------------------------------- 2024-05-27T00:11:15.595 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: ntp-4 is maintained by Network Time Foundation, 2024-05-27T00:11:15.595 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-27T00:11:15.596 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: corporation. Support and training for ntp-4 are 2024-05-27T00:11:15.596 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: available at https://www.nwtime.org/support 2024-05-27T00:11:15.596 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: ---------------------------------------------------- 2024-05-27T00:11:15.596 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: proto: precision = 0.043 usec (-24) 2024-05-27T00:11:15.596 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: basedate set to 2022-02-04 2024-05-27T00:11:15.596 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: gps base set to 2022-02-06 (week 2196) 2024-05-27T00:11:15.597 INFO:teuthology.orchestra.run.smithi098.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-27T00:11:15.597 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-27T00:11:15.597 INFO:teuthology.orchestra.run.smithi098.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-27T00:11:15.598 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: restrict ::: KOD does nothing without LIMITED. 2024-05-27T00:11:15.598 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: Listen and drop on 0 v6wildcard [::]:123 2024-05-27T00:11:15.598 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-27T00:11:15.599 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: Listen normally on 2 lo 127.0.0.1:123 2024-05-27T00:11:15.599 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: Listen normally on 3 enp3s0f1 172.21.15.98:123 2024-05-27T00:11:15.599 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: Listen normally on 4 lo [::1]:123 2024-05-27T00:11:15.599 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d26b%5]:123 2024-05-27T00:11:15.599 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:15 ntpd[17594]: Listening on routing socket on fd #22 for interface updates 2024-05-27T00:11:15.611 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-27T00:11:15.611 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: Command line: ntpd -gq 2024-05-27T00:11:15.611 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: ---------------------------------------------------- 2024-05-27T00:11:15.611 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: ntp-4 is maintained by Network Time Foundation, 2024-05-27T00:11:15.611 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-27T00:11:15.612 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: corporation. Support and training for ntp-4 are 2024-05-27T00:11:15.612 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: available at https://www.nwtime.org/support 2024-05-27T00:11:15.612 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: ---------------------------------------------------- 2024-05-27T00:11:15.612 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: proto: precision = 0.043 usec (-24) 2024-05-27T00:11:15.612 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: basedate set to 2022-02-04 2024-05-27T00:11:15.612 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: gps base set to 2022-02-06 (week 2196) 2024-05-27T00:11:15.613 INFO:teuthology.orchestra.run.smithi204.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-27T00:11:15.613 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-27T00:11:15.613 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: restrict ::: KOD does nothing without LIMITED. 2024-05-27T00:11:15.613 INFO:teuthology.orchestra.run.smithi204.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-27T00:11:15.614 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: Listen and drop on 0 v6wildcard [::]:123 2024-05-27T00:11:15.614 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-27T00:11:15.614 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: Listen normally on 2 lo 127.0.0.1:123 2024-05-27T00:11:15.614 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: Listen normally on 3 enp3s0f1 172.21.15.204:123 2024-05-27T00:11:15.615 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: Listen normally on 4 lo [::1]:123 2024-05-27T00:11:15.615 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:77ab%5]:123 2024-05-27T00:11:15.615 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:15 ntpd[17999]: Listening on routing socket on fd #22 for interface updates 2024-05-27T00:11:16.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:16 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:16.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:16 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:16.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:16 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:16.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:16 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:16.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:16 ntpd[17594]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:16.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:16 ntpd[17594]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:17.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:17 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:17.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:17 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:17.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:17 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:17.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:17 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:17 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:17 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:17 ntpd[17594]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:17.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:17 ntpd[17594]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:18 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:18 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:18 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:18 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:18 ntpd[17594]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:18 ntpd[17594]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:18.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:18 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:18.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:18 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:18.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:18 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:18.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:18 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:18.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:18 ntpd[17999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:18.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:18 ntpd[17999]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:19.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:19 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:19.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:19 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:19.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:19 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:19.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:19 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:19.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:19 ntpd[17594]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:19.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:19 ntpd[17594]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:19.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:19.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:19.614 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:19.614 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:19.614 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:19.614 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:19.614 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:19.614 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:19.615 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:19.615 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:19.615 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:19.615 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:19.615 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:19.615 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:19 ntpd[17999]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:20.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:20.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:20 ntpd[17594]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:20.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:20 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:20.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:20 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:20.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:20 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:20.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:20 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:20.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:20 ntpd[17999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:20.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:20 ntpd[17999]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:21.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:21 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:21.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:21 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:21.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:21 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:21.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:21 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:21.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:21 ntpd[17594]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:21.596 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:21 ntpd[17594]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:21.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:21.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:21 ntpd[17999]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:22.598 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:22.598 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:22.598 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:22.598 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:22.598 INFO:teuthology.orchestra.run.smithi098.stdout:27 May 00:11:22 ntpd[17594]: ntpd: time slew -0.000727 s 2024-05-27T00:11:22.598 INFO:teuthology.orchestra.run.smithi098.stdout:ntpd: time slew -0.000727s 2024-05-27T00:11:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-27T00:11:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:27 May 00:11:22 ntpd[17594]: can't open /var/log/ntpstats/loopstats.20240527: Permission denied 2024-05-27T00:11:22.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:22 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:22.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:22 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:22.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:22 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:22.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:22 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:22.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:22 ntpd[17999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:22.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:22 ntpd[17999]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2024-05-27T00:11:22.691 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2024-05-27T00:11:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T00:11:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T00:11:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T00:11:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:23.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:23.614 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:23 ntpd[17999]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:24.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:24 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:24.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:24 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:24.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:24 ntpd[17999]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T00:11:24.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:24 ntpd[17999]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T00:11:24.612 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:24 ntpd[17999]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T00:11:24.612 INFO:teuthology.orchestra.run.smithi204.stdout:27 May 00:11:24 ntpd[17999]: ntpd: time slew -0.000576 s 2024-05-27T00:11:24.613 INFO:teuthology.orchestra.run.smithi204.stdout:ntpd: time slew -0.000576s 2024-05-27T00:11:24.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:24 ntpd[17999]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T00:11:24.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:24 ntpd[17999]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-27T00:11:24.613 INFO:teuthology.orchestra.run.smithi204.stderr:27 May 00:11:24 ntpd[17999]: can't open /var/log/ntpstats/loopstats.20240527: Permission denied 2024-05-27T00:11:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: remote refid st t when poll reach delay offset jitter 2024-05-27T00:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================== 2024-05-27T00:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T00:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T00:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T00:11:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T00:11:24.697 INFO:teuthology.run_tasks:Running task install... 2024-05-27T00:11:24.704 DEBUG:teuthology.task.install:project ceph 2024-05-27T00:11:24.705 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0ef46a246e1118e5adafa1857b4189e7d780c816'}} 2024-05-27T00:11:24.705 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '0ef46a246e1118e5adafa1857b4189e7d780c816'} 2024-05-27T00:11:24.705 INFO:teuthology.task.install:Using flavor: default 2024-05-27T00:11:24.710 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-27T00:11:24.710 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-27T00:11:24.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-key list | grep Ceph 2024-05-27T00:11:24.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-key list | grep Ceph 2024-05-27T00:11:24.807 INFO:teuthology.orchestra.run.smithi204.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-27T00:11:24.815 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-27T00:11:24.836 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-27T00:11:24.836 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph.com (release key) 2024-05-27T00:11:24.837 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-05-27T00:11:24.837 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-05-27T00:11:24.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0ef46a246e1118e5adafa1857b4189e7d780c816 2024-05-27T00:11:24.872 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-27T00:11:24.872 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph.com (release key) 2024-05-27T00:11:24.872 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-05-27T00:11:24.872 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-05-27T00:11:24.873 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0ef46a246e1118e5adafa1857b4189e7d780c816 2024-05-27T00:11:24.986 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default/ 2024-05-27T00:11:24.986 INFO:teuthology.task.install.deb:Package version is 19.0.0-2857-g0ef46a24-1jammy 2024-05-27T00:11:25.013 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default/ 2024-05-27T00:11:25.013 INFO:teuthology.task.install.deb:Package version is 19.0.0-2857-g0ef46a24-1jammy 2024-05-27T00:11:25.127 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:11:25.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-27T00:11:25.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-05-27T00:11:25.157 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:11:25.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-27T00:11:25.170 DEBUG:teuthology.orchestra.run.smithi204:> sudo apt-get update 2024-05-27T00:11:25.359 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-27T00:11:25.359 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-27T00:11:25.371 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-27T00:11:25.379 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy InRelease 2024-05-27T00:11:25.391 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-27T00:11:25.400 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-27T00:11:25.423 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-27T00:11:25.444 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-05-27T00:11:25.452 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy InRelease 2024-05-27T00:11:25.476 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,401 B] 2024-05-27T00:11:25.480 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-27T00:11:25.480 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-27T00:11:25.489 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-27T00:11:25.530 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-27T00:11:25.554 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-27T00:11:25.567 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-05-27T00:11:25.610 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,401 B] 2024-05-27T00:11:25.636 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-27T00:11:25.844 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 28.9 kB in 0s (58.2 kB/s) 2024-05-27T00:11:25.950 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 28.9 kB in 1s (49.7 kB/s) 2024-05-27T00:11:26.815 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-05-27T00:11:26.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2857-g0ef46a24-1jammy cephadm=19.0.0-2857-g0ef46a24-1jammy ceph-mds=19.0.0-2857-g0ef46a24-1jammy ceph-mgr=19.0.0-2857-g0ef46a24-1jammy ceph-common=19.0.0-2857-g0ef46a24-1jammy ceph-fuse=19.0.0-2857-g0ef46a24-1jammy ceph-test=19.0.0-2857-g0ef46a24-1jammy ceph-volume=19.0.0-2857-g0ef46a24-1jammy radosgw=19.0.0-2857-g0ef46a24-1jammy python3-rados=19.0.0-2857-g0ef46a24-1jammy python3-rgw=19.0.0-2857-g0ef46a24-1jammy python3-cephfs=19.0.0-2857-g0ef46a24-1jammy python3-rbd=19.0.0-2857-g0ef46a24-1jammy libcephfs2=19.0.0-2857-g0ef46a24-1jammy libcephfs-dev=19.0.0-2857-g0ef46a24-1jammy librados2=19.0.0-2857-g0ef46a24-1jammy librbd1=19.0.0-2857-g0ef46a24-1jammy rbd-fuse=19.0.0-2857-g0ef46a24-1jammy python3-cephfs=19.0.0-2857-g0ef46a24-1jammy cephfs-shell=19.0.0-2857-g0ef46a24-1jammy cephfs-top=19.0.0-2857-g0ef46a24-1jammy cephfs-mirror=19.0.0-2857-g0ef46a24-1jammy 2024-05-27T00:11:26.884 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-05-27T00:11:26.938 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-05-27T00:11:26.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2857-g0ef46a24-1jammy cephadm=19.0.0-2857-g0ef46a24-1jammy ceph-mds=19.0.0-2857-g0ef46a24-1jammy ceph-mgr=19.0.0-2857-g0ef46a24-1jammy ceph-common=19.0.0-2857-g0ef46a24-1jammy ceph-fuse=19.0.0-2857-g0ef46a24-1jammy ceph-test=19.0.0-2857-g0ef46a24-1jammy ceph-volume=19.0.0-2857-g0ef46a24-1jammy radosgw=19.0.0-2857-g0ef46a24-1jammy python3-rados=19.0.0-2857-g0ef46a24-1jammy python3-rgw=19.0.0-2857-g0ef46a24-1jammy python3-cephfs=19.0.0-2857-g0ef46a24-1jammy python3-rbd=19.0.0-2857-g0ef46a24-1jammy libcephfs2=19.0.0-2857-g0ef46a24-1jammy libcephfs-dev=19.0.0-2857-g0ef46a24-1jammy librados2=19.0.0-2857-g0ef46a24-1jammy librbd1=19.0.0-2857-g0ef46a24-1jammy rbd-fuse=19.0.0-2857-g0ef46a24-1jammy python3-cephfs=19.0.0-2857-g0ef46a24-1jammy cephfs-shell=19.0.0-2857-g0ef46a24-1jammy cephfs-top=19.0.0-2857-g0ef46a24-1jammy cephfs-mirror=19.0.0-2857-g0ef46a24-1jammy 2024-05-27T00:11:27.027 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-05-27T00:11:27.063 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2024-05-27T00:11:27.063 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2024-05-27T00:11:27.209 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-05-27T00:11:27.210 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-05-27T00:11:27.238 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2024-05-27T00:11:27.239 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-27T00:11:27.239 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-27T00:11:27.239 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-27T00:11:27.240 INFO:teuthology.orchestra.run.smithi204.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-27T00:11:27.240 INFO:teuthology.orchestra.run.smithi204.stdout: python2.7-minimal 2024-05-27T00:11:27.240 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-05-27T00:11:27.242 INFO:teuthology.orchestra.run.smithi204.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-05-27T00:11:27.243 INFO:teuthology.orchestra.run.smithi204.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-05-27T00:11:27.244 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2024-05-27T00:11:27.244 INFO:teuthology.orchestra.run.smithi204.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-27T00:11:27.244 INFO:teuthology.orchestra.run.smithi204.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-27T00:11:27.244 INFO:teuthology.orchestra.run.smithi204.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-27T00:11:27.244 INFO:teuthology.orchestra.run.smithi204.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-27T00:11:27.244 INFO:teuthology.orchestra.run.smithi204.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-27T00:11:27.244 INFO:teuthology.orchestra.run.smithi204.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-27T00:11:27.244 INFO:teuthology.orchestra.run.smithi204.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-27T00:11:27.244 INFO:teuthology.orchestra.run.smithi204.stdout:Recommended packages: 2024-05-27T00:11:27.245 INFO:teuthology.orchestra.run.smithi204.stdout: btrfs-tools 2024-05-27T00:11:27.349 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2024-05-27T00:11:27.349 INFO:teuthology.orchestra.run.smithi204.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-27T00:11:27.349 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-27T00:11:27.349 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-05-27T00:11:27.349 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-05-27T00:11:27.351 INFO:teuthology.orchestra.run.smithi204.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-27T00:11:27.351 INFO:teuthology.orchestra.run.smithi204.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-05-27T00:11:27.351 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-05-27T00:11:27.351 INFO:teuthology.orchestra.run.smithi204.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-05-27T00:11:27.351 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-05-27T00:11:27.351 INFO:teuthology.orchestra.run.smithi204.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-05-27T00:11:27.352 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-05-27T00:11:27.353 INFO:teuthology.orchestra.run.smithi204.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-27T00:11:27.353 INFO:teuthology.orchestra.run.smithi204.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-27T00:11:27.353 INFO:teuthology.orchestra.run.smithi204.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-27T00:11:27.353 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-27T00:11:27.353 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-05-27T00:11:27.353 INFO:teuthology.orchestra.run.smithi204.stdout: socat unzip xmlstarlet zip 2024-05-27T00:11:27.391 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-05-27T00:11:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-27T00:11:27.391 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-27T00:11:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-27T00:11:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-27T00:11:27.392 INFO:teuthology.orchestra.run.smithi098.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-27T00:11:27.392 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-27T00:11:27.393 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-05-27T00:11:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-27T00:11:27.393 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-27T00:11:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-27T00:11:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-27T00:11:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-27T00:11:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-27T00:11:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-27T00:11:27.395 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-27T00:11:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-27T00:11:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-05-27T00:11:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-05-27T00:11:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-05-27T00:11:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-05-27T00:11:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-05-27T00:11:27.396 INFO:teuthology.orchestra.run.smithi098.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-05-27T00:11:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-05-27T00:11:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-05-27T00:11:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-05-27T00:11:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-05-27T00:11:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-05-27T00:11:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-05-27T00:11:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-05-27T00:11:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-05-27T00:11:27.397 INFO:teuthology.orchestra.run.smithi098.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-05-27T00:11:27.398 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-05-27T00:11:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-27T00:11:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-27T00:11:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-27T00:11:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-27T00:11:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-27T00:11:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-27T00:11:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-27T00:11:27.399 INFO:teuthology.orchestra.run.smithi098.stdout:Recommended packages: 2024-05-27T00:11:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: btrfs-tools 2024-05-27T00:11:27.541 INFO:teuthology.orchestra.run.smithi204.stdout:0 upgraded, 124 newly installed, 0 to remove and 310 not upgraded. 2024-05-27T00:11:27.541 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 180 MB of archives. 2024-05-27T00:11:27.542 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 823 MB of additional disk space will be used. 2024-05-27T00:11:27.542 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-05-27T00:11:27.554 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-05-27T00:11:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-27T00:11:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-27T00:11:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-27T00:11:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-05-27T00:11:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: socat unzip xmlstarlet zip 2024-05-27T00:11:27.570 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2857-g0ef46a24-1jammy [3,547 kB] 2024-05-27T00:11:27.623 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-27T00:11:27.628 INFO:teuthology.orchestra.run.smithi098.stdout:0 upgraded, 124 newly installed, 0 to remove and 310 not upgraded. 2024-05-27T00:11:27.628 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 180 MB of archives. 2024-05-27T00:11:27.628 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 823 MB of additional disk space will be used. 2024-05-27T00:11:27.628 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-05-27T00:11:27.650 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-27T00:11:27.661 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-05-27T00:11:27.669 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-05-27T00:11:27.703 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2857-g0ef46a24-1jammy [3,547 kB] 2024-05-27T00:11:27.711 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-27T00:11:27.736 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-27T00:11:27.739 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-27T00:11:27.753 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-27T00:11:27.799 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-27T00:11:27.801 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-27T00:11:27.809 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-27T00:11:27.812 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-27T00:11:27.812 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-27T00:11:27.819 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-27T00:11:27.821 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-27T00:11:27.823 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-27T00:11:27.824 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-27T00:11:27.824 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-27T00:11:27.843 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-27T00:11:27.845 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-27T00:11:27.846 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-27T00:11:27.847 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-27T00:11:27.847 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-27T00:11:27.863 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-27T00:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-27T00:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-27T00:11:27.864 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-27T00:11:27.865 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-27T00:11:27.884 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-27T00:11:27.884 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-27T00:11:27.886 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-27T00:11:27.887 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-27T00:11:27.887 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-27T00:11:27.897 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-27T00:11:27.904 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-27T00:11:27.905 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-27T00:11:27.905 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-27T00:11:27.906 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-27T00:11:27.907 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-27T00:11:27.925 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-27T00:11:27.926 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-27T00:11:27.931 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-27T00:11:27.931 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-27T00:11:27.931 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-27T00:11:27.946 INFO:teuthology.orchestra.run.smithi098.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-27T00:11:27.947 INFO:teuthology.orchestra.run.smithi098.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-27T00:11:27.949 INFO:teuthology.orchestra.run.smithi098.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-05-27T00:11:27.949 INFO:teuthology.orchestra.run.smithi098.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-27T00:11:27.951 INFO:teuthology.orchestra.run.smithi098.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-27T00:11:27.966 INFO:teuthology.orchestra.run.smithi098.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-27T00:11:27.968 INFO:teuthology.orchestra.run.smithi098.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-27T00:11:27.968 INFO:teuthology.orchestra.run.smithi098.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-27T00:11:27.969 INFO:teuthology.orchestra.run.smithi098.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-27T00:11:27.971 INFO:teuthology.orchestra.run.smithi098.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-27T00:11:27.973 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2857-g0ef46a24-1jammy [3,218 kB] 2024-05-27T00:11:27.976 INFO:teuthology.orchestra.run.smithi098.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2857-g0ef46a24-1jammy [3,218 kB] 2024-05-27T00:11:28.023 INFO:teuthology.orchestra.run.smithi098.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-27T00:11:28.024 INFO:teuthology.orchestra.run.smithi098.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-27T00:11:28.026 INFO:teuthology.orchestra.run.smithi098.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-27T00:11:28.026 INFO:teuthology.orchestra.run.smithi098.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-27T00:11:28.027 INFO:teuthology.orchestra.run.smithi098.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-27T00:11:28.046 INFO:teuthology.orchestra.run.smithi098.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-27T00:11:28.048 INFO:teuthology.orchestra.run.smithi098.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-27T00:11:28.049 INFO:teuthology.orchestra.run.smithi098.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-27T00:11:28.066 INFO:teuthology.orchestra.run.smithi098.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-27T00:11:28.067 INFO:teuthology.orchestra.run.smithi098.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-27T00:11:28.067 INFO:teuthology.orchestra.run.smithi098.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-27T00:11:28.068 INFO:teuthology.orchestra.run.smithi098.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-27T00:11:28.068 INFO:teuthology.orchestra.run.smithi098.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-27T00:11:28.069 INFO:teuthology.orchestra.run.smithi098.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-27T00:11:28.084 INFO:teuthology.orchestra.run.smithi098.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-27T00:11:28.085 INFO:teuthology.orchestra.run.smithi098.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-27T00:11:28.088 INFO:teuthology.orchestra.run.smithi098.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-27T00:11:28.094 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-27T00:11:28.104 INFO:teuthology.orchestra.run.smithi098.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-27T00:11:28.106 INFO:teuthology.orchestra.run.smithi098.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-27T00:11:28.107 INFO:teuthology.orchestra.run.smithi098.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-27T00:11:28.111 INFO:teuthology.orchestra.run.smithi098.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-27T00:11:28.113 INFO:teuthology.orchestra.run.smithi098.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-05-27T00:11:28.114 INFO:teuthology.orchestra.run.smithi098.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-27T00:11:28.115 INFO:teuthology.orchestra.run.smithi098.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-27T00:11:28.126 INFO:teuthology.orchestra.run.smithi098.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-27T00:11:28.131 INFO:teuthology.orchestra.run.smithi098.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-27T00:11:28.132 INFO:teuthology.orchestra.run.smithi098.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-27T00:11:28.136 INFO:teuthology.orchestra.run.smithi098.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2857-g0ef46a24-1jammy [713 kB] 2024-05-27T00:11:28.146 INFO:teuthology.orchestra.run.smithi098.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-27T00:11:28.147 INFO:teuthology.orchestra.run.smithi098.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-27T00:11:28.147 INFO:teuthology.orchestra.run.smithi098.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-27T00:11:28.148 INFO:teuthology.orchestra.run.smithi098.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-27T00:11:28.149 INFO:teuthology.orchestra.run.smithi098.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-27T00:11:28.150 INFO:teuthology.orchestra.run.smithi098.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-27T00:11:28.154 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-27T00:11:28.158 INFO:teuthology.orchestra.run.smithi098.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-27T00:11:28.167 INFO:teuthology.orchestra.run.smithi098.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-27T00:11:28.167 INFO:teuthology.orchestra.run.smithi098.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-27T00:11:28.170 INFO:teuthology.orchestra.run.smithi098.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-27T00:11:28.177 INFO:teuthology.orchestra.run.smithi098.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2857-g0ef46a24-1jammy [356 kB] 2024-05-27T00:11:28.187 INFO:teuthology.orchestra.run.smithi098.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-27T00:11:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2857-g0ef46a24-1jammy [32.8 kB] 2024-05-27T00:11:28.196 INFO:teuthology.orchestra.run.smithi098.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2857-g0ef46a24-1jammy [180 kB] 2024-05-27T00:11:28.210 INFO:teuthology.orchestra.run.smithi098.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2857-g0ef46a24-1jammy [67.1 kB] 2024-05-27T00:11:28.213 INFO:teuthology.orchestra.run.smithi098.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2857-g0ef46a24-1jammy [334 kB] 2024-05-27T00:11:28.219 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2857-g0ef46a24-1jammy [713 kB] 2024-05-27T00:11:28.235 INFO:teuthology.orchestra.run.smithi098.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2857-g0ef46a24-1jammy [6,780 kB] 2024-05-27T00:11:28.276 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2857-g0ef46a24-1jammy [356 kB] 2024-05-27T00:11:28.299 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2857-g0ef46a24-1jammy [32.8 kB] 2024-05-27T00:11:28.302 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2857-g0ef46a24-1jammy [180 kB] 2024-05-27T00:11:28.319 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2857-g0ef46a24-1jammy [67.1 kB] 2024-05-27T00:11:28.328 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2857-g0ef46a24-1jammy [334 kB] 2024-05-27T00:11:28.349 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2857-g0ef46a24-1jammy [6,780 kB] 2024-05-27T00:11:28.397 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-27T00:11:28.621 INFO:teuthology.orchestra.run.smithi098.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2857-g0ef46a24-1jammy [112 kB] 2024-05-27T00:11:28.627 INFO:teuthology.orchestra.run.smithi098.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2857-g0ef46a24-1jammy [466 kB] 2024-05-27T00:11:28.653 INFO:teuthology.orchestra.run.smithi098.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2857-g0ef46a24-1jammy [25.6 MB] 2024-05-27T00:11:28.874 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2857-g0ef46a24-1jammy [112 kB] 2024-05-27T00:11:28.880 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2857-g0ef46a24-1jammy [466 kB] 2024-05-27T00:11:28.915 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2857-g0ef46a24-1jammy [25.6 MB] 2024-05-27T00:11:30.118 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-27T00:11:30.119 INFO:teuthology.orchestra.run.smithi098.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2857-g0ef46a24-1jammy [5,084 kB] 2024-05-27T00:11:30.401 INFO:teuthology.orchestra.run.smithi098.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2857-g0ef46a24-1jammy [244 kB] 2024-05-27T00:11:30.412 INFO:teuthology.orchestra.run.smithi098.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2857-g0ef46a24-1jammy [124 kB] 2024-05-27T00:11:30.419 INFO:teuthology.orchestra.run.smithi098.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2857-g0ef46a24-1jammy [1,525 kB] 2024-05-27T00:11:30.499 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-27T00:11:30.506 INFO:teuthology.orchestra.run.smithi098.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2857-g0ef46a24-1jammy [6,248 kB] 2024-05-27T00:11:30.864 INFO:teuthology.orchestra.run.smithi098.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2857-g0ef46a24-1jammy [22.7 MB] 2024-05-27T00:11:31.023 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2857-g0ef46a24-1jammy [5,084 kB] 2024-05-27T00:11:31.113 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-27T00:11:31.297 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2857-g0ef46a24-1jammy [244 kB] 2024-05-27T00:11:31.326 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2857-g0ef46a24-1jammy [124 kB] 2024-05-27T00:11:31.333 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2857-g0ef46a24-1jammy [1,525 kB] 2024-05-27T00:11:31.422 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-27T00:11:31.444 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2857-g0ef46a24-1jammy [6,248 kB] 2024-05-27T00:11:31.618 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-27T00:11:31.927 INFO:teuthology.orchestra.run.smithi204.stdout:Get:30 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2857-g0ef46a24-1jammy [22.7 MB] 2024-05-27T00:11:32.161 INFO:teuthology.orchestra.run.smithi098.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2857-g0ef46a24-1jammy [14.0 kB] 2024-05-27T00:11:32.161 INFO:teuthology.orchestra.run.smithi098.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2857-g0ef46a24-1jammy [869 kB] 2024-05-27T00:11:32.206 INFO:teuthology.orchestra.run.smithi098.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2857-g0ef46a24-1jammy [2,439 kB] 2024-05-27T00:11:32.348 INFO:teuthology.orchestra.run.smithi098.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2857-g0ef46a24-1jammy [792 kB] 2024-05-27T00:11:32.392 INFO:teuthology.orchestra.run.smithi098.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2857-g0ef46a24-1jammy [148 kB] 2024-05-27T00:11:32.401 INFO:teuthology.orchestra.run.smithi098.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2857-g0ef46a24-1jammy [2,324 kB] 2024-05-27T00:11:32.534 INFO:teuthology.orchestra.run.smithi098.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2857-g0ef46a24-1jammy [8,625 kB] 2024-05-27T00:11:32.761 INFO:teuthology.orchestra.run.smithi204.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-27T00:11:32.811 INFO:teuthology.orchestra.run.smithi204.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-27T00:11:32.873 INFO:teuthology.orchestra.run.smithi204.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-27T00:11:32.987 INFO:teuthology.orchestra.run.smithi204.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-27T00:11:32.987 INFO:teuthology.orchestra.run.smithi204.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-27T00:11:33.072 INFO:teuthology.orchestra.run.smithi098.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2857-g0ef46a24-1jammy [14.2 kB] 2024-05-27T00:11:33.072 INFO:teuthology.orchestra.run.smithi098.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2857-g0ef46a24-1jammy [50.7 MB] 2024-05-27T00:11:33.122 INFO:teuthology.orchestra.run.smithi204.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-27T00:11:33.280 INFO:teuthology.orchestra.run.smithi204.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-27T00:11:33.413 INFO:teuthology.orchestra.run.smithi204.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-27T00:11:33.452 INFO:teuthology.orchestra.run.smithi204.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-27T00:11:33.459 INFO:teuthology.orchestra.run.smithi204.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-27T00:11:33.698 INFO:teuthology.orchestra.run.smithi204.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-27T00:11:33.698 INFO:teuthology.orchestra.run.smithi204.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2857-g0ef46a24-1jammy [14.0 kB] 2024-05-27T00:11:33.699 INFO:teuthology.orchestra.run.smithi204.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2857-g0ef46a24-1jammy [869 kB] 2024-05-27T00:11:33.717 INFO:teuthology.orchestra.run.smithi204.stdout:Get:44 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2857-g0ef46a24-1jammy [2,439 kB] 2024-05-27T00:11:33.781 INFO:teuthology.orchestra.run.smithi204.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-27T00:11:33.787 INFO:teuthology.orchestra.run.smithi204.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-27T00:11:33.787 INFO:teuthology.orchestra.run.smithi204.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2857-g0ef46a24-1jammy [792 kB] 2024-05-27T00:11:33.848 INFO:teuthology.orchestra.run.smithi204.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2857-g0ef46a24-1jammy [148 kB] 2024-05-27T00:11:33.854 INFO:teuthology.orchestra.run.smithi204.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2857-g0ef46a24-1jammy [2,324 kB] 2024-05-27T00:11:33.863 INFO:teuthology.orchestra.run.smithi204.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-27T00:11:33.889 INFO:teuthology.orchestra.run.smithi204.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-27T00:11:33.905 INFO:teuthology.orchestra.run.smithi204.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-27T00:11:33.925 INFO:teuthology.orchestra.run.smithi204.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-27T00:11:34.001 INFO:teuthology.orchestra.run.smithi204.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2857-g0ef46a24-1jammy [8,625 kB] 2024-05-27T00:11:34.133 INFO:teuthology.orchestra.run.smithi204.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-27T00:11:34.151 INFO:teuthology.orchestra.run.smithi204.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-27T00:11:34.159 INFO:teuthology.orchestra.run.smithi204.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-27T00:11:34.166 INFO:teuthology.orchestra.run.smithi204.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-27T00:11:34.365 INFO:teuthology.orchestra.run.smithi204.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-27T00:11:34.455 INFO:teuthology.orchestra.run.smithi204.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-27T00:11:34.468 INFO:teuthology.orchestra.run.smithi204.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-27T00:11:34.568 INFO:teuthology.orchestra.run.smithi204.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2857-g0ef46a24-1jammy [14.2 kB] 2024-05-27T00:11:34.569 INFO:teuthology.orchestra.run.smithi204.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2857-g0ef46a24-1jammy [50.7 MB] 2024-05-27T00:11:34.659 INFO:teuthology.orchestra.run.smithi204.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-27T00:11:34.756 INFO:teuthology.orchestra.run.smithi204.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-27T00:11:34.785 INFO:teuthology.orchestra.run.smithi204.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-27T00:11:35.507 INFO:teuthology.orchestra.run.smithi204.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-27T00:11:35.511 INFO:teuthology.orchestra.run.smithi204.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-27T00:11:35.600 INFO:teuthology.orchestra.run.smithi204.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-27T00:11:35.636 INFO:teuthology.orchestra.run.smithi204.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-27T00:11:35.830 INFO:teuthology.orchestra.run.smithi204.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-27T00:11:36.230 INFO:teuthology.orchestra.run.smithi204.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-05-27T00:11:36.356 INFO:teuthology.orchestra.run.smithi098.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2857-g0ef46a24-1jammy [121 kB] 2024-05-27T00:11:36.357 INFO:teuthology.orchestra.run.smithi098.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2857-g0ef46a24-1jammy [229 kB] 2024-05-27T00:11:36.361 INFO:teuthology.orchestra.run.smithi098.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2857-g0ef46a24-1jammy [34.0 kB] 2024-05-27T00:11:36.364 INFO:teuthology.orchestra.run.smithi098.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2857-g0ef46a24-1jammy [23.4 kB] 2024-05-27T00:11:36.366 INFO:teuthology.orchestra.run.smithi098.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2857-g0ef46a24-1jammy [38.9 kB] 2024-05-27T00:11:36.368 INFO:teuthology.orchestra.run.smithi098.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2857-g0ef46a24-1jammy [13.4 MB] 2024-05-27T00:11:36.500 INFO:teuthology.orchestra.run.smithi204.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-27T00:11:36.614 INFO:teuthology.orchestra.run.smithi204.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-27T00:11:36.657 INFO:teuthology.orchestra.run.smithi204.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-27T00:11:36.739 INFO:teuthology.orchestra.run.smithi204.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-27T00:11:36.752 INFO:teuthology.orchestra.run.smithi204.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-27T00:11:36.766 INFO:teuthology.orchestra.run.smithi204.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-27T00:11:36.805 INFO:teuthology.orchestra.run.smithi204.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-27T00:11:37.069 INFO:teuthology.orchestra.run.smithi204.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-27T00:11:37.071 INFO:teuthology.orchestra.run.smithi204.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-27T00:11:37.073 INFO:teuthology.orchestra.run.smithi204.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-27T00:11:37.232 INFO:teuthology.orchestra.run.smithi098.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2857-g0ef46a24-1jammy [91.4 kB] 2024-05-27T00:11:37.240 INFO:teuthology.orchestra.run.smithi204.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-27T00:11:37.578 INFO:teuthology.orchestra.run.smithi204.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-27T00:11:37.868 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 180 MB in 10s (18.6 MB/s) 2024-05-27T00:11:37.878 INFO:teuthology.orchestra.run.smithi204.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2857-g0ef46a24-1jammy [121 kB] 2024-05-27T00:11:37.880 INFO:teuthology.orchestra.run.smithi204.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2857-g0ef46a24-1jammy [229 kB] 2024-05-27T00:11:37.885 INFO:teuthology.orchestra.run.smithi204.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2857-g0ef46a24-1jammy [34.0 kB] 2024-05-27T00:11:37.886 INFO:teuthology.orchestra.run.smithi204.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2857-g0ef46a24-1jammy [23.4 kB] 2024-05-27T00:11:37.887 INFO:teuthology.orchestra.run.smithi204.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2857-g0ef46a24-1jammy [38.9 kB] 2024-05-27T00:11:37.888 INFO:teuthology.orchestra.run.smithi204.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2857-g0ef46a24-1jammy [13.4 MB] 2024-05-27T00:11:38.017 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyperclip. 2024-05-27T00:11:38.082 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132924 files and directories currently installed.) 2024-05-27T00:11:38.085 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-05-27T00:11:38.124 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-05-27T00:11:38.242 INFO:teuthology.orchestra.run.smithi204.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/squid/0ef46a246e1118e5adafa1857b4189e7d780c816/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2857-g0ef46a24-1jammy [91.4 kB] 2024-05-27T00:11:38.374 INFO:teuthology.orchestra.run.smithi204.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-27T00:11:38.414 INFO:teuthology.orchestra.run.smithi204.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-27T00:11:38.416 INFO:teuthology.orchestra.run.smithi204.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-27T00:11:38.427 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-27T00:11:38.440 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-27T00:11:38.485 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-27T00:11:38.797 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cmd2. 2024-05-27T00:11:38.811 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-05-27T00:11:38.846 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-05-27T00:11:39.158 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-27T00:11:39.173 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-27T00:11:39.258 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-27T00:11:39.567 INFO:teuthology.orchestra.run.smithi204.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-27T00:11:39.568 INFO:teuthology.orchestra.run.smithi204.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-27T00:11:39.595 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-27T00:11:39.609 INFO:teuthology.orchestra.run.smithi204.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-27T00:11:39.610 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-27T00:11:39.626 INFO:teuthology.orchestra.run.smithi204.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-27T00:11:39.653 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-27T00:11:39.661 INFO:teuthology.orchestra.run.smithi204.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-27T00:11:39.744 INFO:teuthology.orchestra.run.smithi204.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-27T00:11:39.929 INFO:teuthology.orchestra.run.smithi204.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-27T00:11:39.981 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-27T00:11:39.996 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-27T00:11:40.040 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-27T00:11:40.134 INFO:teuthology.orchestra.run.smithi204.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-27T00:11:40.410 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-27T00:11:40.425 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T00:11:40.445 INFO:teuthology.orchestra.run.smithi204.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-27T00:11:40.478 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:11:40.489 INFO:teuthology.orchestra.run.smithi204.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-27T00:11:40.517 INFO:teuthology.orchestra.run.smithi204.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-27T00:11:40.526 INFO:teuthology.orchestra.run.smithi204.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-27T00:11:40.574 INFO:teuthology.orchestra.run.smithi204.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-27T00:11:40.598 INFO:teuthology.orchestra.run.smithi204.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-05-27T00:11:40.600 INFO:teuthology.orchestra.run.smithi204.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-27T00:11:40.607 INFO:teuthology.orchestra.run.smithi204.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-27T00:11:40.611 INFO:teuthology.orchestra.run.smithi204.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-27T00:11:40.810 INFO:teuthology.orchestra.run.smithi204.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-27T00:11:40.905 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-27T00:11:40.920 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T00:11:40.963 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:11:41.069 INFO:teuthology.orchestra.run.smithi204.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-27T00:11:41.308 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-27T00:11:41.323 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T00:11:41.357 INFO:teuthology.orchestra.run.smithi204.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-27T00:11:41.358 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:11:41.361 INFO:teuthology.orchestra.run.smithi204.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-27T00:11:41.389 INFO:teuthology.orchestra.run.smithi204.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-27T00:11:41.394 INFO:teuthology.orchestra.run.smithi204.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-27T00:11:41.604 INFO:teuthology.orchestra.run.smithi204.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-27T00:11:41.626 INFO:teuthology.orchestra.run.smithi204.stdout:Get:119 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-27T00:11:41.704 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-27T00:11:41.718 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-27T00:11:41.761 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-27T00:11:42.064 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librados2. 2024-05-27T00:11:42.079 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../010-librados2_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:42.122 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librados2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:42.201 INFO:teuthology.orchestra.run.smithi204.stdout:Get:120 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-27T00:11:42.231 INFO:teuthology.orchestra.run.smithi204.stdout:Get:121 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-27T00:11:42.241 INFO:teuthology.orchestra.run.smithi204.stdout:Get:122 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-27T00:11:42.565 INFO:teuthology.orchestra.run.smithi204.stdout:Get:123 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-27T00:11:42.575 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librbd1. 2024-05-27T00:11:42.591 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../011-librbd1_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:42.634 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librbd1 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:42.837 INFO:teuthology.orchestra.run.smithi204.stdout:Get:124 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-27T00:11:42.986 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs2. 2024-05-27T00:11:43.001 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:43.045 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:43.331 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rados. 2024-05-27T00:11:43.346 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:43.389 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rados (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:43.892 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 180 MB in 16s (11.3 MB/s) 2024-05-27T00:11:44.103 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyperclip. 2024-05-27T00:11:44.125 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-27T00:11:44.140 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:11:44.183 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-argparse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:44.188 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 130504 files and directories currently installed.) 2024-05-27T00:11:44.192 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-05-27T00:11:44.234 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-05-27T00:11:44.428 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cephfs. 2024-05-27T00:11:44.443 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:44.470 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-27T00:11:44.478 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cephfs (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:44.491 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-27T00:11:44.529 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-27T00:11:44.798 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-27T00:11:44.812 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:11:44.848 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cmd2. 2024-05-27T00:11:44.856 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-common (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:44.868 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-05-27T00:11:44.906 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-05-27T00:11:45.167 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-prettytable. 2024-05-27T00:11:45.182 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-05-27T00:11:45.226 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-27T00:11:45.251 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-27T00:11:45.271 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-27T00:11:45.351 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-27T00:11:45.470 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rbd. 2024-05-27T00:11:45.484 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:45.553 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rbd (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:45.705 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-27T00:11:45.725 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-27T00:11:45.763 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-27T00:11:45.890 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-27T00:11:45.904 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-27T00:11:45.940 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-27T00:11:46.058 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-27T00:11:46.078 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-27T00:11:46.107 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-27T00:11:46.184 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-27T00:11:46.199 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-27T00:11:46.242 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-27T00:11:46.545 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-27T00:11:46.565 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T00:11:46.604 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-27T00:11:46.611 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:11:46.618 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-27T00:11:46.671 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-27T00:11:46.999 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua5.1. 2024-05-27T00:11:47.014 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-27T00:11:47.023 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-27T00:11:47.044 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T00:11:47.057 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-27T00:11:47.081 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:11:47.369 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-any. 2024-05-27T00:11:47.383 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-05-27T00:11:47.419 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-27T00:11:47.437 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-27T00:11:47.455 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T00:11:47.492 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:11:47.680 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package zip. 2024-05-27T00:11:47.695 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-05-27T00:11:47.730 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking zip (3.0-12build2) ... 2024-05-27T00:11:47.888 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-27T00:11:47.908 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-27T00:11:47.954 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-27T00:11:48.025 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package unzip. 2024-05-27T00:11:48.040 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-27T00:11:48.083 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-27T00:11:48.374 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librados2. 2024-05-27T00:11:48.394 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../010-librados2_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:48.440 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librados2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:48.454 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package luarocks. 2024-05-27T00:11:48.468 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-27T00:11:48.529 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-27T00:11:48.911 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librgw2. 2024-05-27T00:11:48.925 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../027-librgw2_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:48.951 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librbd1. 2024-05-27T00:11:48.970 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librgw2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:48.972 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../011-librbd1_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:49.009 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librbd1 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:49.539 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rgw. 2024-05-27T00:11:49.554 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:49.598 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rgw (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:49.687 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs2. 2024-05-27T00:11:49.708 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:49.754 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:49.918 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-27T00:11:49.932 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-27T00:11:49.967 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-27T00:11:50.081 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rados. 2024-05-27T00:11:50.102 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:50.148 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rados (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:50.245 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libradosstriper1. 2024-05-27T00:11:50.260 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:50.303 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libradosstriper1 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:50.484 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-27T00:11:50.505 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:11:50.542 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-ceph-argparse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:50.741 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-common. 2024-05-27T00:11:50.755 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:50.798 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-common (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:50.803 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cephfs. 2024-05-27T00:11:50.815 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:50.853 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cephfs (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:51.131 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-27T00:11:51.143 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:11:51.173 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-ceph-common (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:51.426 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-prettytable. 2024-05-27T00:11:51.437 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-05-27T00:11:51.467 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-27T00:11:51.754 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rbd. 2024-05-27T00:11:51.774 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:51.820 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rbd (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:51.896 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-base. 2024-05-27T00:11:51.911 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:51.963 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-base (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:52.240 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-27T00:11:52.261 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-27T00:11:52.323 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-27T00:11:52.551 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-27T00:11:52.564 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-27T00:11:52.609 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-27T00:11:52.634 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-27T00:11:52.655 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-27T00:11:52.717 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-27T00:11:52.895 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cheroot. 2024-05-27T00:11:52.909 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-27T00:11:52.945 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-27T00:11:53.087 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-27T00:11:53.099 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-27T00:11:53.137 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-27T00:11:53.224 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-27T00:11:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-27T00:11:53.281 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-27T00:11:53.524 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package lua5.1. 2024-05-27T00:11:53.545 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-27T00:11:53.590 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-27T00:11:53.593 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-27T00:11:53.607 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-27T00:11:53.651 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-27T00:11:53.954 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-27T00:11:53.960 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package lua-any. 2024-05-27T00:11:53.969 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-27T00:11:53.981 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-05-27T00:11:54.012 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-27T00:11:54.026 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-27T00:11:54.274 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tempora. 2024-05-27T00:11:54.288 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-05-27T00:11:54.304 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package zip. 2024-05-27T00:11:54.316 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-05-27T00:11:54.323 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-27T00:11:54.354 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking zip (3.0-12build2) ... 2024-05-27T00:11:54.643 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-portend. 2024-05-27T00:11:54.658 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-05-27T00:11:54.691 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package unzip. 2024-05-27T00:11:54.701 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-27T00:11:54.711 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-27T00:11:54.757 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-27T00:11:54.996 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-27T00:11:55.011 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-27T00:11:55.054 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-27T00:11:55.245 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package luarocks. 2024-05-27T00:11:55.266 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-27T00:11:55.311 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-27T00:11:55.366 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-27T00:11:55.381 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-27T00:11:55.416 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-27T00:11:55.678 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librgw2. 2024-05-27T00:11:55.699 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../027-librgw2_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:55.735 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librgw2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:55.770 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-natsort. 2024-05-27T00:11:55.784 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-05-27T00:11:55.820 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-27T00:11:56.115 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-logutils. 2024-05-27T00:11:56.129 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-05-27T00:11:56.173 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-27T00:11:56.305 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rgw. 2024-05-27T00:11:56.317 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:56.355 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rgw (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:56.476 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-27T00:11:56.490 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-27T00:11:56.542 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-27T00:11:56.700 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-27T00:11:56.720 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-27T00:11:56.758 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-27T00:11:56.829 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-mako. 2024-05-27T00:11:56.843 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-27T00:11:56.887 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-27T00:11:57.036 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libradosstriper1. 2024-05-27T00:11:57.057 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:57.094 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libradosstriper1 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:57.224 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-27T00:11:57.238 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-27T00:11:57.273 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-27T00:11:57.556 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-common. 2024-05-27T00:11:57.577 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:57.585 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-27T00:11:57.600 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-27T00:11:57.614 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-common (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:57.643 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-27T00:11:57.946 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webob. 2024-05-27T00:11:57.961 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-27T00:11:58.004 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-27T00:11:58.341 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-waitress. 2024-05-27T00:11:58.355 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-27T00:11:58.402 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-27T00:11:59.045 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-base. 2024-05-27T00:11:59.066 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:11:59.111 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-base (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:11:59.461 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tempita. 2024-05-27T00:11:59.476 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-27T00:11:59.736 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-27T00:11:59.874 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-27T00:11:59.894 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-27T00:11:59.924 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-27T00:12:00.022 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-paste. 2024-05-27T00:12:00.044 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-27T00:12:00.089 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-27T00:12:00.143 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cheroot. 2024-05-27T00:12:00.155 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-27T00:12:00.184 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-27T00:12:00.400 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-27T00:12:00.414 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-27T00:12:00.459 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-27T00:12:00.479 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-27T00:12:00.501 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-27T00:12:00.538 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-27T00:12:00.770 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-27T00:12:00.785 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-27T00:12:00.828 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-27T00:12:00.841 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-27T00:12:00.862 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-27T00:12:00.899 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-27T00:12:01.132 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webtest. 2024-05-27T00:12:01.146 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-05-27T00:12:01.176 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-27T00:12:01.189 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-27T00:12:01.190 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-27T00:12:01.226 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-27T00:12:01.471 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-tempora. 2024-05-27T00:12:01.492 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-05-27T00:12:01.501 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pecan. 2024-05-27T00:12:01.516 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-27T00:12:01.521 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-27T00:12:01.562 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-27T00:12:01.841 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-portend. 2024-05-27T00:12:01.862 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-05-27T00:12:01.899 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-27T00:12:01.956 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-certifi. 2024-05-27T00:12:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-05-27T00:12:02.022 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-27T00:12:02.202 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-27T00:12:02.223 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-27T00:12:02.260 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-27T00:12:02.309 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-idna. 2024-05-27T00:12:02.324 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../057-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-05-27T00:12:02.367 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-05-27T00:12:02.521 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-27T00:12:02.533 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-27T00:12:02.563 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-27T00:12:02.704 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-urllib3. 2024-05-27T00:12:02.718 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-27T00:12:02.762 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-27T00:12:02.925 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-natsort. 2024-05-27T00:12:02.946 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-05-27T00:12:02.983 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-27T00:12:03.082 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-requests. 2024-05-27T00:12:03.097 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-27T00:12:03.140 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-27T00:12:03.295 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-logutils. 2024-05-27T00:12:03.316 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-05-27T00:12:03.353 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-27T00:12:03.452 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-27T00:12:03.467 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-27T00:12:03.518 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-27T00:12:03.656 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-27T00:12:03.677 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-27T00:12:03.714 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-27T00:12:03.838 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-27T00:12:03.853 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:03.896 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:03.975 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-mako. 2024-05-27T00:12:03.996 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-27T00:12:04.033 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-27T00:12:04.243 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-27T00:12:04.257 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:04.301 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:04.362 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-27T00:12:04.383 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-27T00:12:04.420 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-27T00:12:04.629 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-27T00:12:04.644 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-27T00:12:04.687 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-27T00:12:04.725 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-27T00:12:04.746 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-27T00:12:04.783 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-27T00:12:04.990 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-openssl. 2024-05-27T00:12:05.005 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-05-27T00:12:05.049 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-27T00:12:05.368 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr. 2024-05-27T00:12:05.383 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:05.418 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:05.797 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mon. 2024-05-27T00:12:05.811 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:05.836 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-webob. 2024-05-27T00:12:05.854 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mon (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:05.858 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-27T00:12:05.895 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-27T00:12:06.248 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-waitress. 2024-05-27T00:12:06.270 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-27T00:12:06.309 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-27T00:12:06.433 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-osd. 2024-05-27T00:12:06.448 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:06.492 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-osd (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:06.618 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-tempita. 2024-05-27T00:12:06.639 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-27T00:12:06.679 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-27T00:12:06.979 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-paste. 2024-05-27T00:12:06.999 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-27T00:12:07.037 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-27T00:12:07.287 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph. 2024-05-27T00:12:07.302 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../068-ceph_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:07.345 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:07.357 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-27T00:12:07.369 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-27T00:12:07.407 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-27T00:12:07.690 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-fuse. 2024-05-27T00:12:07.705 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:07.727 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-27T00:12:07.747 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-fuse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:07.748 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-27T00:12:07.785 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-27T00:12:08.076 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mds. 2024-05-27T00:12:08.091 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:08.096 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-webtest. 2024-05-27T00:12:08.117 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-05-27T00:12:08.126 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mds (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:08.167 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-27T00:12:08.449 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pecan. 2024-05-27T00:12:08.461 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-27T00:12:08.487 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephadm. 2024-05-27T00:12:08.491 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-27T00:12:08.502 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../071-cephadm_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:08.545 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephadm (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:08.779 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-certifi. 2024-05-27T00:12:08.791 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-05-27T00:12:08.829 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-27T00:12:08.881 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-27T00:12:08.896 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-27T00:12:08.940 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-27T00:12:09.166 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-idna. 2024-05-27T00:12:09.187 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../057-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-05-27T00:12:09.224 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-05-27T00:12:09.235 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-27T00:12:09.250 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-27T00:12:09.292 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-27T00:12:09.552 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-urllib3. 2024-05-27T00:12:09.573 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-27T00:12:09.610 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-27T00:12:09.955 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-requests. 2024-05-27T00:12:09.976 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-27T00:12:10.013 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-27T00:12:10.292 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-27T00:12:10.312 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-27T00:12:10.350 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-27T00:12:10.489 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-babel. 2024-05-27T00:12:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-27T00:12:10.547 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-27T00:12:10.644 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-27T00:12:10.656 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:10.694 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:10.892 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jinja2. 2024-05-27T00:12:10.906 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-27T00:12:10.950 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-27T00:12:11.075 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-27T00:12:11.095 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:11.133 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:11.304 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-27T00:12:11.318 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:11.362 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:11.436 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-27T00:12:11.456 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-27T00:12:11.494 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-27T00:12:11.732 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-27T00:12:11.747 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-05-27T00:12:11.772 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-openssl. 2024-05-27T00:12:11.790 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-27T00:12:11.792 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-05-27T00:12:11.830 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-27T00:12:12.093 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-routes. 2024-05-27T00:12:12.108 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-27T00:12:12.151 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-27T00:12:12.166 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr. 2024-05-27T00:12:12.187 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:12.225 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:12.522 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-27T00:12:12.537 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:12.617 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:12.661 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mon. 2024-05-27T00:12:12.682 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:12.719 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mon (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:13.307 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-osd. 2024-05-27T00:12:13.328 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:13.365 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-osd (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:13.829 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-27T00:12:13.843 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-27T00:12:13.887 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-27T00:12:14.366 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-joblib. 2024-05-27T00:12:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-27T00:12:14.423 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-27T00:12:14.443 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph. 2024-05-27T00:12:14.464 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../068-ceph_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:14.501 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:14.802 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-27T00:12:14.816 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-27T00:12:14.821 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-fuse. 2024-05-27T00:12:14.842 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:14.879 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-fuse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:14.946 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-27T00:12:15.216 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mds. 2024-05-27T00:12:15.237 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:15.274 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mds (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:15.282 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn. 2024-05-27T00:12:15.296 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-27T00:12:15.340 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-27T00:12:15.652 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package cephadm. 2024-05-27T00:12:15.673 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../071-cephadm_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:15.710 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking cephadm (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:16.033 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-27T00:12:16.047 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:16.092 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:16.289 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-27T00:12:16.309 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-27T00:12:16.346 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-27T00:12:16.683 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-27T00:12:16.695 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-27T00:12:16.733 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-27T00:12:17.087 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cachetools. 2024-05-27T00:12:17.102 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-05-27T00:12:17.137 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-27T00:12:17.423 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-27T00:12:17.438 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-27T00:12:17.481 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-27T00:12:17.813 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-babel. 2024-05-27T00:12:17.835 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-27T00:12:17.835 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-27T00:12:17.851 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-27T00:12:17.879 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-27T00:12:17.893 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-27T00:12:18.205 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rsa. 2024-05-27T00:12:18.216 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-jinja2. 2024-05-27T00:12:18.220 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-05-27T00:12:18.238 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-27T00:12:18.262 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-27T00:12:18.283 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-27T00:12:18.558 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-google-auth. 2024-05-27T00:12:18.573 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-05-27T00:12:18.607 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-27T00:12:18.628 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-27T00:12:18.649 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:18.686 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:18.942 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-27T00:12:18.951 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-27T00:12:18.994 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-27T00:12:19.048 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-27T00:12:19.069 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-05-27T00:12:19.106 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-27T00:12:19.306 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-websocket. 2024-05-27T00:12:19.321 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-05-27T00:12:19.364 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-27T00:12:19.393 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-routes. 2024-05-27T00:12:19.414 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-27T00:12:19.451 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-27T00:12:19.676 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-27T00:12:19.689 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-27T00:12:19.725 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-27T00:12:19.788 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-27T00:12:19.809 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:19.854 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:20.707 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-27T00:12:20.718 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:20.756 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:21.084 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-27T00:12:21.096 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-27T00:12:21.143 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-27T00:12:21.210 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-27T00:12:21.222 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-27T00:12:21.260 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-27T00:12:21.462 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-27T00:12:21.476 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-27T00:12:21.520 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-27T00:12:21.772 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-joblib. 2024-05-27T00:12:21.793 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-27T00:12:21.807 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package jq. 2024-05-27T00:12:21.820 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-27T00:12:21.830 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-27T00:12:21.865 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-27T00:12:22.101 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package socat. 2024-05-27T00:12:22.114 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-27T00:12:22.150 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-27T00:12:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-27T00:12:22.162 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-27T00:12:22.311 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-27T00:12:22.438 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package xmlstarlet. 2024-05-27T00:12:22.454 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-27T00:12:22.496 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-27T00:12:22.622 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-sklearn. 2024-05-27T00:12:22.643 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-27T00:12:22.688 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-27T00:12:22.859 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-test. 2024-05-27T00:12:22.872 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:22.908 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-test (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:23.439 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-27T00:12:23.461 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:23.505 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:24.442 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-cachetools. 2024-05-27T00:12:24.465 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-05-27T00:12:24.492 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-27T00:12:24.787 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-27T00:12:24.808 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-27T00:12:24.821 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-volume. 2024-05-27T00:12:24.836 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:24.845 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-27T00:12:24.896 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-volume (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:25.182 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-27T00:12:25.203 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-27T00:12:25.248 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-27T00:12:25.250 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephfs-mirror. 2024-05-27T00:12:25.264 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:25.316 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephfs-mirror (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:25.560 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-rsa. 2024-05-27T00:12:25.581 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-05-27T00:12:25.603 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-colorama. 2024-05-27T00:12:25.617 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-05-27T00:12:25.618 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-27T00:12:25.653 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-05-27T00:12:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-google-auth. 2024-05-27T00:12:25.939 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephfs-shell. 2024-05-27T00:12:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-05-27T00:12:25.953 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-27T00:12:25.997 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephfs-shell (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:26.250 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephfs-top. 2024-05-27T00:12:26.264 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:26.283 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-27T00:12:26.305 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-27T00:12:26.308 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephfs-top (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:26.342 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-27T00:12:26.569 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-27T00:12:26.583 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:26.627 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs-dev (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:26.653 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-websocket. 2024-05-27T00:12:26.675 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-05-27T00:12:26.711 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-27T00:12:26.947 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-27T00:12:26.961 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-27T00:12:26.996 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-27T00:12:27.032 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-27T00:12:27.053 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-27T00:12:27.089 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-27T00:12:27.310 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-27T00:12:27.327 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-05-27T00:12:27.367 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-27T00:12:27.722 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package nvme-cli. 2024-05-27T00:12:27.737 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-27T00:12:27.779 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-27T00:12:27.812 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-27T00:12:27.834 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:27.879 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:28.210 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package pkg-config. 2024-05-27T00:12:28.224 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-27T00:12:28.240 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-27T00:12:28.262 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-27T00:12:28.276 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-27T00:12:28.315 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-27T00:12:28.546 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-27T00:12:28.561 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-27T00:12:28.596 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-27T00:12:28.735 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-27T00:12:28.757 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-27T00:12:28.810 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-27T00:12:29.032 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-27T00:12:29.048 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-27T00:12:29.090 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-27T00:12:29.104 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package jq. 2024-05-27T00:12:29.117 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-27T00:12:29.163 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-27T00:12:29.352 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastescript. 2024-05-27T00:12:29.365 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-05-27T00:12:29.402 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-27T00:12:29.449 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package socat. 2024-05-27T00:12:29.470 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-27T00:12:29.515 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-27T00:12:29.738 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pluggy. 2024-05-27T00:12:29.751 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-27T00:12:29.789 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-27T00:12:29.861 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package xmlstarlet. 2024-05-27T00:12:29.882 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-27T00:12:29.935 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-27T00:12:30.116 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-psutil. 2024-05-27T00:12:30.129 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-27T00:12:30.166 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-27T00:12:30.348 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-test. 2024-05-27T00:12:30.369 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:30.423 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-test (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:30.494 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-py. 2024-05-27T00:12:30.506 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-05-27T00:12:30.544 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-27T00:12:31.098 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pygments. 2024-05-27T00:12:31.110 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-27T00:12:31.148 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-27T00:12:31.585 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-27T00:12:31.596 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-27T00:12:31.635 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-27T00:12:31.979 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-toml. 2024-05-27T00:12:31.991 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-05-27T00:12:32.029 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-27T00:12:32.316 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pytest. 2024-05-27T00:12:32.328 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-27T00:12:32.365 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-27T00:12:32.711 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplejson. 2024-05-27T00:12:32.723 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-27T00:12:32.760 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-27T00:12:32.778 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-volume. 2024-05-27T00:12:32.799 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:32.844 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-volume (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:33.038 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-27T00:12:33.050 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-27T00:12:33.088 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-27T00:12:33.223 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package cephfs-mirror. 2024-05-27T00:12:33.245 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:33.290 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking cephfs-mirror (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:33.585 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-colorama. 2024-05-27T00:12:33.607 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-05-27T00:12:33.651 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-05-27T00:12:33.811 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package radosgw. 2024-05-27T00:12:33.822 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../122-radosgw_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:33.884 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking radosgw (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:33.929 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package cephfs-shell. 2024-05-27T00:12:33.941 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:33.979 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking cephfs-shell (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:34.240 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package cephfs-top. 2024-05-27T00:12:34.253 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2857-g0ef46a24-1jammy_all.deb ... 2024-05-27T00:12:34.290 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking cephfs-top (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:34.503 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package rbd-fuse. 2024-05-27T00:12:34.515 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:34.534 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-27T00:12:34.545 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking rbd-fuse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:34.547 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:34.584 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs-dev (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:34.829 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-27T00:12:34.896 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-27T00:12:34.917 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-27T00:12:34.962 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-27T00:12:35.109 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-27T00:12:35.235 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-27T00:12:35.301 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-27T00:12:35.322 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-05-27T00:12:35.367 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-27T00:12:35.491 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-27T00:12:35.730 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package nvme-cli. 2024-05-27T00:12:35.751 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-27T00:12:35.796 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-27T00:12:35.908 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-27T00:12:36.227 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package pkg-config. 2024-05-27T00:12:36.248 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-27T00:12:36.293 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-27T00:12:36.481 INFO:teuthology.orchestra.run.smithi098.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-27T00:12:36.533 INFO:teuthology.orchestra.run.smithi098.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-05-27T00:12:36.554 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-27T00:12:36.576 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-27T00:12:36.578 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephadm (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:36.621 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-27T00:12:37.074 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-27T00:12:37.096 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-27T00:12:37.106 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user cephadm....done 2024-05-27T00:12:37.140 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-27T00:12:37.243 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-27T00:12:37.452 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pastescript. 2024-05-27T00:12:37.474 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-05-27T00:12:37.518 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-27T00:12:37.545 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-27T00:12:37.829 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-27T00:12:37.838 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pluggy. 2024-05-27T00:12:37.860 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-27T00:12:37.897 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-27T00:12:37.955 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-27T00:12:38.208 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-psutil. 2024-05-27T00:12:38.214 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-27T00:12:38.230 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-27T00:12:38.275 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-27T00:12:38.482 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-27T00:12:38.608 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-27T00:12:38.611 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-py. 2024-05-27T00:12:38.634 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-05-27T00:12:38.678 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-27T00:12:38.919 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-27T00:12:39.023 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pygments. 2024-05-27T00:12:39.045 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-27T00:12:39.089 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-27T00:12:39.296 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-27T00:12:39.519 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-27T00:12:39.541 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-27T00:12:39.547 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-05-27T00:12:39.585 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-27T00:12:39.832 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-27T00:12:39.896 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-toml. 2024-05-27T00:12:39.909 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-05-27T00:12:39.946 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-27T00:12:40.116 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-27T00:12:40.249 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-pytest. 2024-05-27T00:12:40.271 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-27T00:12:40.316 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-27T00:12:40.392 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-27T00:12:40.620 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python3-simplejson. 2024-05-27T00:12:40.642 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-27T00:12:40.652 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-27T00:12:40.686 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-27T00:12:40.964 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-27T00:12:40.976 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-27T00:12:41.014 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-27T00:12:41.087 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-27T00:12:41.479 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-argparse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:41.730 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-27T00:12:41.852 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package radosgw. 2024-05-27T00:12:41.874 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../122-radosgw_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:41.881 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-27T00:12:41.918 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking radosgw (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:42.007 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-05-27T00:12:42.275 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-27T00:12:42.401 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-27T00:12:42.661 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-27T00:12:42.729 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package rbd-fuse. 2024-05-27T00:12:42.751 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2857-g0ef46a24-1jammy_amd64.deb ... 2024-05-27T00:12:42.796 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking rbd-fuse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:43.065 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-27T00:12:43.098 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-27T00:12:43.124 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-27T00:12:43.188 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-27T00:12:43.314 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-27T00:12:43.353 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-27T00:12:43.479 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-27T00:12:43.682 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-27T00:12:43.741 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-27T00:12:43.825 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-27T00:12:44.101 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-27T00:12:44.119 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-27T00:12:44.360 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-27T00:12:44.653 INFO:teuthology.orchestra.run.smithi204.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-27T00:12:44.670 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up zip (3.0-12build2) ... 2024-05-27T00:12:44.756 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up cephadm (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:44.829 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-27T00:12:45.290 INFO:teuthology.orchestra.run.smithi204.stdout:Adding system user cephadm....done 2024-05-27T00:12:45.446 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-27T00:12:45.515 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-27T00:12:45.730 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-27T00:12:45.791 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-27T00:12:45.917 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-27T00:12:45.973 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-27T00:12:46.034 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-27T00:12:46.074 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-27T00:12:46.300 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-27T00:12:46.353 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-27T00:12:46.535 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-27T00:12:46.694 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-27T00:12:46.779 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-27T00:12:46.996 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-27T00:12:47.022 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-27T00:12:47.365 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-27T00:12:47.415 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-27T00:12:47.608 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-05-27T00:12:47.747 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-05-27T00:12:47.834 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-27T00:12:48.076 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-27T00:12:48.085 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-27T00:12:48.354 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-27T00:12:48.435 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-27T00:12:48.605 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-27T00:12:48.695 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-27T00:12:48.854 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-common (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:48.957 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-27T00:12:49.156 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-05-27T00:12:49.499 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-ceph-argparse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:49.507 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-27T00:12:49.741 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-27T00:12:49.843 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-27T00:12:49.960 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-05-27T00:12:50.220 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-27T00:12:50.346 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-27T00:12:50.605 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-27T00:12:50.771 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-27T00:12:50.989 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:12:51.023 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-27T00:12:51.054 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-27T00:12:51.101 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-27T00:12:51.124 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-27T00:12:51.218 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-27T00:12:51.409 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-27T00:12:51.534 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-27T00:12:51.595 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-27T00:12:51.660 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-27T00:12:51.721 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-27T00:12:51.990 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-27T00:12:52.004 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-27T00:12:52.257 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-27T00:12:52.263 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-27T00:12:52.506 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:12:52.533 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up zip (3.0-12build2) ... 2024-05-27T00:12:52.632 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-27T00:12:52.642 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-27T00:12:52.983 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-27T00:12:53.234 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-27T00:12:53.278 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-27T00:12:53.511 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-27T00:12:53.529 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-27T00:12:53.655 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-27T00:12:53.778 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-27T00:12:53.781 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-27T00:12:54.046 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-27T00:12:54.049 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-27T00:12:54.323 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-27T00:12:54.426 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-27T00:12:54.448 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-27T00:12:54.660 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-27T00:12:54.758 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-27T00:12:54.884 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-27T00:12:55.029 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-27T00:12:55.177 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-27T00:12:55.313 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-05-27T00:12:55.503 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-27T00:12:55.615 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-27T00:12:55.771 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-27T00:12:55.897 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-27T00:12:55.924 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-27T00:12:56.023 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-27T00:12:56.112 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-27T00:12:56.159 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-27T00:12:56.277 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-ceph-common (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:12:56.358 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-27T00:12:56.484 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-27T00:12:56.578 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-05-27T00:12:56.886 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-27T00:12:56.905 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-27T00:12:57.171 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-27T00:12:57.288 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-27T00:12:57.522 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:12:57.648 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-27T00:12:57.999 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-27T00:12:58.111 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-27T00:12:58.309 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-27T00:12:58.362 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:12:58.497 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-27T00:12:58.585 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-27T00:12:58.765 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-27T00:12:58.807 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-27T00:12:58.866 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-27T00:12:58.967 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-27T00:12:59.083 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-27T00:12:59.252 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-27T00:12:59.359 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-27T00:12:59.512 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-27T00:12:59.627 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-27T00:12:59.763 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:12:59.889 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-27T00:13:00.029 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-27T00:13:00.182 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-27T00:13:00.306 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-27T00:13:00.417 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-27T00:13:00.651 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-27T00:13:00.683 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-27T00:13:00.809 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librados2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:00.903 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-27T00:13:00.936 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:01.061 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-27T00:13:01.173 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-27T00:13:01.406 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:01.438 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-27T00:13:01.549 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libradosstriper1 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:01.564 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-27T00:13:01.700 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-27T00:13:01.832 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-27T00:13:01.933 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-27T00:13:01.985 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librbd1 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:02.086 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-modules-core (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:02.193 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-27T00:13:02.212 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-fuse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:02.419 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-27T00:13:02.419 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-27T00:13:02.478 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-27T00:13:02.729 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-27T00:13:02.871 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-27T00:13:02.997 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-27T00:13:03.087 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-27T00:13:03.147 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs-dev (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:03.273 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rados (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:03.332 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-27T00:13:03.399 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librgw2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:03.458 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-27T00:13:03.525 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephfs-top (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:03.667 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-27T00:13:03.843 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-27T00:13:04.112 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-27T00:13:04.221 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-27T00:13:04.489 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T00:13:04.615 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-27T00:13:04.924 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-27T00:13:04.946 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rbd (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:05.072 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up rbd-fuse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:05.198 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rgw (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:05.225 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-27T00:13:05.332 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cephfs (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:05.459 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephfs-shell (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:05.501 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-27T00:13:05.584 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-common (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:05.722 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-27T00:13:05.990 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-27T00:13:06.129 INFO:teuthology.orchestra.run.smithi098.stdout:Adding group ceph....done 2024-05-27T00:13:06.274 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-27T00:13:06.497 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user ceph....done 2024-05-27T00:13:06.517 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-27T00:13:06.700 INFO:teuthology.orchestra.run.smithi098.stdout:Setting system user ceph properties....done 2024-05-27T00:13:06.716 INFO:teuthology.orchestra.run.smithi098.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-27T00:13:06.856 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-27T00:13:06.894 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-27T00:13:07.154 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-27T00:13:07.270 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-27T00:13:07.473 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-27T00:13:07.599 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librados2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:07.726 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:07.852 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-27T00:13:07.936 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephfs-mirror (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:08.127 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-05-27T00:13:08.127 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-05-27T00:13:08.171 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:08.297 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libradosstriper1 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:08.424 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-27T00:13:08.700 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librbd1 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:08.812 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-test (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:08.826 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-modules-core (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:08.938 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up radosgw (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:08.952 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-fuse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:09.171 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-27T00:13:09.171 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-27T00:13:09.494 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-27T00:13:09.494 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-27T00:13:09.704 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs-dev (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:09.822 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rados (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:09.948 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librgw2 (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:10.074 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up cephfs-top (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:10.149 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-base (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:10.200 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-27T00:13:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-27T00:13:11.167 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mds (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:11.389 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-27T00:13:11.389 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-27T00:13:11.412 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rbd (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:11.538 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up rbd-fuse (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:11.664 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-rgw (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:11.790 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python3-cephfs (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:11.916 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up cephfs-shell (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:12.042 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-common (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:12.141 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:12.350 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-27T00:13:12.351 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-27T00:13:12.577 INFO:teuthology.orchestra.run.smithi204.stdout:Adding group ceph....done 2024-05-27T00:13:12.920 INFO:teuthology.orchestra.run.smithi204.stdout:Adding system user ceph....done 2024-05-27T00:13:13.032 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-osd (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:13.089 INFO:teuthology.orchestra.run.smithi204.stdout:Setting system user ceph properties....done 2024-05-27T00:13:13.100 INFO:teuthology.orchestra.run.smithi204.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-27T00:13:13.238 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-27T00:13:13.467 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-27T00:13:13.467 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-27T00:13:13.594 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-27T00:13:14.107 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:14.119 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up cephfs-mirror (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:14.233 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:14.324 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-05-27T00:13:14.324 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-05-27T00:13:14.376 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mon (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:14.576 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-27T00:13:14.576 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-27T00:13:14.854 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-test (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:14.980 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up radosgw (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:15.211 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-cephadm (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:15.336 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:15.462 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-dashboard (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:15.474 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-27T00:13:15.474 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-27T00:13:15.647 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-volume (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:16.000 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-27T00:13:16.007 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-base (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:16.109 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-27T00:13:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-27T00:13:16.361 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-27T00:13:16.876 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mds (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:17.079 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-27T00:13:17.120 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-27T00:13:17.120 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-27T00:13:17.669 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:17.753 INFO:teuthology.orchestra.run.smithi098.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-27T00:13:17.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-05-27T00:13:17.832 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-05-27T00:13:17.875 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-27T00:13:17.875 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-27T00:13:18.007 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-05-27T00:13:18.008 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-05-27T00:13:18.184 INFO:teuthology.orchestra.run.smithi098.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-05-27T00:13:18.184 INFO:teuthology.orchestra.run.smithi098.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-05-27T00:13:18.184 INFO:teuthology.orchestra.run.smithi098.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-05-27T00:13:18.184 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-05-27T00:13:18.184 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-27T00:13:18.185 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-27T00:13:18.185 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-27T00:13:18.185 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-27T00:13:18.185 INFO:teuthology.orchestra.run.smithi098.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-27T00:13:18.185 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-27T00:13:18.186 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-05-27T00:13:18.186 INFO:teuthology.orchestra.run.smithi098.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-05-27T00:13:18.186 INFO:teuthology.orchestra.run.smithi098.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-05-27T00:13:18.186 INFO:teuthology.orchestra.run.smithi098.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-05-27T00:13:18.186 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-05-27T00:13:18.187 INFO:teuthology.orchestra.run.smithi098.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-05-27T00:13:18.187 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-05-27T00:13:18.187 INFO:teuthology.orchestra.run.smithi098.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-05-27T00:13:18.187 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-05-27T00:13:18.187 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-05-27T00:13:18.242 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-05-27T00:13:18.242 INFO:teuthology.orchestra.run.smithi098.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-05-27T00:13:18.242 INFO:teuthology.orchestra.run.smithi098.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-05-27T00:13:18.243 INFO:teuthology.orchestra.run.smithi098.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-05-27T00:13:18.243 INFO:teuthology.orchestra.run.smithi098.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-05-27T00:13:18.243 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-05-27T00:13:18.243 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-05-27T00:13:18.243 INFO:teuthology.orchestra.run.smithi098.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-05-27T00:13:18.244 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages will be upgraded: 2024-05-27T00:13:18.245 INFO:teuthology.orchestra.run.smithi098.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-05-27T00:13:18.429 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-osd (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:18.451 INFO:teuthology.orchestra.run.smithi098.stdout:4 upgraded, 39 newly installed, 0 to remove and 306 not upgraded. 2024-05-27T00:13:18.451 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 55.7 MB of archives. 2024-05-27T00:13:18.451 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 202 MB of additional disk space will be used. 2024-05-27T00:13:18.452 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-05-27T00:13:18.740 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-27T00:13:18.740 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-27T00:13:19.092 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-05-27T00:13:19.108 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-05-27T00:13:19.233 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-05-27T00:13:19.234 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-05-27T00:13:19.235 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-05-27T00:13:19.264 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-05-27T00:13:19.265 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-05-27T00:13:19.268 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-05-27T00:13:19.271 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-05-27T00:13:19.272 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-05-27T00:13:19.282 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:19.437 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-05-27T00:13:19.540 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:19.691 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mon (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:19.730 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-05-27T00:13:19.740 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-05-27T00:13:19.745 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-05-27T00:13:19.829 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-05-27T00:13:19.907 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-27T00:13:19.907 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-27T00:13:20.488 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-cephadm (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:20.662 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:20.771 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-dashboard (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:20.801 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-05-27T00:13:20.804 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-05-27T00:13:20.804 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-05-27T00:13:20.829 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-05-27T00:13:20.829 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-05-27T00:13:20.834 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-05-27T00:13:20.922 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-volume (19.0.0-2857-g0ef46a24-1jammy) ... 2024-05-27T00:13:21.000 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-05-27T00:13:21.291 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-27T00:13:21.392 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-27T00:13:21.545 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-27T00:13:21.605 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-05-27T00:13:21.606 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-05-27T00:13:21.785 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-05-27T00:13:21.785 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-05-27T00:13:21.824 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-05-27T00:13:21.887 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-05-27T00:13:21.892 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-05-27T00:13:21.893 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-05-27T00:13:21.893 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-05-27T00:13:21.894 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-05-27T00:13:22.099 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-05-27T00:13:22.523 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-05-27T00:13:22.630 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-27T00:13:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-05-27T00:13:23.267 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-05-27T00:13:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-05-27T00:13:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-05-27T00:13:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-05-27T00:13:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-05-27T00:13:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-05-27T00:13:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-05-27T00:13:23.301 INFO:teuthology.orchestra.run.smithi204.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-27T00:13:23.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-05-27T00:13:23.405 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2024-05-27T00:13:23.607 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2024-05-27T00:13:23.607 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2024-05-27T00:13:23.673 INFO:teuthology.orchestra.run.smithi098.stdout:Preconfiguring packages ... 2024-05-27T00:13:23.792 INFO:teuthology.orchestra.run.smithi204.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-05-27T00:13:23.793 INFO:teuthology.orchestra.run.smithi204.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-05-27T00:13:23.793 INFO:teuthology.orchestra.run.smithi204.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-05-27T00:13:23.793 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages were automatically installed and are no longer required: 2024-05-27T00:13:23.793 INFO:teuthology.orchestra.run.smithi204.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-27T00:13:23.793 INFO:teuthology.orchestra.run.smithi204.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-27T00:13:23.794 INFO:teuthology.orchestra.run.smithi204.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-27T00:13:23.794 INFO:teuthology.orchestra.run.smithi204.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-27T00:13:23.794 INFO:teuthology.orchestra.run.smithi204.stdout: python2.7-minimal 2024-05-27T00:13:23.794 INFO:teuthology.orchestra.run.smithi204.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-27T00:13:23.795 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2024-05-27T00:13:23.795 INFO:teuthology.orchestra.run.smithi204.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-05-27T00:13:23.795 INFO:teuthology.orchestra.run.smithi204.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-05-27T00:13:23.795 INFO:teuthology.orchestra.run.smithi204.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-05-27T00:13:23.795 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-05-27T00:13:23.795 INFO:teuthology.orchestra.run.smithi204.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-05-27T00:13:23.796 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2024-05-27T00:13:23.796 INFO:teuthology.orchestra.run.smithi204.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-05-27T00:13:23.796 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-05-27T00:13:23.796 INFO:teuthology.orchestra.run.smithi204.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-05-27T00:13:23.819 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 55.7 MB in 5s (11.1 MB/s) 2024-05-27T00:13:23.838 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2024-05-27T00:13:23.838 INFO:teuthology.orchestra.run.smithi204.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-05-27T00:13:23.839 INFO:teuthology.orchestra.run.smithi204.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-05-27T00:13:23.839 INFO:teuthology.orchestra.run.smithi204.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-05-27T00:13:23.839 INFO:teuthology.orchestra.run.smithi204.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-05-27T00:13:23.839 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-05-27T00:13:23.839 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-05-27T00:13:23.839 INFO:teuthology.orchestra.run.smithi204.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-05-27T00:13:23.840 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages will be upgraded: 2024-05-27T00:13:23.840 INFO:teuthology.orchestra.run.smithi204.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-05-27T00:13:23.898 INFO:teuthology.orchestra.run.smithi204.stdout:4 upgraded, 39 newly installed, 0 to remove and 306 not upgraded. 2024-05-27T00:13:23.898 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 55.7 MB of archives. 2024-05-27T00:13:23.898 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 202 MB of additional disk space will be used. 2024-05-27T00:13:23.898 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-05-27T00:13:24.060 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-05-27T00:13:24.068 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-05-27T00:13:24.107 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140913 files and directories currently installed.) 2024-05-27T00:13:24.110 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-05-27T00:13:24.111 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-05-27T00:13:24.111 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T00:13:24.112 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-05-27T00:13:24.123 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-05-27T00:13:24.123 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-05-27T00:13:24.125 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-05-27T00:13:24.127 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-05-27T00:13:24.127 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-05-27T00:13:24.129 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-05-27T00:13:24.134 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-05-27T00:13:24.151 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-05-27T00:13:24.151 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-05-27T00:13:24.152 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-05-27T00:13:24.245 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T00:13:24.479 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-05-27T00:13:24.480 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-05-27T00:13:24.481 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-05-27T00:13:24.483 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-05-27T00:13:24.510 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-05-27T00:13:24.516 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-05-27T00:13:24.517 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-05-27T00:13:24.527 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-05-27T00:13:24.527 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-05-27T00:13:24.556 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-05-27T00:13:24.556 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-05-27T00:13:24.568 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-05-27T00:13:24.588 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-05-27T00:13:24.591 INFO:teuthology.orchestra.run.smithi204.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-05-27T00:13:24.592 INFO:teuthology.orchestra.run.smithi204.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-05-27T00:13:24.592 INFO:teuthology.orchestra.run.smithi204.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-05-27T00:13:24.593 INFO:teuthology.orchestra.run.smithi204.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-05-27T00:13:24.603 INFO:teuthology.orchestra.run.smithi204.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-05-27T00:13:24.604 INFO:teuthology.orchestra.run.smithi204.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-05-27T00:13:24.824 INFO:teuthology.orchestra.run.smithi204.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-05-27T00:13:24.824 INFO:teuthology.orchestra.run.smithi204.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-05-27T00:13:24.824 INFO:teuthology.orchestra.run.smithi204.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-05-27T00:13:24.824 INFO:teuthology.orchestra.run.smithi204.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-05-27T00:13:24.824 INFO:teuthology.orchestra.run.smithi204.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-05-27T00:13:24.824 INFO:teuthology.orchestra.run.smithi204.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-05-27T00:13:24.836 INFO:teuthology.orchestra.run.smithi204.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-05-27T00:13:24.836 INFO:teuthology.orchestra.run.smithi204.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-05-27T00:13:24.886 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-05-27T00:13:25.240 INFO:teuthology.orchestra.run.smithi204.stdout:Preconfiguring packages ... 2024-05-27T00:13:25.425 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 55.7 MB in 1s (56.3 MB/s) 2024-05-27T00:13:25.744 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138493 files and directories currently installed.) 2024-05-27T00:13:25.749 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T00:13:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T00:13:25.959 INFO:teuthology.orchestra.run.smithi098.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-05-27T00:13:26.208 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package flex. 2024-05-27T00:13:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140913 files and directories currently installed.) 2024-05-27T00:13:26.281 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-05-27T00:13:26.323 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking flex (2.6.4-8build2) ... 2024-05-27T00:13:26.548 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-05-27T00:13:26.851 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-05-27T00:13:27.029 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-05-27T00:13:27.436 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T00:13:27.532 INFO:teuthology.orchestra.run.smithi204.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-05-27T00:13:27.810 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package flex. 2024-05-27T00:13:27.900 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138493 files and directories currently installed.) 2024-05-27T00:13:27.904 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-05-27T00:13:27.950 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking flex (2.6.4-8build2) ... 2024-05-27T00:13:27.983 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140986 files and directories currently installed.) 2024-05-27T00:13:27.987 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T00:13:28.096 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T00:13:28.486 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-05-27T00:13:28.558 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T00:13:28.656 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-05-27T00:13:28.694 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T00:13:29.018 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-05-27T00:13:29.082 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T00:13:29.387 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package bison. 2024-05-27T00:13:29.465 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140986 files and directories currently installed.) 2024-05-27T00:13:29.469 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-05-27T00:13:29.539 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-05-27T00:13:29.640 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138566 files and directories currently installed.) 2024-05-27T00:13:29.644 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T00:13:29.773 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T00:13:29.957 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-05-27T00:13:29.972 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-05-27T00:13:30.015 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-05-27T00:13:30.317 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T00:13:30.335 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package dump. 2024-05-27T00:13:30.351 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-05-27T00:13:30.385 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking dump (0.4b47-3) ... 2024-05-27T00:13:30.631 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T00:13:30.697 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package indent. 2024-05-27T00:13:30.712 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:30.755 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-05-27T00:13:30.956 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-05-27T00:13:31.016 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-05-27T00:13:31.032 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-05-27T00:13:31.074 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-05-27T00:13:31.291 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package bison. 2024-05-27T00:13:31.378 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138566 files and directories currently installed.) 2024-05-27T00:13:31.383 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-05-27T00:13:31.387 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-05-27T00:13:31.402 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-05-27T00:13:31.427 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-05-27T00:13:31.445 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-05-27T00:13:31.852 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-05-27T00:13:31.874 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-05-27T00:13:31.935 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-05-27T00:13:31.948 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-05-27T00:13:31.963 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-05-27T00:13:32.006 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-05-27T00:13:32.272 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package dump. 2024-05-27T00:13:32.294 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-05-27T00:13:32.339 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking dump (0.4b47-3) ... 2024-05-27T00:13:32.344 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libfl2:amd64. 2024-05-27T00:13:32.359 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-05-27T00:13:32.394 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-05-27T00:13:32.634 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package indent. 2024-05-27T00:13:32.656 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:32.680 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-05-27T00:13:32.696 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-05-27T00:13:32.701 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-05-27T00:13:32.738 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-05-27T00:13:32.970 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-05-27T00:13:32.983 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-05-27T00:13:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjson-perl. 2024-05-27T00:13:33.007 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-05-27T00:13:33.020 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-05-27T00:13:33.049 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-05-27T00:13:33.301 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-05-27T00:13:33.323 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-05-27T00:13:33.337 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libllvm14:amd64. 2024-05-27T00:13:33.352 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-05-27T00:13:33.368 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-05-27T00:13:33.394 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-05-27T00:13:33.638 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-05-27T00:13:33.659 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-05-27T00:13:33.704 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-05-27T00:13:34.058 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libfl2:amd64. 2024-05-27T00:13:34.081 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-05-27T00:13:34.124 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-05-27T00:13:34.369 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-05-27T00:13:34.390 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-05-27T00:13:34.427 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-05-27T00:13:34.671 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjson-perl. 2024-05-27T00:13:34.684 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-05-27T00:13:34.721 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-05-27T00:13:34.747 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-05-27T00:13:34.763 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-05-27T00:13:34.806 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T00:13:35.041 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libllvm14:amd64. 2024-05-27T00:13:35.063 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-05-27T00:13:35.066 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libmbim-proxy. 2024-05-27T00:13:35.082 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-05-27T00:13:35.108 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-05-27T00:13:35.125 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T00:13:35.444 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-05-27T00:13:35.460 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-05-27T00:13:35.528 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T00:13:35.839 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libndp0:amd64. 2024-05-27T00:13:35.855 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-05-27T00:13:35.897 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-05-27T00:13:36.192 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libnm0:amd64. 2024-05-27T00:13:36.208 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-05-27T00:13:36.250 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-05-27T00:13:36.628 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libpq5:amd64. 2024-05-27T00:13:36.644 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:36.686 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:13:36.837 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-05-27T00:13:36.858 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-05-27T00:13:36.903 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T00:13:37.023 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-05-27T00:13:37.039 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:37.074 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T00:13:37.139 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libmbim-proxy. 2024-05-27T00:13:37.152 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-05-27T00:13:37.189 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T00:13:37.468 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqmi-proxy. 2024-05-27T00:13:37.483 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:37.517 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-05-27T00:13:37.535 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T00:13:37.539 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-05-27T00:13:37.583 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T00:13:37.779 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-05-27T00:13:37.795 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-05-27T00:13:37.838 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T00:13:37.945 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libndp0:amd64. 2024-05-27T00:13:37.967 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-05-27T00:13:38.011 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-05-27T00:13:38.323 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libnm0:amd64. 2024-05-27T00:13:38.345 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-05-27T00:13:38.350 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-05-27T00:13:38.364 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-05-27T00:13:38.389 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-05-27T00:13:38.408 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-05-27T00:13:38.743 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libpq5:amd64. 2024-05-27T00:13:38.753 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package modemmanager. 2024-05-27T00:13:38.764 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:38.768 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-05-27T00:13:38.803 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T00:13:38.809 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:13:39.221 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-05-27T00:13:39.242 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:39.295 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T00:13:39.435 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package network-manager. 2024-05-27T00:13:39.451 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-05-27T00:13:39.526 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-05-27T00:13:39.648 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libqmi-proxy. 2024-05-27T00:13:39.670 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:39.732 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T00:13:39.968 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-05-27T00:13:39.980 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-05-27T00:13:40.017 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T00:13:40.167 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ppp. 2024-05-27T00:13:40.183 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-05-27T00:13:40.544 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-05-27T00:13:40.571 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-05-27T00:13:40.593 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-05-27T00:13:40.638 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-05-27T00:13:40.881 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package pptp-linux. 2024-05-27T00:13:40.895 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-05-27T00:13:40.939 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-05-27T00:13:41.167 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package network-manager-pptp. 2024-05-27T00:13:41.176 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package modemmanager. 2024-05-27T00:13:41.181 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-05-27T00:13:41.198 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-05-27T00:13:41.216 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-05-27T00:13:41.242 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T00:13:41.528 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-client-common. 2024-05-27T00:13:41.543 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-05-27T00:13:41.586 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-client-common (238) ... 2024-05-27T00:13:41.874 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package network-manager. 2024-05-27T00:13:41.896 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-05-27T00:13:41.915 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-client-14. 2024-05-27T00:13:41.931 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:41.950 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-05-27T00:13:41.973 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:13:42.573 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ppp. 2024-05-27T00:13:42.596 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-05-27T00:13:42.934 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-05-27T00:13:42.978 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-common. 2024-05-27T00:13:42.994 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-05-27T00:13:43.046 INFO:teuthology.orchestra.run.smithi098.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-05-27T00:13:43.077 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-common (238) ... 2024-05-27T00:13:43.270 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package pptp-linux. 2024-05-27T00:13:43.294 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-05-27T00:13:43.345 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-05-27T00:13:43.519 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-14. 2024-05-27T00:13:43.534 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:43.580 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:13:43.630 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package network-manager-pptp. 2024-05-27T00:13:43.644 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-05-27T00:13:43.689 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-05-27T00:13:44.067 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package postgresql-client-common. 2024-05-27T00:13:44.089 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-05-27T00:13:44.142 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking postgresql-client-common (238) ... 2024-05-27T00:13:44.479 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package postgresql-client-14. 2024-05-27T00:13:44.501 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:44.554 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:13:44.730 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql. 2024-05-27T00:13:44.745 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-05-27T00:13:44.780 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql (14+238) ... 2024-05-27T00:13:44.991 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-client. 2024-05-27T00:13:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-05-27T00:13:45.049 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-client (14+238) ... 2024-05-27T00:13:45.193 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package postgresql-common. 2024-05-27T00:13:45.215 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-05-27T00:13:45.270 INFO:teuthology.orchestra.run.smithi204.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-05-27T00:13:45.293 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package postgresql-contrib. 2024-05-27T00:13:45.299 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking postgresql-common (238) ... 2024-05-27T00:13:45.307 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-05-27T00:13:45.351 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking postgresql-contrib (14+238) ... 2024-05-27T00:13:45.695 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package tcl8.6. 2024-05-27T00:13:45.710 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-05-27T00:13:45.753 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-05-27T00:13:45.758 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package postgresql-14. 2024-05-27T00:13:45.780 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T00:13:45.820 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:13:46.023 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package tcl. 2024-05-27T00:13:46.037 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-05-27T00:13:46.077 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-05-27T00:13:46.434 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package wpasupplicant. 2024-05-27T00:13:46.449 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-05-27T00:13:46.492 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-05-27T00:13:46.880 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-05-27T00:13:46.895 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-05-27T00:13:46.938 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-05-27T00:13:47.319 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package usb-modeswitch. 2024-05-27T00:13:47.334 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-05-27T00:13:47.385 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-05-27T00:13:47.412 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package postgresql. 2024-05-27T00:13:47.435 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-05-27T00:13:47.487 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking postgresql (14+238) ... 2024-05-27T00:13:47.688 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-client-common (238) ... 2024-05-27T00:13:47.748 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package postgresql-client. 2024-05-27T00:13:47.770 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-05-27T00:13:47.814 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking postgresql-client (14+238) ... 2024-05-27T00:13:48.125 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package postgresql-contrib. 2024-05-27T00:13:48.138 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-05-27T00:13:48.183 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking postgresql-contrib (14+238) ... 2024-05-27T00:13:48.225 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up flex (2.6.4-8build2) ... 2024-05-27T00:13:48.359 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-05-27T00:13:48.485 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:13:48.494 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package tcl8.6. 2024-05-27T00:13:48.517 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-05-27T00:13:48.561 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-05-27T00:13:48.637 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-05-27T00:13:48.763 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-05-27T00:13:48.847 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package tcl. 2024-05-27T00:13:48.869 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-05-27T00:13:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:13:48.919 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-05-27T00:13:49.300 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package wpasupplicant. 2024-05-27T00:13:49.323 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-05-27T00:13:49.375 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-05-27T00:13:49.830 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-05-27T00:13:49.830 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-05-27T00:13:49.852 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-05-27T00:13:49.904 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-05-27T00:13:49.928 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-05-27T00:13:50.307 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-05-27T00:13:50.308 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-05-27T00:13:50.349 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package usb-modeswitch. 2024-05-27T00:13:50.371 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-05-27T00:13:50.424 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-05-27T00:13:50.734 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up postgresql-client-common (238) ... 2024-05-27T00:13:50.947 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up flex (2.6.4-8build2) ... 2024-05-27T00:13:51.039 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-05-27T00:13:51.146 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-05-27T00:13:51.157 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-05-27T00:13:51.275 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T00:13:51.383 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-05-27T00:13:51.510 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-05-27T00:13:51.761 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-05-27T00:13:51.825 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:13:51.887 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-05-27T00:13:51.934 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-05-27T00:13:52.013 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-05-27T00:13:52.068 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-05-27T00:13:52.194 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:13:52.677 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-05-27T00:13:52.726 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-05-27T00:13:52.803 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up dump (0.4b47-3) ... 2024-05-27T00:13:52.817 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-05-27T00:13:52.898 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-05-27T00:13:52.962 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T00:13:53.088 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-05-27T00:13:53.172 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-05-27T00:13:53.172 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-05-27T00:13:53.214 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up tcl (8.6.11+1build2) ... 2024-05-27T00:13:53.340 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjson-perl (4.04000-1) ... 2024-05-27T00:13:53.466 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-05-27T00:13:53.589 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-05-27T00:13:53.660 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-05-27T00:13:53.711 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-05-27T00:13:53.786 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-05-27T00:13:53.837 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-05-27T00:13:53.962 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-05-27T00:13:53.963 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T00:13:54.088 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T00:13:54.089 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-05-27T00:13:54.189 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-05-27T00:13:54.215 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-05-27T00:13:54.357 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-client (14+238) ... 2024-05-27T00:13:54.367 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-05-27T00:13:54.483 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-05-27T00:13:54.526 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-05-27T00:13:54.769 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-05-27T00:13:55.126 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:13:55.126 INFO:teuthology.orchestra.run.smithi098.stdout:The following network interfaces were found in /etc/network/interfaces 2024-05-27T00:13:55.126 INFO:teuthology.orchestra.run.smithi098.stdout:which means they are currently configured by ifupdown: 2024-05-27T00:13:55.126 INFO:teuthology.orchestra.run.smithi098.stdout:- enp3s0f1 2024-05-27T00:13:55.126 INFO:teuthology.orchestra.run.smithi098.stdout:If you want to manage those interfaces with NetworkManager instead 2024-05-27T00:13:55.126 INFO:teuthology.orchestra.run.smithi098.stdout:remove their configuration from /etc/network/interfaces. 2024-05-27T00:13:55.127 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:13:55.259 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-05-27T00:13:55.591 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-05-27T00:13:55.687 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-05-27T00:13:55.734 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up dump (0.4b47-3) ... 2024-05-27T00:13:55.844 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-05-27T00:13:55.919 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T00:13:56.070 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-05-27T00:13:56.119 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-05-27T00:13:56.221 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up tcl (8.6.11+1build2) ... 2024-05-27T00:13:56.372 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjson-perl (4.04000-1) ... 2024-05-27T00:13:56.525 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-05-27T00:13:56.623 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-05-27T00:13:56.700 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-05-27T00:13:56.811 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T00:13:56.843 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-05-27T00:13:56.962 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-05-27T00:13:57.088 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-05-27T00:13:57.189 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-common (238) ... 2024-05-27T00:13:57.461 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-05-27T00:13:57.612 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T00:13:57.763 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-05-27T00:13:57.965 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up postgresql-client (14+238) ... 2024-05-27T00:13:58.107 INFO:teuthology.orchestra.run.smithi098.stdout:Adding user postgres to group ssl-cert 2024-05-27T00:13:58.108 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-05-27T00:13:58.360 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:13:58.360 INFO:teuthology.orchestra.run.smithi098.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-05-27T00:13:58.611 INFO:teuthology.orchestra.run.smithi098.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-05-27T00:13:58.611 INFO:teuthology.orchestra.run.smithi098.stdout:Removing obsolete dictionary files: 2024-05-27T00:13:58.639 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:13:58.639 INFO:teuthology.orchestra.run.smithi204.stdout:The following network interfaces were found in /etc/network/interfaces 2024-05-27T00:13:58.639 INFO:teuthology.orchestra.run.smithi204.stdout:which means they are currently configured by ifupdown: 2024-05-27T00:13:58.639 INFO:teuthology.orchestra.run.smithi204.stdout:- enp3s0f1 2024-05-27T00:13:58.639 INFO:teuthology.orchestra.run.smithi204.stdout:If you want to manage those interfaces with NetworkManager instead 2024-05-27T00:13:58.640 INFO:teuthology.orchestra.run.smithi204.stdout:remove their configuration from /etc/network/interfaces. 2024-05-27T00:13:58.640 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:13:58.754 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-05-27T00:13:59.104 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-05-27T00:13:59.332 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-05-27T00:13:59.465 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-05-27T00:13:59.815 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:14:00.027 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T00:14:00.141 INFO:teuthology.orchestra.run.smithi098.stdout:Creating new PostgreSQL cluster 14/main ... 2024-05-27T00:14:00.155 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-05-27T00:14:00.178 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-05-27T00:14:00.211 INFO:teuthology.orchestra.run.smithi098.stdout:The files belonging to this database system will be owned by user "postgres". 2024-05-27T00:14:00.211 INFO:teuthology.orchestra.run.smithi098.stdout:This user must also own the server process. 2024-05-27T00:14:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:00.211 INFO:teuthology.orchestra.run.smithi098.stdout:The database cluster will be initialized with locale "en_US". 2024-05-27T00:14:00.211 INFO:teuthology.orchestra.run.smithi098.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-05-27T00:14:00.211 INFO:teuthology.orchestra.run.smithi098.stdout:The default text search configuration will be set to "english". 2024-05-27T00:14:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:00.212 INFO:teuthology.orchestra.run.smithi098.stdout:Data page checksums are disabled. 2024-05-27T00:14:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:00.212 INFO:teuthology.orchestra.run.smithi098.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-05-27T00:14:00.212 INFO:teuthology.orchestra.run.smithi098.stdout:creating subdirectories ... ok 2024-05-27T00:14:00.212 INFO:teuthology.orchestra.run.smithi098.stdout:selecting dynamic shared memory implementation ... posix 2024-05-27T00:14:00.236 INFO:teuthology.orchestra.run.smithi098.stdout:selecting default max_connections ... 100 2024-05-27T00:14:00.269 INFO:teuthology.orchestra.run.smithi098.stdout:selecting default shared_buffers ... 128MB 2024-05-27T00:14:00.285 INFO:teuthology.orchestra.run.smithi098.stdout:selecting default time zone ... Etc/UTC 2024-05-27T00:14:00.285 INFO:teuthology.orchestra.run.smithi098.stdout:creating configuration files ... ok 2024-05-27T00:14:00.305 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-05-27T00:14:00.431 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up postgresql-common (238) ... 2024-05-27T00:14:00.453 INFO:teuthology.orchestra.run.smithi098.stdout:running bootstrap script ... ok 2024-05-27T00:14:01.109 INFO:teuthology.orchestra.run.smithi098.stdout:performing post-bootstrap initialization ... ok 2024-05-27T00:14:01.367 INFO:teuthology.orchestra.run.smithi204.stdout:Adding user postgres to group ssl-cert 2024-05-27T00:14:01.555 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:01.555 INFO:teuthology.orchestra.run.smithi204.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-05-27T00:14:01.665 INFO:teuthology.orchestra.run.smithi098.stdout:syncing data to disk ... ok 2024-05-27T00:14:01.722 INFO:teuthology.orchestra.run.smithi204.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-05-27T00:14:01.722 INFO:teuthology.orchestra.run.smithi204.stdout:Removing obsolete dictionary files: 2024-05-27T00:14:02.212 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-05-27T00:14:02.309 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-05-27T00:14:02.733 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T00:14:03.060 INFO:teuthology.orchestra.run.smithi204.stdout:Creating new PostgreSQL cluster 14/main ... 2024-05-27T00:14:03.083 INFO:teuthology.orchestra.run.smithi204.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-05-27T00:14:03.137 INFO:teuthology.orchestra.run.smithi204.stdout:The files belonging to this database system will be owned by user "postgres". 2024-05-27T00:14:03.137 INFO:teuthology.orchestra.run.smithi204.stdout:This user must also own the server process. 2024-05-27T00:14:03.137 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:03.138 INFO:teuthology.orchestra.run.smithi204.stdout:The database cluster will be initialized with locale "en_US". 2024-05-27T00:14:03.138 INFO:teuthology.orchestra.run.smithi204.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-05-27T00:14:03.138 INFO:teuthology.orchestra.run.smithi204.stdout:The default text search configuration will be set to "english". 2024-05-27T00:14:03.138 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:03.138 INFO:teuthology.orchestra.run.smithi204.stdout:Data page checksums are disabled. 2024-05-27T00:14:03.138 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:03.138 INFO:teuthology.orchestra.run.smithi204.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-05-27T00:14:03.138 INFO:teuthology.orchestra.run.smithi204.stdout:creating subdirectories ... ok 2024-05-27T00:14:03.138 INFO:teuthology.orchestra.run.smithi204.stdout:selecting dynamic shared memory implementation ... posix 2024-05-27T00:14:03.153 INFO:teuthology.orchestra.run.smithi204.stdout:selecting default max_connections ... 100 2024-05-27T00:14:03.173 INFO:teuthology.orchestra.run.smithi204.stdout:selecting default shared_buffers ... 128MB 2024-05-27T00:14:03.183 INFO:teuthology.orchestra.run.smithi204.stdout:selecting default time zone ... Etc/UTC 2024-05-27T00:14:03.184 INFO:teuthology.orchestra.run.smithi204.stdout:creating configuration files ... ok 2024-05-27T00:14:03.297 INFO:teuthology.orchestra.run.smithi204.stdout:running bootstrap script ... ok 2024-05-27T00:14:03.984 INFO:teuthology.orchestra.run.smithi204.stdout:performing post-bootstrap initialization ... ok 2024-05-27T00:14:04.737 INFO:teuthology.orchestra.run.smithi204.stdout:syncing data to disk ... ok 2024-05-27T00:14:05.141 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-05-27T00:14:05.240 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T00:14:05.365 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql-contrib (14+238) ... 2024-05-27T00:14:05.533 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up postgresql (14+238) ... 2024-05-27T00:14:05.668 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T00:14:05.794 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T00:14:06.056 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-05-27T00:14:06.056 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-05-27T00:14:06.846 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-27T00:14:06.947 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-05-27T00:14:07.274 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-27T00:14:07.455 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-27T00:14:07.981 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T00:14:08.107 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up postgresql-contrib (14+238) ... 2024-05-27T00:14:08.233 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up postgresql (14+238) ... 2024-05-27T00:14:08.359 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T00:14:08.502 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T00:14:08.755 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-05-27T00:14:08.755 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-05-27T00:14:08.896 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-05-27T00:14:08.996 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-27T00:14:09.347 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-27T00:14:09.439 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-05-27T00:14:09.524 INFO:teuthology.orchestra.run.smithi098.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-27T00:14:09.532 DEBUG:teuthology.parallel:result is None 2024-05-27T00:14:09.733 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-27T00:14:09.885 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-27T00:14:11.405 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-05-27T00:14:11.505 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-27T00:14:12.008 INFO:teuthology.orchestra.run.smithi204.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-27T00:14:12.012 DEBUG:teuthology.parallel:result is None 2024-05-27T00:14:12.013 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0ef46a246e1118e5adafa1857b4189e7d780c816 2024-05-27T00:14:12.246 DEBUG:teuthology.orchestra.run.smithi098:> dpkg-query -W -f '${Version}' ceph 2024-05-27T00:14:12.269 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-2857-g0ef46a24-1jammy 2024-05-27T00:14:12.270 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2857-g0ef46a24-1jammy 2024-05-27T00:14:12.270 INFO:teuthology.task.install:The correct ceph version 19.0.0-2857-g0ef46a24-1jammy is installed. 2024-05-27T00:14:12.272 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0ef46a246e1118e5adafa1857b4189e7d780c816 2024-05-27T00:14:12.500 DEBUG:teuthology.orchestra.run.smithi204:> dpkg-query -W -f '${Version}' ceph 2024-05-27T00:14:12.519 INFO:teuthology.orchestra.run.smithi204.stdout:19.0.0-2857-g0ef46a24-1jammy 2024-05-27T00:14:12.520 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2857-g0ef46a24-1jammy 2024-05-27T00:14:12.520 INFO:teuthology.task.install:The correct ceph version 19.0.0-2857-g0ef46a24-1jammy is installed. 2024-05-27T00:14:12.521 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-27T00:14:12.522 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:12.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T00:14:12.543 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:12.544 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T00:14:12.575 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-27T00:14:12.575 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:12.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T00:14:12.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T00:14:12.674 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:12.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T00:14:12.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T00:14:12.744 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-27T00:14:12.744 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:12.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T00:14:12.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T00:14:12.829 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:12.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T00:14:12.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T00:14:12.896 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-27T00:14:12.896 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:12.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T00:14:12.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T00:14:12.981 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:12.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T00:14:12.994 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T00:14:13.048 INFO:teuthology.run_tasks:Running task ceph... 2024-05-27T00:14:13.123 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-27T00:14:13.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-05-27T00:14:13.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2024-05-27T00:14:13.144 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-27T00:14:13.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-27T00:14:13.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-27T00:14:13.205 INFO:tasks.ceph:Creating extra log directories... 2024-05-27T00:14:13.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-27T00:14:13.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-27T00:14:13.276 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-05-27T00:14:13.276 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-05-27T00:14:13.276 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-05-27T00:14:13.276 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:13.276 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-27T00:14:13.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-27T00:14:13.414 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:13.415 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-27T00:14:13.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-27T00:14:13.498 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-27T00:14:13.498 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-27T00:14:13.498 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174', 'branch': 'squid', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7727174', 'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': '0ef46a246e1118e5adafa1857b4189e7d780c816'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '0ef46a246e1118e5adafa1857b4189e7d780c816'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '7d7cb32d1d130135a98121a564d709025c2154b6'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 1555, 'sha1': '0ef46a246e1118e5adafa1857b4189e7d780c816', 'sleep_before_teardown': 0, 'subset': '10/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7d7cb32d1d130135a98121a564d709025c2154b6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7d7cb32d1d130135a98121a564d709025c2154b6', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9za9Kcz6urGNPKFQx9cMZxEB85f2qJjQnWlSb2kKHd8vMx5iWcIKGPxR5jZS/OJovet3O2cVUb4NJCurn5HGA=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJK5fupUbTcvOxwVRSb9OBYO5yahwCCHNCt1CPD2Z40ka7gsBnHd+cfgmxqx6HDJiC7pxd1KDBLKvVozEtBX8K0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '0ef46a246e1118e5adafa1857b4189e7d780c816'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds'], 'sha1': '0ef46a246e1118e5adafa1857b4189e7d780c816', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e7ae7580c417d6ab27fda98fecadca5d9f85d63e', 'timestamp': '2024-05-26_21:24:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-05-27T00:14:13.499 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-27T00:14:13.502 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-27T00:14:13.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-05-27T00:14:13.552 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2024-05-27T00:14:13.596 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:13.596 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T00:14:13.622 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T00:14:13.622 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-05-27T00:14:13.674 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T00:14:13.675 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T00:14:13.675 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 884 Links: 1 2024-05-27T00:14:13.675 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T00:14:13.675 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-05-27 00:07:47.099241683 +0000 2024-05-27T00:14:13.675 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-05-27 00:07:46.803244319 +0000 2024-05-27T00:14:13.675 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-05-27 00:07:46.803244319 +0000 2024-05-27T00:14:13.675 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-05-27 00:07:46.803244319 +0000 2024-05-27T00:14:13.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T00:14:13.740 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-05-27T00:14:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-05-27T00:14:13.741 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000403257 s, 1.3 MB/s 2024-05-27T00:14:13.743 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T00:14:13.796 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-05-27T00:14:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T00:14:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T00:14:13.846 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 903 Links: 1 2024-05-27T00:14:13.846 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T00:14:13.847 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-05-27 00:07:47.659236692 +0000 2024-05-27T00:14:13.847 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-05-27 00:07:47.387239117 +0000 2024-05-27T00:14:13.847 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-05-27 00:07:47.387239117 +0000 2024-05-27T00:14:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-05-27 00:07:47.387239117 +0000 2024-05-27T00:14:13.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T00:14:13.908 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-05-27T00:14:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-05-27T00:14:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000387989 s, 1.3 MB/s 2024-05-27T00:14:13.911 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T00:14:13.964 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-05-27T00:14:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T00:14:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T00:14:14.018 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 918 Links: 1 2024-05-27T00:14:14.018 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T00:14:14.018 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-05-27 00:07:48.275231205 +0000 2024-05-27T00:14:14.018 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-05-27 00:07:48.003233628 +0000 2024-05-27T00:14:14.018 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-05-27 00:07:48.003233628 +0000 2024-05-27T00:14:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-05-27 00:07:48.003233628 +0000 2024-05-27T00:14:14.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T00:14:14.084 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-05-27T00:14:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-05-27T00:14:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000648127 s, 790 kB/s 2024-05-27T00:14:14.086 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T00:14:14.144 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-05-27T00:14:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T00:14:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T00:14:14.198 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 936 Links: 1 2024-05-27T00:14:14.198 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T00:14:14.198 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-05-27 00:07:48.871225894 +0000 2024-05-27T00:14:14.198 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-05-27 00:07:48.591228390 +0000 2024-05-27T00:14:14.198 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-05-27 00:07:48.591228390 +0000 2024-05-27T00:14:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-05-27 00:07:48.591228390 +0000 2024-05-27T00:14:14.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T00:14:14.264 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-05-27T00:14:14.264 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-05-27T00:14:14.264 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000398571 s, 1.3 MB/s 2024-05-27T00:14:14.266 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T00:14:14.320 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-27T00:14:14.320 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:14.320 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T00:14:14.326 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T00:14:14.326 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-05-27T00:14:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T00:14:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T00:14:14.374 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-05-27T00:14:14.374 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T00:14:14.375 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-05-27 00:07:30.375633152 +0000 2024-05-27T00:14:14.375 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-05-27 00:07:30.103640207 +0000 2024-05-27T00:14:14.375 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-05-27 00:07:30.103640207 +0000 2024-05-27T00:14:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-05-27T00:14:14.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T00:14:14.429 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-05-27T00:14:14.429 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-05-27T00:14:14.429 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000592669 s, 864 kB/s 2024-05-27T00:14:14.430 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T00:14:14.479 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-05-27T00:14:14.525 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T00:14:14.525 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T00:14:14.526 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 901 Links: 1 2024-05-27T00:14:14.526 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T00:14:14.526 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-05-27 00:07:30.855620705 +0000 2024-05-27T00:14:14.526 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-05-27 00:07:30.595627447 +0000 2024-05-27T00:14:14.526 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-05-27 00:07:30.595627447 +0000 2024-05-27T00:14:14.526 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-05-27T00:14:14.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T00:14:14.580 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-05-27T00:14:14.580 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-05-27T00:14:14.580 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000494977 s, 1.0 MB/s 2024-05-27T00:14:14.581 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T00:14:14.631 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-05-27T00:14:14.678 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T00:14:14.678 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T00:14:14.678 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 919 Links: 1 2024-05-27T00:14:14.678 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T00:14:14.678 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-05-27 00:07:31.351607842 +0000 2024-05-27T00:14:14.678 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-05-27 00:07:31.099614377 +0000 2024-05-27T00:14:14.678 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-05-27 00:07:31.099614377 +0000 2024-05-27T00:14:14.678 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-05-27T00:14:14.679 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T00:14:14.732 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-05-27T00:14:14.732 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-05-27T00:14:14.732 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000473145 s, 1.1 MB/s 2024-05-27T00:14:14.733 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T00:14:14.783 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-05-27T00:14:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T00:14:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T00:14:14.829 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 932 Links: 1 2024-05-27T00:14:14.829 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T00:14:14.829 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-05-27 00:07:31.567602242 +0000 2024-05-27T00:14:14.830 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-05-27 00:07:31.563602345 +0000 2024-05-27T00:14:14.830 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-05-27 00:07:31.563602345 +0000 2024-05-27T00:14:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-05-27T00:14:14.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T00:14:14.884 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-05-27T00:14:14.884 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-05-27T00:14:14.884 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000395993 s, 1.3 MB/s 2024-05-27T00:14:14.885 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T00:14:14.935 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-27T00:14:14.935 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-27T00:14:14.935 INFO:tasks.ceph:Generating config... 2024-05-27T00:14:14.937 INFO:tasks.ceph:[client] client mount timeout = 600 2024-05-27T00:14:14.937 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-05-27T00:14:14.937 INFO:tasks.ceph:[client] debug client = 20 2024-05-27T00:14:14.937 INFO:tasks.ceph:[client] debug ms = 1 2024-05-27T00:14:14.937 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-05-27T00:14:14.937 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-05-27T00:14:14.937 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-05-27T00:14:14.937 INFO:tasks.ceph:[mds] debug mds = 20 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] debug ms = 1 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] mds debug frag = True 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-05-27T00:14:14.938 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mgr] debug client = 20 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-27T00:14:14.939 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-05-27T00:14:14.940 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-05-27T00:14:14.940 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-27T00:14:14.940 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-27T00:14:14.940 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-05-27T00:14:14.940 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-27T00:14:14.940 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-27T00:14:14.940 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-27T00:14:14.940 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-27T00:14:14.940 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-27T00:14:14.940 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-27T00:14:14.940 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-27T00:14:14.941 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-27T00:14:14.941 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-27T00:14:14.941 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-27T00:14:14.941 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-05-27T00:14:14.941 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-27T00:14:14.941 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-05-27T00:14:14.941 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-27T00:14:14.941 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-05-27T00:14:14.941 INFO:tasks.ceph:Setting up mon.a... 2024-05-27T00:14:14.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-27T00:14:14.987 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.keyring 2024-05-27T00:14:14.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-27T00:14:15.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-27T00:14:15.104 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.98'), ('mon.b', '172.21.15.204'), ('mon.c', '[v2:172.21.15.204:3301,v1:172.21.15.204:6790]')] 2024-05-27T00:14:15.105 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.98,172.21.15.204,[v2:172.21.15.204:3301,v1:172.21.15.204:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-05-27T00:14:15.106 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:15.107 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-27T00:14:15.153 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.98 --add b 172.21.15.204 --addv c '[v2:172.21.15.204:3301,v1:172.21.15.204:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:15.242 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:15.242 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid f6d0c69d-6040-4984-af90-0cab9df1ba47 2024-05-27T00:14:15.243 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-05-27T00:14:15.243 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2024-05-27T00:14:15.243 INFO:teuthology.orchestra.run.smithi098.stdout:fsid f6d0c69d-6040-4984-af90-0cab9df1ba47 2024-05-27T00:14:15.243 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2024-05-27T00:14:15.242767+0000 2024-05-27T00:14:15.243 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-05-27T00:14:15.242767+0000 2024-05-27T00:14:15.243 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 17 (quincy) 2024-05-27T00:14:15.243 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 1 2024-05-27T00:14:15.243 INFO:teuthology.orchestra.run.smithi098.stdout:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-05-27T00:14:15.244 INFO:teuthology.orchestra.run.smithi098.stdout:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-05-27T00:14:15.244 INFO:teuthology.orchestra.run.smithi098.stdout:2: [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] mon.c 2024-05-27T00:14:15.244 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-27T00:14:15.245 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-27T00:14:15.253 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f6d0c69d-6040-4984-af90-0cab9df1ba47... 2024-05-27T00:14:15.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-27T00:14:15.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-27T00:14:15.331 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2024-05-27T00:14:15.331 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2024-05-27T00:14:15.331 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-27T00:14:15.331 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2024-05-27T00:14:15.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-27T00:14:15.333 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2024-05-27T00:14:15.333 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.333 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2024-05-27T00:14:15.333 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-27T00:14:15.333 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.333 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2024-05-27T00:14:15.334 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2024-05-27T00:14:15.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2024-05-27T00:14:15.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2024-05-27T00:14:15.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2024-05-27T00:14:15.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2024-05-27T00:14:15.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2024-05-27T00:14:15.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2024-05-27T00:14:15.334 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2024-05-27T00:14:15.335 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = "172.21.15.98,172.21.15.204,[v2:172.21.15.204:3301,v1:172.21.15.204:6790]" 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on skipped message = False 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = f6d0c69d-6040-4984-af90-0cab9df1ba47 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.336 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug trim objects = true 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.337 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2024-05-27T00:14:15.338 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.338 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2024-05-27T00:14:15.338 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2024-05-27T00:14:15.338 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.338 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2024-05-27T00:14:15.338 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2024-05-27T00:14:15.338 INFO:teuthology.orchestra.run.smithi204.stdout: bdev async discard = True 2024-05-27T00:14:15.338 INFO:teuthology.orchestra.run.smithi204.stdout: bdev enable discard = True 2024-05-27T00:14:15.338 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore allocator = bitmap 2024-05-27T00:14:15.339 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore block size = 96636764160 2024-05-27T00:14:15.339 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore fsck on mount = True 2024-05-27T00:14:15.339 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluefs = 1/20 2024-05-27T00:14:15.339 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluestore = 1/20 2024-05-27T00:14:15.339 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-05-27T00:14:15.339 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2024-05-27T00:14:15.339 INFO:teuthology.orchestra.run.smithi204.stdout: debug rocksdb = 4/10 2024-05-27T00:14:15.339 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd backfillfull_ratio = 0.85 2024-05-27T00:14:15.339 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd full ratio = 0.9 2024-05-27T00:14:15.339 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd nearfull ratio = 0.8 2024-05-27T00:14:15.340 INFO:teuthology.orchestra.run.smithi204.stdout: osd blocked scrub grace period = 3600 2024-05-27T00:14:15.340 INFO:teuthology.orchestra.run.smithi204.stdout: osd failsafe full ratio = 0.95 2024-05-27T00:14:15.340 INFO:teuthology.orchestra.run.smithi204.stdout: osd max markdown count = 1000 2024-05-27T00:14:15.340 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = bluestore 2024-05-27T00:14:15.340 INFO:teuthology.orchestra.run.smithi204.stdout: osd op complaint time = 180 2024-05-27T00:14:15.340 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.340 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2024-05-27T00:14:15.340 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-05-27T00:14:15.340 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2024-05-27T00:14:15.340 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: debug client = 20 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-05-27T00:14:15.341 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.342 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2024-05-27T00:14:15.343 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-27T00:14:15.343 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-27T00:14:15.343 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.343 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2024-05-27T00:14:15.343 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2024-05-27T00:14:15.343 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.343 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2024-05-27T00:14:15.343 INFO:teuthology.orchestra.run.smithi204.stdout: mon down mkfs grace = 300 2024-05-27T00:14:15.343 INFO:teuthology.orchestra.run.smithi204.stdout: mon op complaint time = 120 2024-05-27T00:14:15.343 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-27T00:14:15.344 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2024-05-27T00:14:15.344 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2024-05-27T00:14:15.344 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2024-05-27T00:14:15.344 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2024-05-27T00:14:15.344 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-27T00:14:15.344 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-27T00:14:15.344 INFO:teuthology.orchestra.run.smithi204.stdout: client mount timeout = 600 2024-05-27T00:14:15.344 INFO:teuthology.orchestra.run.smithi204.stdout: client_shutdown_timeout = 120 2024-05-27T00:14:15.344 INFO:teuthology.orchestra.run.smithi204.stdout: debug client = 20 2024-05-27T00:14:15.344 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout: rados mon op timeout = 900 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout: rados osd op timeout = 900 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.b] 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.c] 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.a] 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.c] 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.e] 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.b] 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.d] 2024-05-27T00:14:15.345 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.f] 2024-05-27T00:14:15.346 INFO:teuthology.orchestra.run.smithi204.stdout:[mds] 2024-05-27T00:14:15.346 INFO:teuthology.orchestra.run.smithi204.stdout: debug mds = 20 2024-05-27T00:14:15.346 INFO:teuthology.orchestra.run.smithi204.stdout: debug mds balancer = 20 2024-05-27T00:14:15.346 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-05-27T00:14:15.346 INFO:teuthology.orchestra.run.smithi204.stdout: mds bal fragment size max = 10000 2024-05-27T00:14:15.346 INFO:teuthology.orchestra.run.smithi204.stdout: mds bal merge size = 5 2024-05-27T00:14:15.346 INFO:teuthology.orchestra.run.smithi204.stdout: mds bal split bits = 3 2024-05-27T00:14:15.346 INFO:teuthology.orchestra.run.smithi204.stdout: mds bal split size = 100 2024-05-27T00:14:15.346 INFO:teuthology.orchestra.run.smithi204.stdout: mds debug frag = True 2024-05-27T00:14:15.346 INFO:teuthology.orchestra.run.smithi204.stdout: mds debug scatterstat = True 2024-05-27T00:14:15.347 INFO:teuthology.orchestra.run.smithi204.stdout: mds oft prefetch dirfrags = True 2024-05-27T00:14:15.347 INFO:teuthology.orchestra.run.smithi204.stdout: mds op complaint time = 180 2024-05-27T00:14:15.347 INFO:teuthology.orchestra.run.smithi204.stdout: mds verify scatter = True 2024-05-27T00:14:15.347 INFO:teuthology.orchestra.run.smithi204.stdout: mds_dir_prefetch = True 2024-05-27T00:14:15.347 INFO:teuthology.orchestra.run.smithi204.stdout: mds_sleep_rank_change = 5000000.0 2024-05-27T00:14:15.347 INFO:teuthology.orchestra.run.smithi204.stdout: osd op complaint time = 180 2024-05-27T00:14:15.347 INFO:teuthology.orchestra.run.smithi204.stdout: rados mon op timeout = 900 2024-05-27T00:14:15.347 INFO:teuthology.orchestra.run.smithi204.stdout: rados osd op timeout = 900 2024-05-27T00:14:15.348 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-05-27T00:14:15.348 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-05-27T00:14:15.348 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-27T00:14:15.348 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-05-27T00:14:15.348 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.348 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-05-27T00:14:15.348 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.348 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-05-27T00:14:15.348 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-27T00:14:15.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-27T00:14:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-05-27T00:14:15.351 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "172.21.15.98,172.21.15.204,[v2:172.21.15.204:3301,v1:172.21.15.204:6790]" 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on skipped message = False 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = f6d0c69d-6040-4984-af90-0cab9df1ba47 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-05-27T00:14:15.352 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.353 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-05-27T00:14:15.353 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-05-27T00:14:15.353 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-05-27T00:14:15.353 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-05-27T00:14:15.353 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.353 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-05-27T00:14:15.353 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-05-27T00:14:15.353 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.353 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: bdev async discard = True 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: bdev enable discard = True 2024-05-27T00:14:15.354 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore allocator = bitmap 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 1/20 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 1/20 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 4/10 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-05-27T00:14:15.355 INFO:teuthology.orchestra.run.smithi098.stdout: osd blocked scrub grace period = 3600 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout: osd max markdown count = 1000 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-05-27T00:14:15.356 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: debug client = 20 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-05-27T00:14:15.357 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.358 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout: mon down mkfs grace = 300 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout: mon op complaint time = 120 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-05-27T00:14:15.359 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout: client mount timeout = 600 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout: client_shutdown_timeout = 120 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout: debug client = 20 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2024-05-27T00:14:15.360 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e] 2024-05-27T00:14:15.361 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2024-05-27T00:14:15.361 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2024-05-27T00:14:15.361 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f] 2024-05-27T00:14:15.361 INFO:teuthology.orchestra.run.smithi098.stdout:[mds] 2024-05-27T00:14:15.361 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds = 20 2024-05-27T00:14:15.361 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds balancer = 20 2024-05-27T00:14:15.361 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-05-27T00:14:15.361 INFO:teuthology.orchestra.run.smithi098.stdout: mds bal fragment size max = 10000 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: mds bal merge size = 5 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: mds bal split bits = 3 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: mds bal split size = 100 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug frag = True 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug scatterstat = True 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: mds oft prefetch dirfrags = True 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: mds op complaint time = 180 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: mds verify scatter = True 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: mds_dir_prefetch = True 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: mds_sleep_rank_change = 5000000.0 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2024-05-27T00:14:15.362 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2024-05-27T00:14:15.363 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2024-05-27T00:14:15.364 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-27T00:14:15.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-27T00:14:15.462 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-27T00:14:15.462 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:15.462 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-27T00:14:15.471 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:15.471 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-27T00:14:15.525 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:14:15.526 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:15.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-27T00:14:15.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-27T00:14:15.600 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:15.601 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:15.649 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:14:15.649 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:15.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-27T00:14:15.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-27T00:14:15.675 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:15.675 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:15.721 INFO:tasks.ceph:Setting up mon nodes... 2024-05-27T00:14:15.722 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-27T00:14:15.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-27T00:14:15.776 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-27T00:14:15.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-27T00:14:15.825 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-27T00:14:15.839 INFO:tasks.ceph:Setting up mds nodes... 2024-05-27T00:14:15.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-05-27T00:14:15.892 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-05-27T00:14:15.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-05-27T00:14:15.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-05-27T00:14:16.014 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-05-27T00:14:16.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-05-27T00:14:16.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-05-27T00:14:16.139 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-05-27T00:14:16.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-05-27T00:14:16.162 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-05-27T00:14:16.207 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-05-27T00:14:16.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-05-27T00:14:16.265 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-05-27T00:14:16.349 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-05-27T00:14:16.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-05-27T00:14:16.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-05-27T00:14:16.492 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-05-27T00:14:16.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-05-27T00:14:16.548 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-27T00:14:16.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-27T00:14:16.590 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-27T00:14:16.607 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-27T00:14:16.608 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-27T00:14:16.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-27T00:14:16.669 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-27T00:14:16.670 INFO:tasks.ceph:role: osd.0 2024-05-27T00:14:16.670 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:14:16.670 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-27T00:14:16.739 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T00:14:16.740 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T00:14:16.740 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T00:14:16.740 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T00:14:16.740 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T00:14:16.740 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-05-27T00:14:16.740 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T00:14:16.740 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T00:14:16.740 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T00:14:16.740 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T00:14:17.135 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-05-27T00:14:17.138 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-05-27T00:14:17.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-27T00:14:17.235 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-27T00:14:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T00:14:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:14:17.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-27T00:14:17.317 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-27T00:14:17.317 INFO:tasks.ceph:role: osd.1 2024-05-27T00:14:17.317 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:14:17.317 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-27T00:14:17.388 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T00:14:17.388 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T00:14:17.388 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T00:14:17.388 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T00:14:17.389 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T00:14:17.389 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-05-27T00:14:17.389 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T00:14:17.389 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T00:14:17.389 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T00:14:17.389 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T00:14:17.785 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-05-27T00:14:17.788 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-05-27T00:14:17.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-27T00:14:17.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-27T00:14:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T00:14:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:14:17.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-27T00:14:18.015 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-27T00:14:18.015 INFO:tasks.ceph:role: osd.2 2024-05-27T00:14:18.015 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:14:18.015 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-27T00:14:18.086 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T00:14:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T00:14:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T00:14:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T00:14:18.087 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T00:14:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-05-27T00:14:18.087 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T00:14:18.087 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T00:14:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T00:14:18.087 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T00:14:18.502 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-05-27T00:14:18.505 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-05-27T00:14:18.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-27T00:14:18.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-27T00:14:18.666 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T00:14:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:14:18.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-27T00:14:18.730 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-27T00:14:18.730 INFO:tasks.ceph:role: osd.3 2024-05-27T00:14:18.731 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:14:18.731 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-27T00:14:18.802 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T00:14:18.802 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T00:14:18.802 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T00:14:18.802 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T00:14:18.803 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T00:14:18.803 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-05-27T00:14:18.803 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T00:14:18.803 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T00:14:18.803 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T00:14:18.803 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T00:14:19.157 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-05-27T00:14:19.160 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-05-27T00:14:19.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-27T00:14:19.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-27T00:14:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T00:14:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:14:19.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:19.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:19.419+0000 7f6cd086b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-27T00:14:19.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:19.427+0000 7f6cd086b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-27T00:14:19.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:19.427+0000 7f6cd086b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-27T00:14:19.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:19.427+0000 7f6cd086b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-27T00:14:19.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:19.427+0000 7f6cd086b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-27T00:14:19.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:19.431+0000 7f6cd086b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-27T00:14:22.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-27T00:14:22.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:22.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:22.618+0000 7f1b3f8047c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-27T00:14:22.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:22.622+0000 7f1b3f8047c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-27T00:14:22.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:22.622+0000 7f1b3f8047c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-27T00:14:22.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:22.622+0000 7f1b3f8047c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-27T00:14:22.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:22.622+0000 7f1b3f8047c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-27T00:14:22.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:22.622+0000 7f1b3f8047c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-27T00:14:25.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-27T00:14:25.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:25.798+0000 7f114dd927c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-27T00:14:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:25.798+0000 7f114dd927c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-27T00:14:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:25.802+0000 7f114dd927c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-27T00:14:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:25.802+0000 7f114dd927c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-27T00:14:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:25.802+0000 7f114dd927c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-27T00:14:25.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:25.802+0000 7f114dd927c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-27T00:14:28.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-27T00:14:28.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:29.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:29.006+0000 7f5ecb2957c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-27T00:14:29.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:29.006+0000 7f5ecb2957c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-27T00:14:29.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:29.006+0000 7f5ecb2957c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-27T00:14:29.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:29.006+0000 7f5ecb2957c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-27T00:14:29.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:29.006+0000 7f5ecb2957c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-27T00:14:29.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:29.006+0000 7f5ecb2957c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-27T00:14:32.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-27T00:14:32.066 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-27T00:14:32.081 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-27T00:14:32.081 INFO:tasks.ceph:role: osd.4 2024-05-27T00:14:32.081 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:14:32.081 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-27T00:14:32.145 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T00:14:32.145 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T00:14:32.145 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T00:14:32.145 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T00:14:32.145 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T00:14:32.145 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-05-27T00:14:32.146 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T00:14:32.146 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T00:14:32.146 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T00:14:32.146 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T00:14:32.593 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-05-27T00:14:32.595 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-05-27T00:14:32.596 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-27T00:14:32.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-27T00:14:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:14:32.693 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T00:14:32.693 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-27T00:14:32.751 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-27T00:14:32.751 INFO:tasks.ceph:role: osd.5 2024-05-27T00:14:32.751 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:14:32.751 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-27T00:14:32.814 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T00:14:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T00:14:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T00:14:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T00:14:32.815 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T00:14:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-05-27T00:14:32.815 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T00:14:32.815 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T00:14:32.815 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T00:14:32.815 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T00:14:33.282 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-05-27T00:14:33.284 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-05-27T00:14:33.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-27T00:14:33.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-27T00:14:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:14:33.459 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T00:14:33.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-27T00:14:33.514 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-27T00:14:33.514 INFO:tasks.ceph:role: osd.6 2024-05-27T00:14:33.514 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:14:33.514 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-27T00:14:33.577 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T00:14:33.578 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T00:14:33.578 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T00:14:33.578 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T00:14:33.578 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T00:14:33.578 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-05-27T00:14:33.578 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T00:14:33.578 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T00:14:33.578 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T00:14:33.578 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T00:14:34.059 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-05-27T00:14:34.061 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-05-27T00:14:34.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-27T00:14:34.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-27T00:14:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:14:34.226 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T00:14:34.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-27T00:14:34.281 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-27T00:14:34.281 INFO:tasks.ceph:role: osd.7 2024-05-27T00:14:34.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:14:34.281 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-27T00:14:34.345 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T00:14:34.345 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T00:14:34.345 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T00:14:34.345 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T00:14:34.345 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T00:14:34.345 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-05-27T00:14:34.345 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T00:14:34.345 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T00:14:34.345 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T00:14:34.346 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T00:14:34.778 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-05-27T00:14:34.781 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-05-27T00:14:34.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-27T00:14:34.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-27T00:14:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:14:34.888 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T00:14:34.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:35.003 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:34.998+0000 7f7f514c87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-27T00:14:35.006 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:35.002+0000 7f7f514c87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-27T00:14:35.007 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:35.002+0000 7f7f514c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-27T00:14:35.007 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:35.002+0000 7f7f514c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-27T00:14:35.007 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:35.002+0000 7f7f514c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-27T00:14:35.007 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:35.002+0000 7f7f514c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-27T00:14:37.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-27T00:14:37.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:38.073 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:38.070+0000 7efe85b357c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-27T00:14:38.074 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:38.070+0000 7efe85b357c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-27T00:14:38.074 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:38.070+0000 7efe85b357c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-27T00:14:38.074 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:38.070+0000 7efe85b357c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-27T00:14:38.074 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:38.070+0000 7efe85b357c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-27T00:14:38.075 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:38.070+0000 7efe85b357c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-27T00:14:41.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-27T00:14:41.091 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:41.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:41.166+0000 7f829b9477c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-27T00:14:41.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:41.166+0000 7f829b9477c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-27T00:14:41.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:41.166+0000 7f829b9477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-27T00:14:41.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:41.166+0000 7f829b9477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-27T00:14:41.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:41.166+0000 7f829b9477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-27T00:14:41.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:41.166+0000 7f829b9477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-27T00:14:43.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:14:43.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:14:44.175 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-27T00:14:44.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:44.270 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:44.266+0000 7fa4800a27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-27T00:14:44.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:44.266+0000 7fa4800a27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-27T00:14:44.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:44.266+0000 7fa4800a27c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-27T00:14:44.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:44.266+0000 7fa4800a27c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-27T00:14:44.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:44.266+0000 7fa4800a27c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-27T00:14:44.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:14:44.266+0000 7fa4800a27c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-27T00:14:47.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-27T00:14:47.306 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-27T00:14:47.306 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:47.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-27T00:14:47.329 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:47.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-05-27T00:14:47.394 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:47.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-05-27T00:14:47.463 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:47.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-05-27T00:14:47.530 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:47.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-27T00:14:47.594 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:47.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-27T00:14:47.658 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:47.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-27T00:14:47.726 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:47.726 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-27T00:14:47.793 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:47.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-27T00:14:47.807 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:47.807 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-05-27T00:14:47.861 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:47.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-05-27T00:14:47.917 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:47.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-05-27T00:14:47.974 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:47.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-27T00:14:48.029 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:48.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-27T00:14:48.085 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:48.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-27T00:14:48.140 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:14:48.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-27T00:14:48.197 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:14:48.197 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-27T00:14:48.205 INFO:tasks.ceph:Adding keys to all mons... 2024-05-27T00:14:48.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-27T00:14:48.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-27T00:14:48.264 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.y] 2024-05-27T00:14:48.264 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBX0FNm3+tOLhAAJELPI/1NbbEM2NbW6Q3Q5w== 2024-05-27T00:14:48.264 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.a] 2024-05-27T00:14:48.264 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBX0FNmIAk+NRAAY0h1wIxQgCF3La8zlg23DA== 2024-05-27T00:14:48.264 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.c] 2024-05-27T00:14:48.264 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBY0FNmTeHiABAAvJOxk4WyAMWGB6MGv99jmw== 2024-05-27T00:14:48.264 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.e] 2024-05-27T00:14:48.264 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBY0FNm6JhUCBAAhkN+DBr1/52Tf+K1ztT+Gg== 2024-05-27T00:14:48.264 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBb0FNm9olmGRAAOaQsvtUehrOCiTGVxY5FIQ== 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBe0FNmY3dQJRAAe1CofHUagxK1MdtrAQkHiQ== 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBh0FNmRwsHMBAAnCdI55oekcjVHPm61QlOQQ== 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.3] 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBl0FNm1rulABAAQRc7+eg9lQnpoNkVaSYpFA== 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBX0FNmYV0+MRAAxUljV3gNGukufRFZZS3FWg== 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.b] 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBY0FNmozJwDBAAzgEG7kG3WhqFdLDBEjN5Mw== 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.d] 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBY0FNmk5jfFBAAsL8MxB2RkEdtIciQcMSlhg== 2024-05-27T00:14:48.265 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.f] 2024-05-27T00:14:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBY0FNmmWhoHRAADwT60BrELCYios3+Z7M81A== 2024-05-27T00:14:48.266 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.4] 2024-05-27T00:14:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBr0FNmPc5CABAAhmxq8BUDTMt98lAYKq9tzg== 2024-05-27T00:14:48.266 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.5] 2024-05-27T00:14:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBu0FNmqrx1BBAA8FZPpQYJmqzI0zSoo6x+zg== 2024-05-27T00:14:48.266 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.6] 2024-05-27T00:14:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBx0FNmO349ChAA92VlP/7859pOxu0AsBdzHQ== 2024-05-27T00:14:48.266 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.7] 2024-05-27T00:14:48.266 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB00FNm0zwpEBAA1p/9rkeo/9vAJZ1Bq+IOmw== 2024-05-27T00:14:48.266 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2024-05-27T00:14:48.270 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBY0FNmr+Y+IxAAmEDpKQoQ1iABr+n1Ha0Ixg== 2024-05-27T00:14:48.271 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2024-05-27T00:14:48.271 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBX0FNm3+tOLhAAJELPI/1NbbEM2NbW6Q3Q5w== 2024-05-27T00:14:48.271 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2024-05-27T00:14:48.271 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBX0FNmIAk+NRAAY0h1wIxQgCF3La8zlg23DA== 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBY0FNmTeHiABAAvJOxk4WyAMWGB6MGv99jmw== 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e] 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBY0FNm6JhUCBAAhkN+DBr1/52Tf+K1ztT+Gg== 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBb0FNm9olmGRAAOaQsvtUehrOCiTGVxY5FIQ== 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBe0FNmY3dQJRAAe1CofHUagxK1MdtrAQkHiQ== 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBh0FNmRwsHMBAAnCdI55oekcjVHPm61QlOQQ== 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBl0FNm1rulABAAQRc7+eg9lQnpoNkVaSYpFA== 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-05-27T00:14:48.272 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBX0FNmYV0+MRAAxUljV3gNGukufRFZZS3FWg== 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBY0FNmozJwDBAAzgEG7kG3WhqFdLDBEjN5Mw== 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBY0FNmk5jfFBAAsL8MxB2RkEdtIciQcMSlhg== 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f] 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBY0FNmmWhoHRAADwT60BrELCYios3+Z7M81A== 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBr0FNmPc5CABAAhmxq8BUDTMt98lAYKq9tzg== 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBu0FNmqrx1BBAA8FZPpQYJmqzI0zSoo6x+zg== 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBx0FNmO349ChAA92VlP/7859pOxu0AsBdzHQ== 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2024-05-27T00:14:48.273 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQB00FNm0zwpEBAA1p/9rkeo/9vAJZ1Bq+IOmw== 2024-05-27T00:14:48.274 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-05-27T00:14:48.274 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBY0FNmr+Y+IxAAmEDpKQoQ1iABr+n1Ha0Ixg== 2024-05-27T00:14:48.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-27T00:14:48.317 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-27T00:14:48.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:48.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:48.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:48.431 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:48.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:48.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:48.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:48.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:48.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:48.648 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:48.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:48.718 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:48.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:48.798 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:48.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-27T00:14:48.885 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-27T00:14:48.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:48.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:49.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:49.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:49.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:49.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T00:14:49.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:49.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:49.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:49.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:49.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:49.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:49.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:49.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T00:14:49.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-27T00:14:49.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-27T00:14:49.575 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-27T00:14:49.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-27T00:14:49.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-27T00:14:49.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-27T00:14:49.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-27T00:14:49.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-27T00:14:49.831 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-27T00:14:49.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-27T00:14:49.894 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-27T00:14:49.988 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-27T00:14:49.999 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:50.002 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-27T00:14:50.048 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-27T00:14:50.049 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-27T00:14:50.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-27T00:14:50.052 INFO:tasks.ceph.mon.a:Started 2024-05-27T00:14:50.052 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-27T00:14:50.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-27T00:14:50.095 INFO:tasks.ceph.mon.b:Started 2024-05-27T00:14:50.095 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-27T00:14:50.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-27T00:14:50.097 INFO:tasks.ceph.mon.c:Started 2024-05-27T00:14:50.098 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-27T00:14:50.098 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-27T00:14:50.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-27T00:14:50.101 INFO:tasks.ceph.mgr.y:Started 2024-05-27T00:14:50.101 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-27T00:14:50.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-27T00:14:50.106 INFO:tasks.ceph.mgr.x:Started 2024-05-27T00:14:50.106 INFO:tasks.ceph:Setting crush tunables to default 2024-05-27T00:14:50.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd crush tunables default 2024-05-27T00:14:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5829800640 1 Processor -- start 2024-05-27T00:14:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5829800640 1 -- start start 2024-05-27T00:14:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5829800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:14:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5829800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5824159010 con 0x7f5824152990 2024-05-27T00:14:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5829800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f582415bc40 con 0x7f58240b1c20 2024-05-27T00:14:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5829800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f582415c850 con 0x7f5824158640 2024-05-27T00:14:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5822a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5824152990 0x7f582406e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36120/0 (socket says 172.21.15.98:36120) 2024-05-27T00:14:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 learned_addr learned my addr 172.21.15.98:0/2657653872 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:14:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:50.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f5824158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-27T00:14:50.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f5824158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:50.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:50.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.246+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T00:14:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.450+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.450+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f5824158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-27T00:14:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.450+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f5824158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.450+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.450+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T00:14:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.854+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.854+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f5824158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-27T00:14:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.854+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f5824158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.854+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:50.854+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T00:14:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:51.658+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:51.658+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f5824158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-27T00:14:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:51.658+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f5824158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:51.658+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:51.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:51.658+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T00:14:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f5824158ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T00:14:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f581be00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f5824158ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:14:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 >> v1:172.21.15.204:6789/0 conn(0x7f58240b1c20 legacy=0x7f58240c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:14:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 >> v1:172.21.15.98:6789/0 conn(0x7f5824152990 legacy=0x7f582406e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:14:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f581be00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:14:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f581be00640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:14:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f582415c850 con 0x7f58040036f0 2024-05-27T00:14:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5804008d50 con 0x7f5804006190 2024-05-27T00:14:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 msgr2=0x7f5804005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-27T00:14:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 msgr2=0x7f5804005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T00:14:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 msgr2=0x7f5804008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-27T00:14:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 msgr2=0x7f5804008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.246+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T00:14:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.450+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.450+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 msgr2=0x7f5804005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-27T00:14:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.450+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 msgr2=0x7f5804005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.450+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.450+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T00:14:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.450+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.450+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 msgr2=0x7f5804008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-27T00:14:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.450+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 msgr2=0x7f5804008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.450+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.450+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T00:14:53.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.854+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:53.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.854+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 msgr2=0x7f5804005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-27T00:14:53.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.854+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 msgr2=0x7f5804005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:53.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.854+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:53.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.854+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:53.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.854+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T00:14:53.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.854+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 msgr2=0x7f5804008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-27T00:14:53.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.854+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 msgr2=0x7f5804008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:53.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.854+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:53.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:53.854+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T00:14:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:54.658+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:54.658+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 msgr2=0x7f5804005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-27T00:14:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:54.658+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 msgr2=0x7f5804005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:54.658+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:54.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:54.658+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:54.658+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T00:14:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:54.658+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 msgr2=0x7f5804008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-27T00:14:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:54.658+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 msgr2=0x7f5804008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:54.658+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:54.658+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 msgr2=0x7f5804008630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.246+0000 7f581be00640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804006190 0x7f5804008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 msgr2=0x7f5804005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.246+0000 7f581be00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f58040036f0 0x7f5804005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.246+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 reap_dead start 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.246+0000 7f581be00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.246+0000 7f581be00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f582415bf20 con 0x7f5804003b40 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f582415c850 con 0x7f5804007d40 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f5804008d50 con 0x7f5824158640 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.250+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:56.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.250+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.250+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 msgr2=0x7f58040081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-27T00:14:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.250+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 msgr2=0x7f58040081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.250+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.250+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T00:14:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.250+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f580400b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-27T00:14:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.250+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f580400b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.250+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:56.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.250+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T00:14:56.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.454+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.454+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 msgr2=0x7f58040081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-27T00:14:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.454+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 msgr2=0x7f58040081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.454+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.454+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.454+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T00:14:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.454+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f580400b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-27T00:14:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.454+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f580400b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.454+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:56.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.454+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T00:14:56.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.858+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.858+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 msgr2=0x7f58040081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-27T00:14:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.858+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 msgr2=0x7f58040081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.858+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.858+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.858+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T00:14:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.858+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f580400b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-27T00:14:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.858+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f580400b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.858+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:56.858+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T00:14:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:57.662+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:57.662+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 msgr2=0x7f58040081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-27T00:14:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:57.662+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 msgr2=0x7f58040081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:57.662+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:57.662+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:57.662+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T00:14:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:57.662+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f580400b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-27T00:14:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:57.662+0000 7f5822a00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f580400b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:57.662+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:57.662+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T00:14:59.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 msgr2=0x7f580400b200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T00:14:59.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.246+0000 7f581be00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.204:3300/0 conn(0x7f5824158640 0x7f580400b200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:14:59.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 msgr2=0x7f58040081d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T00:14:59.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.246+0000 7f581be00640 1 --2- 172.21.15.98:0/2657653872 >> v2:172.21.15.98:3300/0 conn(0x7f5804007d40 0x7f58040081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:14:59.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.246+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 >> v1:172.21.15.98:6789/0 conn(0x7f5804003b40 legacy=0x7f5804006620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:14:59.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.250+0000 7f581be00640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:14:59.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.250+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5804008860 con 0x7f580400d7d0 2024-05-27T00:14:59.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.250+0000 7f581be00640 1 -- 172.21.15.98:0/2657653872 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f582415c850 con 0x7f5804011440 2024-05-27T00:14:59.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.250+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:59.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.250+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 msgr2=0x7f5804013860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-27T00:14:59.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.250+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 msgr2=0x7f5804013860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:59.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.250+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:59.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.250+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T00:14:59.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.454+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:59.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.454+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 msgr2=0x7f5804013860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-27T00:14:59.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.454+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 msgr2=0x7f5804013860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:59.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.454+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:59.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.454+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T00:14:59.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.858+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:14:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.858+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 msgr2=0x7f5804013860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-27T00:14:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.858+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 msgr2=0x7f5804013860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T00:14:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.858+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T00:14:59.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:14:59.858+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T00:15:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.474+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3336177690 0 0) 0x7f5804008860 con 0x7f580400d7d0 2024-05-27T00:15:00.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.478+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5808003530 con 0x7f580400d7d0 2024-05-27T00:15:00.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2820896880 0 0) 0x7f5808003530 con 0x7f580400d7d0 2024-05-27T00:15:00.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 msgr2=0x7f5804013860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T00:15:00.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5804011440 0x7f5804013860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5804015030 con 0x7f580400d7d0 2024-05-27T00:15:00.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f5814001c70 con 0x7f580400d7d0 2024-05-27T00:15:00.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 >> v1:172.21.15.204:6789/0 conn(0x7f580400d7d0 legacy=0x7f580400fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:00.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5823400640 1 -- 172.21.15.98:0/2657653872 reap_dead start 2024-05-27T00:15:00.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f5808003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:00.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5808004810 0x7f5808004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:00.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58080052f0 0x7f580800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:00.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f582415c850 con 0x7f5808004810 2024-05-27T00:15:00.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f580800c1a0 con 0x7f58080052f0 2024-05-27T00:15:00.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f580800c3a0 con 0x7f58080039c0 2024-05-27T00:15:00.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f5808003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:00.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5808004810 0x7f5808004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:00.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5823400640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58080052f0 0x7f580800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:00.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 msgr2=0x7f5808003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:00.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f5808003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58080052f0 msgr2=0x7f580800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:00.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58080052f0 0x7f580800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5822000640 1 -- 172.21.15.98:0/2657653872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5808003530 con 0x7f5808004810 2024-05-27T00:15:00.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.486+0000 7f5822a00640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f5808003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:00.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5822000640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5808004810 0x7f5808004c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5810007c40 tx=0x7f581001d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e17b3dd3422b59 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:00.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58100ae070 con 0x7f5808004810 2024-05-27T00:15:00.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58100170f0 con 0x7f5808004810 2024-05-27T00:15:00.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5821600640 1 -- 172.21.15.98:0/2657653872 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5810017530 con 0x7f5808004810 2024-05-27T00:15:00.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5829800640 1 -- 172.21.15.98:0/2657653872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5808004810 msgr2=0x7f5808004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:00.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5829800640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5808004810 0x7f5808004c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5810007c40 tx=0x7f581001d820 comp rx=0 tx=0).stop 2024-05-27T00:15:00.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5829800640 1 -- 172.21.15.98:0/2657653872 shutdown_connections 2024-05-27T00:15:00.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5829800640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58080052f0 0x7f580800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5829800640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5808004810 0x7f5808004c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5829800640 1 --2- 172.21.15.98:0/2657653872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f5808003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5829800640 1 -- 172.21.15.98:0/2657653872 >> 172.21.15.98:0/2657653872 conn(0x7f58240cbf00 msgr2=0x7f58240cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:00.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5829800640 1 -- 172.21.15.98:0/2657653872 shutdown_connections 2024-05-27T00:15:00.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.490+0000 7f5829800640 1 -- 172.21.15.98:0/2657653872 wait complete. 2024-05-27T00:15:00.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5829800640 1 Processor -- start 2024-05-27T00:15:00.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5829800640 1 -- start start 2024-05-27T00:15:00.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5829800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f58240c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:00.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5829800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5808004810 0x7f58241f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:00.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5829800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58080052f0 0x7f58241f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5829800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5824158ad0 con 0x7f58080052f0 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5829800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f582415c4e0 con 0x7f5808004810 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5829800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5824158c50 con 0x7f58080039c0 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5822a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f58240c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5822a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f58240c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:54878/0 (socket says 172.21.15.98:54878) 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5822a00640 1 -- 172.21.15.98:0/3654599846 learned_addr learned my addr 172.21.15.98:0/3654599846 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5823400640 1 --2- 172.21.15.98:0/3654599846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58080052f0 0x7f58241f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5822000640 1 --2- 172.21.15.98:0/3654599846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5808004810 0x7f58241f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5822a00640 1 -- 172.21.15.98:0/3654599846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5808004810 msgr2=0x7f58241f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5822a00640 1 --2- 172.21.15.98:0/3654599846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5808004810 0x7f58241f3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5822a00640 1 -- 172.21.15.98:0/3654599846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58080052f0 msgr2=0x7f58241f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5822a00640 1 --2- 172.21.15.98:0/3654599846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58080052f0 0x7f58241f3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5822a00640 1 -- 172.21.15.98:0/3654599846 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58241f43b0 con 0x7f58080039c0 2024-05-27T00:15:00.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5823400640 1 --2- 172.21.15.98:0/3654599846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58080052f0 0x7f58241f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5822a00640 1 --2- 172.21.15.98:0/3654599846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f58240c35c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58140186a0 tx=0x7f5814018aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f581aa00640 1 -- 172.21.15.98:0/3654599846 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5814004b70 con 0x7f58080039c0 2024-05-27T00:15:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f581aa00640 1 -- 172.21.15.98:0/3654599846 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5814004d10 con 0x7f58080039c0 2024-05-27T00:15:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5829800640 1 -- 172.21.15.98:0/3654599846 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f582415a520 con 0x7f58080039c0 2024-05-27T00:15:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f581aa00640 1 -- 172.21.15.98:0/3654599846 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58140073e0 con 0x7f58080039c0 2024-05-27T00:15:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.494+0000 7f5829800640 1 -- 172.21.15.98:0/3654599846 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f582414ffd0 con 0x7f58080039c0 2024-05-27T00:15:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.498+0000 7f581aa00640 1 -- 172.21.15.98:0/3654599846 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f581402b020 con 0x7f58080039c0 2024-05-27T00:15:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.498+0000 7f581aa00640 1 -- 172.21.15.98:0/3654599846 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5814005500 con 0x7f58080039c0 2024-05-27T00:15:00.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.498+0000 7f5829800640 1 -- 172.21.15.98:0/3654599846 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5824075cb0 con 0x7f58080039c0 2024-05-27T00:15:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.506+0000 7f581aa00640 1 -- 172.21.15.98:0/3654599846 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5814005020 con 0x7f58080039c0 2024-05-27T00:15:00.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.566+0000 7f5829800640 1 -- 172.21.15.98:0/3654599846 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f58240c76f0 con 0x7f58080039c0 2024-05-27T00:15:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.578+0000 7f581aa00640 1 -- 172.21.15.98:0/3654599846 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f5814005700 con 0x7f58080039c0 2024-05-27T00:15:00.578 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-05-27T00:15:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.578+0000 7f5829800640 1 -- 172.21.15.98:0/3654599846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 msgr2=0x7f58240c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.578+0000 7f5829800640 1 --2- 172.21.15.98:0/3654599846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f58240c35c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58140186a0 tx=0x7f5814018aa0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.578+0000 7f5829800640 1 -- 172.21.15.98:0/3654599846 shutdown_connections 2024-05-27T00:15:00.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.578+0000 7f5829800640 1 --2- 172.21.15.98:0/3654599846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58080052f0 0x7f58241f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.578+0000 7f5829800640 1 --2- 172.21.15.98:0/3654599846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5808004810 0x7f58241f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.578+0000 7f5829800640 1 --2- 172.21.15.98:0/3654599846 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58080039c0 0x7f58240c35c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.578+0000 7f5829800640 1 -- 172.21.15.98:0/3654599846 >> 172.21.15.98:0/3654599846 conn(0x7f58240cbf00 msgr2=0x7f58240c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.578+0000 7f5829800640 1 -- 172.21.15.98:0/3654599846 shutdown_connections 2024-05-27T00:15:00.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:00.578+0000 7f5829800640 1 -- 172.21.15.98:0/3654599846 wait complete. 2024-05-27T00:15:00.598 INFO:tasks.ceph:check_enable_crimson: False 2024-05-27T00:15:00.598 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-27T00:15:00.598 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:15:00.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-27T00:15:00.618 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:15:00.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-27T00:15:00.682 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:15:00.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-27T00:15:00.684 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:00.681+0000 7f8a7217b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T00:15:00.746 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:15:00.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-27T00:15:00.805 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:15:00.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-27T00:15:00.819 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:15:00.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-27T00:15:00.841 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:00.838+0000 7fc043f7e180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T00:15:00.872 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:15:00.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-27T00:15:00.924 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-27T00:15:00.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-27T00:15:00.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new b68fe95c-cc54-40c9-a1ac-363d1ed74dea 0 2024-05-27T00:15:01.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.141+0000 7f785cd12640 1 Processor -- start 2024-05-27T00:15:01.145 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f785cd12640 1 -- start start 2024-05-27T00:15:01.147 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f785cd12640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78580c84b0 0x7f7858162d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.148 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f785cd12640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7858071a20 con 0x7f78581517c0 2024-05-27T00:15:01.148 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f785cd12640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7858071d00 con 0x7f78580c8070 2024-05-27T00:15:01.148 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f785cd12640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f78580c8940 con 0x7f78580c84b0 2024-05-27T00:15:01.148 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f784ffff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f78580c8070 0x7f78580c6110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:50886/0 (socket says 172.21.15.204:50886) 2024-05-27T00:15:01.148 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f784ffff640 1 -- 172.21.15.204:0/2042567171 learned_addr learned my addr 172.21.15.204:0/2042567171 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-05-27T00:15:01.148 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f7856d9d640 1 --2- 172.21.15.204:0/2042567171 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78580c84b0 0x7f7858162d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.148 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f7855d9b640 1 -- 172.21.15.204:0/2042567171 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 237143253 0 0) 0x7f7858071d00 con 0x7f78580c8070 2024-05-27T00:15:01.148 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f7855d9b640 1 -- 172.21.15.204:0/2042567171 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7844003530 con 0x7f78580c8070 2024-05-27T00:15:01.148 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f7856d9d640 1 -- 172.21.15.204:0/2042567171 >> v1:172.21.15.204:6789/0 conn(0x7f78580c8070 legacy=0x7f78580c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.149 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f7856d9d640 1 -- 172.21.15.204:0/2042567171 >> v1:172.21.15.98:6789/0 conn(0x7f78581517c0 legacy=0x7f78580c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:01.149 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f7856d9d640 1 -- 172.21.15.204:0/2042567171 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7858071fe0 con 0x7f78580c84b0 2024-05-27T00:15:01.151 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f7856d9d640 1 --2- 172.21.15.204:0/2042567171 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78580c84b0 0x7f7858162d40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7850009dc0 tx=0x7f785001c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5102b9dacfe62400 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:01.151 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.145+0000 7f7855d9b640 1 -- 172.21.15.204:0/2042567171 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78500b2070 con 0x7f78580c84b0 2024-05-27T00:15:01.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f7855d9b640 1 -- 172.21.15.204:0/2042567171 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78500a59e0 con 0x7f78580c84b0 2024-05-27T00:15:01.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f7855d9b640 1 -- 172.21.15.204:0/2042567171 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78500a5db0 con 0x7f78580c84b0 2024-05-27T00:15:01.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- 172.21.15.204:0/2042567171 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78580c84b0 msgr2=0x7f7858162d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 --2- 172.21.15.204:0/2042567171 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78580c84b0 0x7f7858162d40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7850009dc0 tx=0x7f785001c920 comp rx=0 tx=0).stop 2024-05-27T00:15:01.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- 172.21.15.204:0/2042567171 shutdown_connections 2024-05-27T00:15:01.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 --2- 172.21.15.204:0/2042567171 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78580c84b0 0x7f7858162d40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- 172.21.15.204:0/2042567171 >> 172.21.15.204:0/2042567171 conn(0x7f785806b470 msgr2=0x7f785806b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:01.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- 172.21.15.204:0/2042567171 shutdown_connections 2024-05-27T00:15:01.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- 172.21.15.204:0/2042567171 wait complete. 2024-05-27T00:15:01.153 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 Processor -- start 2024-05-27T00:15:01.153 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- start start 2024-05-27T00:15:01.155 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78580c8070 0x7f78580cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.155 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78580c84b0 0x7f7858144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.155 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78581517c0 0x7f7858145340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.155 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7858071650 con 0x7f78580c84b0 2024-05-27T00:15:01.155 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7858151ba0 con 0x7f78580c8070 2024-05-27T00:15:01.155 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f78580717d0 con 0x7f78581517c0 2024-05-27T00:15:01.155 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78580c8070 0x7f78580cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.155 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78580c8070 0x7f78580cc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:44462/0 (socket says 172.21.15.204:44462) 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785659c640 1 -- 172.21.15.204:0/3849148217 learned_addr learned my addr 172.21.15.204:0/3849148217 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f7856d9d640 1 --2- 172.21.15.204:0/3849148217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78581517c0 0x7f7858145340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f784ffff640 1 --2- 172.21.15.204:0/3849148217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78580c84b0 0x7f7858144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785659c640 1 -- 172.21.15.204:0/3849148217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78581517c0 msgr2=0x7f7858145340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785659c640 1 --2- 172.21.15.204:0/3849148217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78581517c0 0x7f7858145340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785659c640 1 -- 172.21.15.204:0/3849148217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78580c84b0 msgr2=0x7f7858144e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785659c640 1 --2- 172.21.15.204:0/3849148217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78580c84b0 0x7f7858144e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785659c640 1 -- 172.21.15.204:0/3849148217 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78581f6f00 con 0x7f78580c8070 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f784ffff640 1 --2- 172.21.15.204:0/3849148217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78580c84b0 0x7f7858144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785659c640 1 --2- 172.21.15.204:0/3849148217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78580c8070 0x7f78580cc6b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f784800b410 tx=0x7f784800b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f784effd640 1 -- 172.21.15.204:0/3849148217 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848015020 con 0x7f78580c8070 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f784effd640 1 -- 172.21.15.204:0/3849148217 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7848011830 con 0x7f78580c8070 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- 172.21.15.204:0/3849148217 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78581f71c0 con 0x7f78580c8070 2024-05-27T00:15:01.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f784effd640 1 -- 172.21.15.204:0/3849148217 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848011c20 con 0x7f78580c8070 2024-05-27T00:15:01.157 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f785cd12640 1 -- 172.21.15.204:0/3849148217 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78581f34a0 con 0x7f78580c8070 2024-05-27T00:15:01.157 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f784effd640 1 -- 172.21.15.204:0/3849148217 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7848018020 con 0x7f78580c8070 2024-05-27T00:15:01.157 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.149+0000 7f784effd640 1 -- 172.21.15.204:0/3849148217 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f78480197f0 con 0x7f78580c8070 2024-05-27T00:15:01.157 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.153+0000 7f785cd12640 1 -- 172.21.15.204:0/3849148217 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7858075cb0 con 0x7f78580c8070 2024-05-27T00:15:01.157 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.153+0000 7f784effd640 1 -- 172.21.15.204:0/3849148217 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f784801f020 con 0x7f78580c8070 2024-05-27T00:15:01.187 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:01.185+0000 7f8a7217b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T00:15:01.225 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.225+0000 7f785cd12640 1 -- 172.21.15.204:0/3849148217 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b68fe95c-cc54-40c9-a1ac-363d1ed74dea", "id": 0} v 0) -- 0x7f7858149ed0 con 0x7f78580c8070 2024-05-27T00:15:01.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.229+0000 7f784effd640 1 -- 172.21.15.204:0/3849148217 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b68fe95c-cc54-40c9-a1ac-363d1ed74dea", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f7848011dc0 con 0x7f78580c8070 2024-05-27T00:15:01.230 INFO:teuthology.orchestra.run.smithi204.stdout:0 2024-05-27T00:15:01.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.229+0000 7f785cd12640 1 -- 172.21.15.204:0/3849148217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78580c8070 msgr2=0x7f78580cc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.229+0000 7f785cd12640 1 --2- 172.21.15.204:0/3849148217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78580c8070 0x7f78580cc6b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f784800b410 tx=0x7f784800b8d0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.229+0000 7f785cd12640 1 -- 172.21.15.204:0/3849148217 shutdown_connections 2024-05-27T00:15:01.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.229+0000 7f785cd12640 1 --2- 172.21.15.204:0/3849148217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78581517c0 0x7f7858145340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.229+0000 7f785cd12640 1 --2- 172.21.15.204:0/3849148217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78580c84b0 0x7f7858144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.229+0000 7f785cd12640 1 --2- 172.21.15.204:0/3849148217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78580c8070 0x7f78580cc6b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.229+0000 7f785cd12640 1 -- 172.21.15.204:0/3849148217 >> 172.21.15.204:0/3849148217 conn(0x7f785806b470 msgr2=0x7f78580c9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:01.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.229+0000 7f785cd12640 1 -- 172.21.15.204:0/3849148217 shutdown_connections 2024-05-27T00:15:01.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.229+0000 7f785cd12640 1 -- 172.21.15.204:0/3849148217 wait complete. 2024-05-27T00:15:01.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 3b76f535-44e3-4dcd-90be-4dd6e2b326b0 1 2024-05-27T00:15:01.360 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:01.358+0000 7fc043f7e180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T00:15:01.360 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.357+0000 7fc975d19640 1 Processor -- start 2024-05-27T00:15:01.360 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.357+0000 7fc975d19640 1 -- start start 2024-05-27T00:15:01.362 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc975d19640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc970159a80 0x7fc9700c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.362 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc975d19640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9700c4d20 con 0x7fc9700c3090 2024-05-27T00:15:01.362 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc975d19640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9700c5000 con 0x7fc9701526e0 2024-05-27T00:15:01.362 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc975d19640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fc970152ac0 con 0x7fc970159a80 2024-05-27T00:15:01.362 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc96effd640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc970159a80 0x7fc9700c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.362 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc96f7fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fc9701526e0 0x7fc970158fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:50894/0 (socket says 172.21.15.204:50894) 2024-05-27T00:15:01.362 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc96f7fe640 1 -- 172.21.15.204:0/4045785765 learned_addr learned my addr 172.21.15.204:0/4045785765 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-05-27T00:15:01.362 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc96effd640 1 -- 172.21.15.204:0/4045785765 >> v1:172.21.15.204:6789/0 conn(0x7fc9701526e0 legacy=0x7fc970158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.363 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc96effd640 1 -- 172.21.15.204:0/4045785765 >> v1:172.21.15.98:6789/0 conn(0x7fc9700c3090 legacy=0x7fc97006f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.363 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc96e7fc640 1 -- 172.21.15.204:0/4045785765 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3383248875 0 0) 0x7fc9700c5000 con 0x7fc9701526e0 2024-05-27T00:15:01.363 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.361+0000 7fc96effd640 1 -- 172.21.15.204:0/4045785765 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9701587a0 con 0x7fc970159a80 2024-05-27T00:15:01.365 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96effd640 1 --2- 172.21.15.204:0/4045785765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc970159a80 0x7fc9700c8a70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc9600097c0 tx=0x7fc96001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63ce6894792fe70c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:01.365 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96e7fc640 1 -- 172.21.15.204:0/4045785765 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9600b5070 con 0x7fc970159a80 2024-05-27T00:15:01.365 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96e7fc640 1 -- 172.21.15.204:0/4045785765 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9600a8990 con 0x7fc970159a80 2024-05-27T00:15:01.366 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96e7fc640 1 -- 172.21.15.204:0/4045785765 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9600a8d60 con 0x7fc970159a80 2024-05-27T00:15:01.367 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 -- 172.21.15.204:0/4045785765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc970159a80 msgr2=0x7fc9700c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.367 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 --2- 172.21.15.204:0/4045785765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc970159a80 0x7fc9700c8a70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc9600097c0 tx=0x7fc96001ec90 comp rx=0 tx=0).stop 2024-05-27T00:15:01.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 -- 172.21.15.204:0/4045785765 shutdown_connections 2024-05-27T00:15:01.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 --2- 172.21.15.204:0/4045785765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc970159a80 0x7fc9700c8a70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 -- 172.21.15.204:0/4045785765 >> 172.21.15.204:0/4045785765 conn(0x7fc97006d680 msgr2=0x7fc97006da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:01.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 -- 172.21.15.204:0/4045785765 shutdown_connections 2024-05-27T00:15:01.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 -- 172.21.15.204:0/4045785765 wait complete. 2024-05-27T00:15:01.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 Processor -- start 2024-05-27T00:15:01.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 -- start start 2024-05-27T00:15:01.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9700c3090 0x7fc9700cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9701526e0 0x7fc97014eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc970159a80 0x7fc9701f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc97006f9e0 con 0x7fc970159a80 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc9700c3640 con 0x7fc9700c3090 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc975d19640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc97006fb60 con 0x7fc9701526e0 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96f7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9700c3090 0x7fc9700cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96effd640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9701526e0 0x7fc97014eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96f7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9700c3090 0x7fc9700cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:44476/0 (socket says 172.21.15.204:44476) 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96f7fe640 1 -- 172.21.15.204:0/2589149586 learned_addr learned my addr 172.21.15.204:0/2589149586 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96ffff640 1 --2- 172.21.15.204:0/2589149586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc970159a80 0x7fc9701f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96effd640 1 -- 172.21.15.204:0/2589149586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9700c3090 msgr2=0x7fc9700cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96effd640 1 --2- 172.21.15.204:0/2589149586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9700c3090 0x7fc9700cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96effd640 1 -- 172.21.15.204:0/2589149586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc970159a80 msgr2=0x7fc9701f0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96effd640 1 --2- 172.21.15.204:0/2589149586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc970159a80 0x7fc9701f0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96effd640 1 -- 172.21.15.204:0/2589149586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9701f3f90 con 0x7fc9701526e0 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.365+0000 7fc96f7fe640 1 --2- 172.21.15.204:0/2589149586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9700c3090 0x7fc9700cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:01.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.369+0000 7fc96effd640 1 --2- 172.21.15.204:0/2589149586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9701526e0 0x7fc97014eda0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc9600af000 tx=0x7fc96001ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:01.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.369+0000 7fc957fff640 1 -- 172.21.15.204:0/2589149586 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9600b5040 con 0x7fc9701526e0 2024-05-27T00:15:01.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.369+0000 7fc957fff640 1 -- 172.21.15.204:0/2589149586 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9600bf070 con 0x7fc9701526e0 2024-05-27T00:15:01.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.369+0000 7fc975d19640 1 -- 172.21.15.204:0/2589149586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9701f4250 con 0x7fc9701526e0 2024-05-27T00:15:01.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.369+0000 7fc957fff640 1 -- 172.21.15.204:0/2589149586 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9600b0a60 con 0x7fc9701526e0 2024-05-27T00:15:01.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.369+0000 7fc975d19640 1 -- 172.21.15.204:0/2589149586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9701f4810 con 0x7fc9701526e0 2024-05-27T00:15:01.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.369+0000 7fc957fff640 1 -- 172.21.15.204:0/2589149586 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc960004020 con 0x7fc9701526e0 2024-05-27T00:15:01.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.369+0000 7fc957fff640 1 -- 172.21.15.204:0/2589149586 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fc9600c3410 con 0x7fc9701526e0 2024-05-27T00:15:01.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.369+0000 7fc975d19640 1 -- 172.21.15.204:0/2589149586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc970075cb0 con 0x7fc9701526e0 2024-05-27T00:15:01.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.369+0000 7fc957fff640 1 -- 172.21.15.204:0/2589149586 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc9600187f0 con 0x7fc9701526e0 2024-05-27T00:15:01.398 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:01.397+0000 7f8a7217b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T00:15:01.442 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.441+0000 7fc975d19640 1 -- 172.21.15.204:0/2589149586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3b76f535-44e3-4dcd-90be-4dd6e2b326b0", "id": 1} v 0) -- 0x7fc97006fdd0 con 0x7fc9701526e0 2024-05-27T00:15:01.448 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.445+0000 7fc957fff640 1 -- 172.21.15.204:0/2589149586 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3b76f535-44e3-4dcd-90be-4dd6e2b326b0", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fc9600a8960 con 0x7fc9701526e0 2024-05-27T00:15:01.448 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-05-27T00:15:01.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.449+0000 7fc975d19640 1 -- 172.21.15.204:0/2589149586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9701526e0 msgr2=0x7fc97014eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.449+0000 7fc975d19640 1 --2- 172.21.15.204:0/2589149586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9701526e0 0x7fc97014eda0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc9600af000 tx=0x7fc96001ba30 comp rx=0 tx=0).stop 2024-05-27T00:15:01.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.449+0000 7fc975d19640 1 -- 172.21.15.204:0/2589149586 shutdown_connections 2024-05-27T00:15:01.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.449+0000 7fc975d19640 1 --2- 172.21.15.204:0/2589149586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc970159a80 0x7fc9701f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.449+0000 7fc975d19640 1 --2- 172.21.15.204:0/2589149586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9701526e0 0x7fc97014eda0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.449+0000 7fc975d19640 1 --2- 172.21.15.204:0/2589149586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9700c3090 0x7fc9700cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.449+0000 7fc975d19640 1 -- 172.21.15.204:0/2589149586 >> 172.21.15.204:0/2589149586 conn(0x7fc97006d680 msgr2=0x7fc970150b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:01.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.449+0000 7fc975d19640 1 -- 172.21.15.204:0/2589149586 shutdown_connections 2024-05-27T00:15:01.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.449+0000 7fc975d19640 1 -- 172.21.15.204:0/2589149586 wait complete. 2024-05-27T00:15:01.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new bac80420-b2e9-4fe6-8a5a-6502ca48305f 2 2024-05-27T00:15:01.578 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:01.578+0000 7fc043f7e180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T00:15:01.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.581+0000 7f3c4ba51640 1 Processor -- start 2024-05-27T00:15:01.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.581+0000 7f3c4ba51640 1 -- start start 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.581+0000 7f3c4ba51640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3c4415b430 0x7f3c440732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.581+0000 7f3c4ba51640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c440737f0 0x7f3c44067130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.581+0000 7f3c4ba51640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3c440c9450 0x7f3c44067670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.581+0000 7f3c4ba51640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3c44072e20 con 0x7f3c440c9450 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.581+0000 7f3c4ba51640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f3c440c7940 con 0x7f3c4415b430 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.581+0000 7f3c4ba51640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3c440c7bb0 con 0x7f3c440737f0 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c497ed640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3c4415b430 0x7f3c440732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c48fec640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c440737f0 0x7f3c44067130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c497ed640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3c4415b430 0x7f3c440732b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:44488/0 (socket says 172.21.15.204:44488) 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c48fec640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c440737f0 0x7f3c44067130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.204:43660/0 (socket says 172.21.15.204:43660) 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c497ed640 1 -- 172.21.15.204:0/2886166348 learned_addr learned my addr 172.21.15.204:0/2886166348 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:01.586 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c49fee640 1 --2- 172.21.15.204:0/2886166348 >> v2:172.21.15.98:3300/0 conn(0x7f3c440c9450 0x7f3c44067670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.587 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c497ed640 1 -- 172.21.15.204:0/2886166348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c440737f0 msgr2=0x7f3c44067130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:01.587 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c497ed640 1 --2- 172.21.15.204:0/2886166348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c440737f0 0x7f3c44067130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.587 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c497ed640 1 -- 172.21.15.204:0/2886166348 >> v2:172.21.15.98:3300/0 conn(0x7f3c440c9450 msgr2=0x7f3c44067670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:01.587 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c497ed640 1 --2- 172.21.15.204:0/2886166348 >> v2:172.21.15.98:3300/0 conn(0x7f3c440c9450 0x7f3c44067670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.587 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c497ed640 1 -- 172.21.15.204:0/2886166348 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4406bac0 con 0x7f3c4415b430 2024-05-27T00:15:01.587 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c48fec640 1 --2- 172.21.15.204:0/2886166348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c440737f0 0x7f3c44067130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:01.587 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c497ed640 1 --2- 172.21.15.204:0/2886166348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c4415b430 0x7f3c440732b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3c44070ad0 tx=0x7f3c4001cf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7624470cb6d9f230 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:01.588 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c3bfff640 1 -- 172.21.15.204:0/2886166348 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c400ab5f0 con 0x7f3c4415b430 2024-05-27T00:15:01.588 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c3bfff640 1 -- 172.21.15.204:0/2886166348 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c400b0070 con 0x7f3c4415b430 2024-05-27T00:15:01.588 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c3bfff640 1 -- 172.21.15.204:0/2886166348 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c400ab9b0 con 0x7f3c4415b430 2024-05-27T00:15:01.588 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c4ba51640 1 -- 172.21.15.204:0/2886166348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c4415b430 msgr2=0x7f3c440732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.589 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c4ba51640 1 --2- 172.21.15.204:0/2886166348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c4415b430 0x7f3c440732b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3c44070ad0 tx=0x7f3c4001cf80 comp rx=0 tx=0).stop 2024-05-27T00:15:01.589 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c4ba51640 1 -- 172.21.15.204:0/2886166348 shutdown_connections 2024-05-27T00:15:01.589 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c4ba51640 1 --2- 172.21.15.204:0/2886166348 >> v2:172.21.15.98:3300/0 conn(0x7f3c440c9450 0x7f3c44067670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.589 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c4ba51640 1 --2- 172.21.15.204:0/2886166348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c440737f0 0x7f3c44067130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.589 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c4ba51640 1 --2- 172.21.15.204:0/2886166348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c4415b430 0x7f3c440732b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.589 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c4ba51640 1 -- 172.21.15.204:0/2886166348 >> 172.21.15.204:0/2886166348 conn(0x7f3c4406e6b0 msgr2=0x7f3c440ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:01.589 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c4ba51640 1 -- 172.21.15.204:0/2886166348 shutdown_connections 2024-05-27T00:15:01.589 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.585+0000 7f3c4ba51640 1 -- 172.21.15.204:0/2886166348 wait complete. 2024-05-27T00:15:01.589 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 Processor -- start 2024-05-27T00:15:01.590 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 -- start start 2024-05-27T00:15:01.592 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440737f0 0x7f3c440cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.592 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c440c9450 0x7f3c44067390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.592 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c4415b430 0x7f3c4415be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.592 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4406c250 con 0x7f3c440737f0 2024-05-27T00:15:01.592 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3c44071f20 con 0x7f3c440c9450 2024-05-27T00:15:01.592 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3c441f4260 con 0x7f3c4415b430 2024-05-27T00:15:01.592 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c48fec640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c440c9450 0x7f3c44067390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.592 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c48fec640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c440c9450 0x7f3c44067390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:44496/0 (socket says 172.21.15.204:44496) 2024-05-27T00:15:01.592 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c48fec640 1 -- 172.21.15.204:0/428030516 learned_addr learned my addr 172.21.15.204:0/428030516 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c49fee640 1 --2- 172.21.15.204:0/428030516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c4415b430 0x7f3c4415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c497ed640 1 --2- 172.21.15.204:0/428030516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440737f0 0x7f3c440cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c48fec640 1 -- 172.21.15.204:0/428030516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c4415b430 msgr2=0x7f3c4415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c48fec640 1 --2- 172.21.15.204:0/428030516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c4415b430 0x7f3c4415be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c48fec640 1 -- 172.21.15.204:0/428030516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440737f0 msgr2=0x7f3c440cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c48fec640 1 --2- 172.21.15.204:0/428030516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440737f0 0x7f3c440cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c48fec640 1 -- 172.21.15.204:0/428030516 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c441f4540 con 0x7f3c440c9450 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c49fee640 1 --2- 172.21.15.204:0/428030516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c4415b430 0x7f3c4415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c48fec640 1 --2- 172.21.15.204:0/428030516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c440c9450 0x7f3c44067390 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400e980 tx=0x7f3c3400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c39ffb640 1 -- 172.21.15.204:0/428030516 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34017070 con 0x7f3c440c9450 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c39ffb640 1 -- 172.21.15.204:0/428030516 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c3400ca00 con 0x7f3c440c9450 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 -- 172.21.15.204:0/428030516 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c441f4820 con 0x7f3c440c9450 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c39ffb640 1 -- 172.21.15.204:0/428030516 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3400cdd0 con 0x7f3c440c9450 2024-05-27T00:15:01.593 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 -- 172.21.15.204:0/428030516 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c441f4de0 con 0x7f3c440c9450 2024-05-27T00:15:01.594 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c39ffb640 1 -- 172.21.15.204:0/428030516 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3c3401a070 con 0x7f3c440c9450 2024-05-27T00:15:01.594 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c39ffb640 1 -- 172.21.15.204:0/428030516 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f3c3401c870 con 0x7f3c440c9450 2024-05-27T00:15:01.594 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.589+0000 7f3c4ba51640 1 -- 172.21.15.204:0/428030516 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c44075cb0 con 0x7f3c440c9450 2024-05-27T00:15:01.596 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.593+0000 7f3c39ffb640 1 -- 172.21.15.204:0/428030516 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3c3401c450 con 0x7f3c440c9450 2024-05-27T00:15:01.664 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.661+0000 7f3c4ba51640 1 -- 172.21.15.204:0/428030516 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bac80420-b2e9-4fe6-8a5a-6502ca48305f", "id": 2} v 0) -- 0x7f3c4415ba90 con 0x7f3c440c9450 2024-05-27T00:15:01.669 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.669+0000 7f3c39ffb640 1 -- 172.21.15.204:0/428030516 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bac80420-b2e9-4fe6-8a5a-6502ca48305f", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f3c34009420 con 0x7f3c440c9450 2024-05-27T00:15:01.669 INFO:teuthology.orchestra.run.smithi204.stdout:2 2024-05-27T00:15:01.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.669+0000 7f3c4ba51640 1 -- 172.21.15.204:0/428030516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c440c9450 msgr2=0x7f3c44067390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.669+0000 7f3c4ba51640 1 --2- 172.21.15.204:0/428030516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c440c9450 0x7f3c44067390 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400e980 tx=0x7f3c3400ee40 comp rx=0 tx=0).stop 2024-05-27T00:15:01.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.669+0000 7f3c4ba51640 1 -- 172.21.15.204:0/428030516 shutdown_connections 2024-05-27T00:15:01.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.669+0000 7f3c4ba51640 1 --2- 172.21.15.204:0/428030516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c4415b430 0x7f3c4415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.669+0000 7f3c4ba51640 1 --2- 172.21.15.204:0/428030516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c440c9450 0x7f3c44067390 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.669+0000 7f3c4ba51640 1 --2- 172.21.15.204:0/428030516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c440737f0 0x7f3c440cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.669+0000 7f3c4ba51640 1 -- 172.21.15.204:0/428030516 >> 172.21.15.204:0/428030516 conn(0x7f3c4406e6b0 msgr2=0x7f3c4414ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:01.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.669+0000 7f3c4ba51640 1 -- 172.21.15.204:0/428030516 shutdown_connections 2024-05-27T00:15:01.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.669+0000 7f3c4ba51640 1 -- 172.21.15.204:0/428030516 wait complete. 2024-05-27T00:15:01.686 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 320acdd6-155a-4661-8ecc-ece9d4b785eb 3 2024-05-27T00:15:01.803 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.801+0000 7f48ca4f6640 1 Processor -- start 2024-05-27T00:15:01.804 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.801+0000 7f48ca4f6640 1 -- start start 2024-05-27T00:15:01.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.801+0000 7f48ca4f6640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f48c41517c0 0x7f48c40c9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.801+0000 7f48ca4f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48c40c8900 con 0x7f48c40c8070 2024-05-27T00:15:01.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.801+0000 7f48ca4f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48c4071a20 con 0x7f48c40c84b0 2024-05-27T00:15:01.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.801+0000 7f48ca4f6640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f48c4071d00 con 0x7f48c41517c0 2024-05-27T00:15:01.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.801+0000 7f48c3fff640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f48c41517c0 0x7f48c40c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.801+0000 7f48c8a93640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f48c40c84b0 0x7f48c4161d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:43356/0 (socket says 172.21.15.204:43356) 2024-05-27T00:15:01.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.801+0000 7f48c8a93640 1 -- 172.21.15.204:0/339314880 learned_addr learned my addr 172.21.15.204:0/339314880 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-05-27T00:15:01.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48c3fff640 1 -- 172.21.15.204:0/339314880 >> v1:172.21.15.204:6789/0 conn(0x7f48c40c84b0 legacy=0x7f48c4161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48c3fff640 1 -- 172.21.15.204:0/339314880 >> v1:172.21.15.98:6789/0 conn(0x7f48c40c8070 legacy=0x7f48c40c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:01.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48c3fff640 1 -- 172.21.15.204:0/339314880 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c4071fe0 con 0x7f48c41517c0 2024-05-27T00:15:01.807 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48c3fff640 1 --2- 172.21.15.204:0/339314880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c41517c0 0x7f48c40c9940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f48b40097c0 tx=0x7f48b401c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cf6b16b5c1b1400 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:01.807 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48c2ffd640 1 -- 172.21.15.204:0/339314880 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b40ab430 con 0x7f48c41517c0 2024-05-27T00:15:01.807 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48c2ffd640 1 -- 172.21.15.204:0/339314880 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48b40b0070 con 0x7f48c41517c0 2024-05-27T00:15:01.807 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48c2ffd640 1 -- 172.21.15.204:0/339314880 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b40ab7f0 con 0x7f48c41517c0 2024-05-27T00:15:01.808 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48ca4f6640 1 -- 172.21.15.204:0/339314880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c41517c0 msgr2=0x7f48c40c9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.808 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48ca4f6640 1 --2- 172.21.15.204:0/339314880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c41517c0 0x7f48c40c9940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f48b40097c0 tx=0x7f48b401c430 comp rx=0 tx=0).stop 2024-05-27T00:15:01.808 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48ca4f6640 1 -- 172.21.15.204:0/339314880 shutdown_connections 2024-05-27T00:15:01.808 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48ca4f6640 1 --2- 172.21.15.204:0/339314880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c41517c0 0x7f48c40c9940 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.809 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48ca4f6640 1 -- 172.21.15.204:0/339314880 >> 172.21.15.204:0/339314880 conn(0x7f48c406b470 msgr2=0x7f48c406b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:01.809 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48ca4f6640 1 -- 172.21.15.204:0/339314880 shutdown_connections 2024-05-27T00:15:01.809 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.805+0000 7f48ca4f6640 1 -- 172.21.15.204:0/339314880 wait complete. 2024-05-27T00:15:01.809 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 Processor -- start 2024-05-27T00:15:01.809 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 -- start start 2024-05-27T00:15:01.811 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c40c8070 0x7f48c41f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.811 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c40c84b0 0x7f48c4145de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48c41517c0 0x7f48c4150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48c40c9e80 con 0x7f48c40c8070 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f48c4151ba0 con 0x7f48c40c84b0 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f48c4071650 con 0x7f48c41517c0 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c37fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c40c84b0 0x7f48c4145de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c37fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c40c84b0 0x7f48c4145de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:47838/0 (socket says 172.21.15.204:47838) 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c37fe640 1 -- 172.21.15.204:0/1014393361 learned_addr learned my addr 172.21.15.204:0/1014393361 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c8a93640 1 --2- 172.21.15.204:0/1014393361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48c41517c0 0x7f48c4150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c3fff640 1 --2- 172.21.15.204:0/1014393361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c40c8070 0x7f48c41f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c37fe640 1 -- 172.21.15.204:0/1014393361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48c41517c0 msgr2=0x7f48c4150650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c37fe640 1 --2- 172.21.15.204:0/1014393361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48c41517c0 0x7f48c4150650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c37fe640 1 -- 172.21.15.204:0/1014393361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c40c8070 msgr2=0x7f48c41f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c37fe640 1 --2- 172.21.15.204:0/1014393361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c40c8070 0x7f48c41f5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c37fe640 1 -- 172.21.15.204:0/1014393361 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c40c34a0 con 0x7f48c40c84b0 2024-05-27T00:15:01.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c8a93640 1 --2- 172.21.15.204:0/1014393361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48c41517c0 0x7f48c4150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:01.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c37fe640 1 --2- 172.21.15.204:0/1014393361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c40c84b0 0x7f48c4145de0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f48b800c450 tx=0x7f48b800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:01.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c0ff9640 1 -- 172.21.15.204:0/1014393361 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8012830 con 0x7f48c40c84b0 2024-05-27T00:15:01.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c0ff9640 1 -- 172.21.15.204:0/1014393361 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48b8017070 con 0x7f48c40c84b0 2024-05-27T00:15:01.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 -- 172.21.15.204:0/1014393361 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c4070100 con 0x7f48c40c84b0 2024-05-27T00:15:01.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c0ff9640 1 -- 172.21.15.204:0/1014393361 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8012c30 con 0x7f48c40c84b0 2024-05-27T00:15:01.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 -- 172.21.15.204:0/1014393361 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c41f1600 con 0x7f48c40c84b0 2024-05-27T00:15:01.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c0ff9640 1 -- 172.21.15.204:0/1014393361 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f48b801c460 con 0x7f48c40c84b0 2024-05-27T00:15:01.814 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48c0ff9640 1 -- 172.21.15.204:0/1014393361 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f48b801bbc0 con 0x7f48c40c84b0 2024-05-27T00:15:01.814 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.809+0000 7f48ca4f6640 1 -- 172.21.15.204:0/1014393361 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48c4075cb0 con 0x7f48c40c84b0 2024-05-27T00:15:01.814 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.813+0000 7f48c0ff9640 1 -- 172.21.15.204:0/1014393361 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f48b801be10 con 0x7f48c40c84b0 2024-05-27T00:15:01.882 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.881+0000 7f48ca4f6640 1 -- 172.21.15.204:0/1014393361 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "320acdd6-155a-4661-8ecc-ece9d4b785eb", "id": 3} v 0) -- 0x7f48c406b850 con 0x7f48c40c84b0 2024-05-27T00:15:01.886 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.885+0000 7f48c0ff9640 1 -- 172.21.15.204:0/1014393361 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "320acdd6-155a-4661-8ecc-ece9d4b785eb", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f48b801b730 con 0x7f48c40c84b0 2024-05-27T00:15:01.886 INFO:teuthology.orchestra.run.smithi204.stdout:3 2024-05-27T00:15:01.887 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.885+0000 7f48ca4f6640 1 -- 172.21.15.204:0/1014393361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c40c84b0 msgr2=0x7f48c4145de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:01.887 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.885+0000 7f48ca4f6640 1 --2- 172.21.15.204:0/1014393361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c40c84b0 0x7f48c4145de0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f48b800c450 tx=0x7f48b800c910 comp rx=0 tx=0).stop 2024-05-27T00:15:01.887 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.885+0000 7f48ca4f6640 1 -- 172.21.15.204:0/1014393361 shutdown_connections 2024-05-27T00:15:01.887 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.885+0000 7f48ca4f6640 1 --2- 172.21.15.204:0/1014393361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48c41517c0 0x7f48c4150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.887 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.885+0000 7f48ca4f6640 1 --2- 172.21.15.204:0/1014393361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48c40c84b0 0x7f48c4145de0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.888 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.885+0000 7f48ca4f6640 1 --2- 172.21.15.204:0/1014393361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c40c8070 0x7f48c41f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:01.888 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.885+0000 7f48ca4f6640 1 -- 172.21.15.204:0/1014393361 >> 172.21.15.204:0/1014393361 conn(0x7f48c406b470 msgr2=0x7f48c40c52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:01.888 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.885+0000 7f48ca4f6640 1 -- 172.21.15.204:0/1014393361 shutdown_connections 2024-05-27T00:15:01.888 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:01.885+0000 7f48ca4f6640 1 -- 172.21.15.204:0/1014393361 wait complete. 2024-05-27T00:15:01.903 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76 4 2024-05-27T00:15:01.991 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:01.989+0000 7f8a7217b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T00:15:02.020 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.017+0000 7fe2981cb640 1 Processor -- start 2024-05-27T00:15:02.020 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.017+0000 7fe2981cb640 1 -- start start 2024-05-27T00:15:02.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe2981cb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe2901526e0 0x7fe2900c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe2981cb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2900c4d20 con 0x7fe290159a80 2024-05-27T00:15:02.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe2981cb640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2900c5000 con 0x7fe2900c3090 2024-05-27T00:15:02.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe2981cb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe290152ac0 con 0x7fe2901526e0 2024-05-27T00:15:02.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe296768640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fe2900c3090 0x7fe29006f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:43362/0 (socket says 172.21.15.204:43362) 2024-05-27T00:15:02.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe296768640 1 -- 172.21.15.204:0/94594830 learned_addr learned my addr 172.21.15.204:0/94594830 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-05-27T00:15:02.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe295f67640 1 --2- 172.21.15.204:0/94594830 >> v2:172.21.15.98:3300/0 conn(0x7fe2901526e0 0x7fe2900c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 894857019 0 0) 0x7fe2900c5000 con 0x7fe2900c3090 2024-05-27T00:15:02.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe27c003530 con 0x7fe2900c3090 2024-05-27T00:15:02.024 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586213931 0 0) 0x7fe2900c4d20 con 0x7fe290159a80 2024-05-27T00:15:02.024 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2900c5000 con 0x7fe290159a80 2024-05-27T00:15:02.024 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 563405977 0 0) 0x7fe27c003530 con 0x7fe2900c3090 2024-05-27T00:15:02.024 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 >> v2:172.21.15.98:3300/0 conn(0x7fe2901526e0 msgr2=0x7fe2900c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:02.024 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 --2- 172.21.15.204:0/94594830 >> v2:172.21.15.98:3300/0 conn(0x7fe2901526e0 0x7fe2900c8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.024 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 >> v1:172.21.15.98:6789/0 conn(0x7fe290159a80 legacy=0x7fe290158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.024 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2901587a0 con 0x7fe2900c3090 2024-05-27T00:15:02.024 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fe288001a00 con 0x7fe2900c3090 2024-05-27T00:15:02.024 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 >> v1:172.21.15.204:6789/0 conn(0x7fe2900c3090 legacy=0x7fe29006f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe27c003b50 0x7fe27c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe27c004810 0x7fe27c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe27c0051d0 0x7fe27c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c00bfc0 con 0x7fe27c004810 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c00c140 con 0x7fe27c003b50 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe27c00c340 con 0x7fe27c0051d0 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe295f67640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe27c003b50 0x7fe27c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe296768640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe27c0051d0 0x7fe27c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe295766640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe27c004810 0x7fe27c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe295f67640 1 -- 172.21.15.204:0/94594830 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe27c0051d0 msgr2=0x7fe27c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe295f67640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe27c0051d0 0x7fe27c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe295f67640 1 -- 172.21.15.204:0/94594830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe27c004810 msgr2=0x7fe27c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe295f67640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe27c004810 0x7fe27c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe295f67640 1 -- 172.21.15.204:0/94594830 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27c003530 con 0x7fe27c003b50 2024-05-27T00:15:02.026 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe296768640 1 -- 172.21.15.204:0/94594830 reap_dead start 2024-05-27T00:15:02.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe295f67640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe27c003b50 0x7fe27c003f50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009ae0 tx=0x7fe28c01ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=340a5465542d0e85 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:02.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c0b2070 con 0x7fe27c003b50 2024-05-27T00:15:02.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe28c0a5600 con 0x7fe27c003b50 2024-05-27T00:15:02.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.021+0000 7fe294f65640 1 -- 172.21.15.204:0/94594830 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c0096b0 con 0x7fe27c003b50 2024-05-27T00:15:02.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- 172.21.15.204:0/94594830 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe27c003b50 msgr2=0x7fe27c003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe27c003b50 0x7fe27c003f50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009ae0 tx=0x7fe28c01ca10 comp rx=0 tx=0).stop 2024-05-27T00:15:02.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- 172.21.15.204:0/94594830 shutdown_connections 2024-05-27T00:15:02.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 --2- 172.21.15.204:0/94594830 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe27c003b50 0x7fe27c003f50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- 172.21.15.204:0/94594830 >> 172.21.15.204:0/94594830 conn(0x7fe29006d680 msgr2=0x7fe29006da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- 172.21.15.204:0/94594830 shutdown_connections 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- 172.21.15.204:0/94594830 wait complete. 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 Processor -- start 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- start start 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe27c003b50 0x7fe2900c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe290159e90 0x7fe290148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe29015b6e0 0x7fe290150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe29006fff0 con 0x7fe27c003b50 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe2900c3640 con 0x7fe29015b6e0 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe290070170 con 0x7fe290159e90 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe295766640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe290159e90 0x7fe290148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe296768640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe29015b6e0 0x7fe290150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe296768640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe29015b6e0 0x7fe290150b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:47856/0 (socket says 172.21.15.204:47856) 2024-05-27T00:15:02.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe295766640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe290159e90 0x7fe290148660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.204:35648/0 (socket says 172.21.15.204:35648) 2024-05-27T00:15:02.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe296768640 1 -- 172.21.15.204:0/2576307145 learned_addr learned my addr 172.21.15.204:0/2576307145 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:02.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe295f67640 1 --2- 172.21.15.204:0/2576307145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe27c003b50 0x7fe2900c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe296768640 1 -- 172.21.15.204:0/2576307145 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe290159e90 msgr2=0x7fe290148660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe296768640 1 --2- 172.21.15.204:0/2576307145 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe290159e90 0x7fe290148660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe296768640 1 -- 172.21.15.204:0/2576307145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe27c003b50 msgr2=0x7fe2900c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe296768640 1 --2- 172.21.15.204:0/2576307145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe27c003b50 0x7fe2900c8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe296768640 1 -- 172.21.15.204:0/2576307145 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe290148ba0 con 0x7fe29015b6e0 2024-05-27T00:15:02.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe296768640 1 --2- 172.21.15.204:0/2576307145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe29015b6e0 0x7fe290150b30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe290069a30 tx=0x7fe288007bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2867fc640 1 -- 172.21.15.204:0/2576307145 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28801c070 con 0x7fe29015b6e0 2024-05-27T00:15:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2867fc640 1 -- 172.21.15.204:0/2576307145 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2880118a0 con 0x7fe29015b6e0 2024-05-27T00:15:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2867fc640 1 -- 172.21.15.204:0/2576307145 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288010040 con 0x7fe29015b6e0 2024-05-27T00:15:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- 172.21.15.204:0/2576307145 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29015af20 con 0x7fe29015b6e0 2024-05-27T00:15:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2981cb640 1 -- 172.21.15.204:0/2576307145 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2900cb2f0 con 0x7fe29015b6e0 2024-05-27T00:15:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2867fc640 1 -- 172.21.15.204:0/2576307145 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe28801f070 con 0x7fe29015b6e0 2024-05-27T00:15:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.025+0000 7fe2867fc640 1 -- 172.21.15.204:0/2576307145 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fe288006980 con 0x7fe29015b6e0 2024-05-27T00:15:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.029+0000 7fe2981cb640 1 -- 172.21.15.204:0/2576307145 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe290075cb0 con 0x7fe29015b6e0 2024-05-27T00:15:02.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.029+0000 7fe2867fc640 1 -- 172.21.15.204:0/2576307145 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe288038dd0 con 0x7fe29015b6e0 2024-05-27T00:15:02.048 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:02.045+0000 7f8a7217b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T00:15:02.099 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.097+0000 7fe2981cb640 1 -- 172.21.15.204:0/2576307145 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76", "id": 4} v 0) -- 0x7fe2900c95d0 con 0x7fe29015b6e0 2024-05-27T00:15:02.101 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:02.101+0000 7f8a7217b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T00:15:02.103 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.101+0000 7fe2867fc640 1 -- 172.21.15.204:0/2576307145 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fe288027030 con 0x7fe29015b6e0 2024-05-27T00:15:02.103 INFO:teuthology.orchestra.run.smithi204.stdout:4 2024-05-27T00:15:02.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.101+0000 7fe2981cb640 1 -- 172.21.15.204:0/2576307145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe29015b6e0 msgr2=0x7fe290150b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.101+0000 7fe2981cb640 1 --2- 172.21.15.204:0/2576307145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe29015b6e0 0x7fe290150b30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe290069a30 tx=0x7fe288007bb0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.101+0000 7fe2981cb640 1 -- 172.21.15.204:0/2576307145 shutdown_connections 2024-05-27T00:15:02.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.101+0000 7fe2981cb640 1 --2- 172.21.15.204:0/2576307145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe29015b6e0 0x7fe290150b30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.101+0000 7fe2981cb640 1 --2- 172.21.15.204:0/2576307145 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe290159e90 0x7fe290148660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.101+0000 7fe2981cb640 1 --2- 172.21.15.204:0/2576307145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe27c003b50 0x7fe2900c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.101+0000 7fe2981cb640 1 -- 172.21.15.204:0/2576307145 >> 172.21.15.204:0/2576307145 conn(0x7fe29006d680 msgr2=0x7fe2900cdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:02.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.101+0000 7fe2981cb640 1 -- 172.21.15.204:0/2576307145 shutdown_connections 2024-05-27T00:15:02.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.101+0000 7fe2981cb640 1 -- 172.21.15.204:0/2576307145 wait complete. 2024-05-27T00:15:02.119 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new b1b6c7ed-78bc-4f71-9522-78446864221c 5 2024-05-27T00:15:02.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:02.181+0000 7f8a7217b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T00:15:02.190 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:02.190+0000 7fc043f7e180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T00:15:02.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f8e45f640 1 Processor -- start 2024-05-27T00:15:02.232 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f8e45f640 1 -- start start 2024-05-27T00:15:02.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f8e45f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f8806f530 0x7f7f8806b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f8e45f640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8815a9f0 0x7f7f8815ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f8e45f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f880c8b10 con 0x7f7f8806f910 2024-05-27T00:15:02.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f8e45f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7f8815b360 con 0x7f7f8806f530 2024-05-27T00:15:02.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f8e45f640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7f880c8df0 con 0x7f7f8815a9f0 2024-05-27T00:15:02.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f8c9fc640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8815a9f0 0x7f7f8815ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f8c9fc640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8815a9f0 0x7f7f8815ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.204:35664/0 (socket says 172.21.15.204:35664) 2024-05-27T00:15:02.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f8c9fc640 1 -- 172.21.15.204:0/2552500387 learned_addr learned my addr 172.21.15.204:0/2552500387 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:02.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.229+0000 7f7f87fff640 1 --2- 172.21.15.204:0/2552500387 >> v2:172.21.15.98:3300/0 conn(0x7f7f8806f530 0x7f7f8806b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.233+0000 7f7f8c9fc640 1 -- 172.21.15.204:0/2552500387 >> v2:172.21.15.98:3300/0 conn(0x7f7f8806f530 msgr2=0x7f7f8806b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:02.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.233+0000 7f7f8c9fc640 1 --2- 172.21.15.204:0/2552500387 >> v2:172.21.15.98:3300/0 conn(0x7f7f8806f530 0x7f7f8806b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.236 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.233+0000 7f7f8c9fc640 1 -- 172.21.15.204:0/2552500387 >> v1:172.21.15.98:6789/0 conn(0x7f7f8806f910 legacy=0x7f7f8806cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.236 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.233+0000 7f7f8c9fc640 1 -- 172.21.15.204:0/2552500387 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f880ce170 con 0x7f7f8815a9f0 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.233+0000 7f7f8c9fc640 1 --2- 172.21.15.204:0/2552500387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8815a9f0 0x7f7f8815ae20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009e50 tx=0x7f7f8001f210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa7e151eed44cb0a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.233+0000 7f7f877fe640 1 -- 172.21.15.204:0/2552500387 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f800b4070 con 0x7f7f8815a9f0 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.233+0000 7f7f877fe640 1 -- 172.21.15.204:0/2552500387 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f800af400 con 0x7f7f8815a9f0 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.233+0000 7f7f877fe640 1 -- 172.21.15.204:0/2552500387 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f800af7d0 con 0x7f7f8815a9f0 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.233+0000 7f7f8e45f640 1 -- 172.21.15.204:0/2552500387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8815a9f0 msgr2=0x7f7f8815ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.233+0000 7f7f8e45f640 1 --2- 172.21.15.204:0/2552500387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8815a9f0 0x7f7f8815ae20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7f80009e50 tx=0x7f7f8001f210 comp rx=0 tx=0).stop 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- 172.21.15.204:0/2552500387 shutdown_connections 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 --2- 172.21.15.204:0/2552500387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8815a9f0 0x7f7f8815ae20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 --2- 172.21.15.204:0/2552500387 >> v2:172.21.15.98:3300/0 conn(0x7f7f8806f530 0x7f7f8806b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- 172.21.15.204:0/2552500387 >> 172.21.15.204:0/2552500387 conn(0x7f7f8806d680 msgr2=0x7f7f8806da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- 172.21.15.204:0/2552500387 shutdown_connections 2024-05-27T00:15:02.238 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- 172.21.15.204:0/2552500387 wait complete. 2024-05-27T00:15:02.239 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f8a7217b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T00:15:02.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 Processor -- start 2024-05-27T00:15:02.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- start start 2024-05-27T00:15:02.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8806f530 0x7f7f880b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8806f910 0x7f7f880c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f8815a9f0 0x7f7f880c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f88070220 con 0x7f7f8806f530 2024-05-27T00:15:02.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7f880ce730 con 0x7f7f8815a9f0 2024-05-27T00:15:02.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7f88157b00 con 0x7f7f8806f910 2024-05-27T00:15:02.241 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f7ffff640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8806f910 0x7f7f880c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f7ffff640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8806f910 0x7f7f880c7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.204:35680/0 (socket says 172.21.15.204:35680) 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f7ffff640 1 -- 172.21.15.204:0/4269202164 learned_addr learned my addr 172.21.15.204:0/4269202164 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8c9fc640 1 --2- 172.21.15.204:0/4269202164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f8815a9f0 0x7f7f880c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f87fff640 1 --2- 172.21.15.204:0/4269202164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8806f530 0x7f7f880b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f7ffff640 1 -- 172.21.15.204:0/4269202164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f8815a9f0 msgr2=0x7f7f880c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f7ffff640 1 --2- 172.21.15.204:0/4269202164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f8815a9f0 0x7f7f880c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f7ffff640 1 -- 172.21.15.204:0/4269202164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8806f530 msgr2=0x7f7f880b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f7ffff640 1 --2- 172.21.15.204:0/4269202164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8806f530 0x7f7f880b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f7ffff640 1 -- 172.21.15.204:0/4269202164 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f880b17b0 con 0x7f7f8806f910 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f87fff640 1 --2- 172.21.15.204:0/4269202164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8806f530 0x7f7f880b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f7ffff640 1 --2- 172.21.15.204:0/4269202164 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8806f910 0x7f7f880c7b30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000c450 tx=0x7f7f7000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f857fa640 1 -- 172.21.15.204:0/4269202164 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700052d0 con 0x7f7f8806f910 2024-05-27T00:15:02.242 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f857fa640 1 -- 172.21.15.204:0/4269202164 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f70005470 con 0x7f7f8806f910 2024-05-27T00:15:02.243 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- 172.21.15.204:0/4269202164 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f881f0df0 con 0x7f7f8806f910 2024-05-27T00:15:02.243 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f857fa640 1 -- 172.21.15.204:0/4269202164 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f70012850 con 0x7f7f8806f910 2024-05-27T00:15:02.243 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- 172.21.15.204:0/4269202164 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f880b1d30 con 0x7f7f8806f910 2024-05-27T00:15:02.243 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f857fa640 1 -- 172.21.15.204:0/4269202164 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7f700129f0 con 0x7f7f8806f910 2024-05-27T00:15:02.243 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f857fa640 1 -- 172.21.15.204:0/4269202164 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f7f70003b80 con 0x7f7f8806f910 2024-05-27T00:15:02.243 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.237+0000 7f7f8e45f640 1 -- 172.21.15.204:0/4269202164 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f88075cb0 con 0x7f7f8806f910 2024-05-27T00:15:02.243 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.241+0000 7f7f857fa640 1 -- 172.21.15.204:0/4269202164 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7f70012ce0 con 0x7f7f8806f910 2024-05-27T00:15:02.253 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:02.250+0000 7fc043f7e180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T00:15:02.310 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.309+0000 7f7f8e45f640 1 -- 172.21.15.204:0/4269202164 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b1b6c7ed-78bc-4f71-9522-78446864221c", "id": 5} v 0) -- 0x7f7f880b1f00 con 0x7f7f8806f910 2024-05-27T00:15:02.314 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:02.314+0000 7fc043f7e180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T00:15:02.314 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.313+0000 7f7f857fa640 1 -- 172.21.15.204:0/4269202164 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b1b6c7ed-78bc-4f71-9522-78446864221c", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f7f70003600 con 0x7f7f8806f910 2024-05-27T00:15:02.315 INFO:teuthology.orchestra.run.smithi204.stdout:5 2024-05-27T00:15:02.316 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.313+0000 7f7f8e45f640 1 -- 172.21.15.204:0/4269202164 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8806f910 msgr2=0x7f7f880c7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.316 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.313+0000 7f7f8e45f640 1 --2- 172.21.15.204:0/4269202164 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8806f910 0x7f7f880c7b30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000c450 tx=0x7f7f7000c910 comp rx=0 tx=0).stop 2024-05-27T00:15:02.316 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.313+0000 7f7f8e45f640 1 -- 172.21.15.204:0/4269202164 shutdown_connections 2024-05-27T00:15:02.316 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.313+0000 7f7f8e45f640 1 --2- 172.21.15.204:0/4269202164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f8815a9f0 0x7f7f880c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.316 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.313+0000 7f7f8e45f640 1 --2- 172.21.15.204:0/4269202164 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f8806f910 0x7f7f880c7b30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.316 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.313+0000 7f7f8e45f640 1 --2- 172.21.15.204:0/4269202164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f8806f530 0x7f7f880b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.316 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.313+0000 7f7f8e45f640 1 -- 172.21.15.204:0/4269202164 >> 172.21.15.204:0/4269202164 conn(0x7f7f8806d680 msgr2=0x7f7f880c9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:02.316 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.313+0000 7f7f8e45f640 1 -- 172.21.15.204:0/4269202164 shutdown_connections 2024-05-27T00:15:02.316 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.313+0000 7f7f8e45f640 1 -- 172.21.15.204:0/4269202164 wait complete. 2024-05-27T00:15:02.318 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:02.317+0000 7f8a7217b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T00:15:02.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new aeda16ff-23e1-4f54-8053-b375e68393b2 6 2024-05-27T00:15:02.374 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:02.373+0000 7f8a7217b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T00:15:02.402 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:02.402+0000 7fc043f7e180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T00:15:02.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe724c28640 1 Processor -- start 2024-05-27T00:15:02.448 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe724c28640 1 -- start start 2024-05-27T00:15:02.449 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe724c28640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe720159a80 0x7fe7200c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe724c28640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe7200c3090 0x7fe72006f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe724c28640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7200c4d20 con 0x7fe7201526e0 2024-05-27T00:15:02.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe724c28640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe720152ac0 con 0x7fe7200c3090 2024-05-27T00:15:02.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe724c28640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fe7200c3640 con 0x7fe720159a80 2024-05-27T00:15:02.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe71dd9b640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe720159a80 0x7fe7200c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe71dd9b640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe720159a80 0x7fe7200c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:47866/0 (socket says 172.21.15.204:47866) 2024-05-27T00:15:02.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe71dd9b640 1 -- 172.21.15.204:0/1633337400 learned_addr learned my addr 172.21.15.204:0/1633337400 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:02.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe71ed9d640 1 --2- 172.21.15.204:0/1633337400 >> v2:172.21.15.98:3300/0 conn(0x7fe7200c3090 0x7fe72006f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe71dd9b640 1 -- 172.21.15.204:0/1633337400 >> v2:172.21.15.98:3300/0 conn(0x7fe7200c3090 msgr2=0x7fe72006f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:02.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe71dd9b640 1 --2- 172.21.15.204:0/1633337400 >> v2:172.21.15.98:3300/0 conn(0x7fe7200c3090 0x7fe72006f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.451 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.445+0000 7fe71dd9b640 1 -- 172.21.15.204:0/1633337400 >> v1:172.21.15.98:6789/0 conn(0x7fe7201526e0 legacy=0x7fe720158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.451 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe71dd9b640 1 -- 172.21.15.204:0/1633337400 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7200c5000 con 0x7fe720159a80 2024-05-27T00:15:02.451 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe71dd9b640 1 --2- 172.21.15.204:0/1633337400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 0x7fe7200c8a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe7200b37d0 tx=0x7fe71401ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e92d407e0f18dfbd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:02.451 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe71d59a640 1 -- 172.21.15.204:0/1633337400 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7140b5070 con 0x7fe720159a80 2024-05-27T00:15:02.451 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe71d59a640 1 -- 172.21.15.204:0/1633337400 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7140a8a10 con 0x7fe720159a80 2024-05-27T00:15:02.451 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe71d59a640 1 -- 172.21.15.204:0/1633337400 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7140a8de0 con 0x7fe720159a80 2024-05-27T00:15:02.452 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe724c28640 1 -- 172.21.15.204:0/1633337400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 msgr2=0x7fe7200c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.452 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe724c28640 1 --2- 172.21.15.204:0/1633337400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 0x7fe7200c8a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe7200b37d0 tx=0x7fe71401ed10 comp rx=0 tx=0).stop 2024-05-27T00:15:02.453 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe724c28640 1 -- 172.21.15.204:0/1633337400 shutdown_connections 2024-05-27T00:15:02.453 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe724c28640 1 --2- 172.21.15.204:0/1633337400 >> v2:172.21.15.98:3300/0 conn(0x7fe7200c3090 0x7fe72006f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.453 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe724c28640 1 --2- 172.21.15.204:0/1633337400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 0x7fe7200c8a70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.453 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe724c28640 1 -- 172.21.15.204:0/1633337400 >> 172.21.15.204:0/1633337400 conn(0x7fe72006d680 msgr2=0x7fe72006da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:02.453 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe724c28640 1 -- 172.21.15.204:0/1633337400 shutdown_connections 2024-05-27T00:15:02.453 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.449+0000 7fe724c28640 1 -- 172.21.15.204:0/1633337400 wait complete. 2024-05-27T00:15:02.455 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 Processor -- start 2024-05-27T00:15:02.455 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 -- start start 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7200c3090 0x7fe720157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7201526e0 0x7fe7200cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 0x7fe7201f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe72006fff0 con 0x7fe7200c3090 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe720158a80 con 0x7fe720159a80 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe720070170 con 0x7fe7201526e0 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71ed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 0x7fe7201f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71dd9b640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7201526e0 0x7fe7200cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71ed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 0x7fe7201f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:47880/0 (socket says 172.21.15.204:47880) 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71ed9d640 1 -- 172.21.15.204:0/4013619498 learned_addr learned my addr 172.21.15.204:0/4013619498 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71e59c640 1 --2- 172.21.15.204:0/4013619498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7200c3090 0x7fe720157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71ed9d640 1 -- 172.21.15.204:0/4013619498 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7201526e0 msgr2=0x7fe7200cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.456 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71ed9d640 1 --2- 172.21.15.204:0/4013619498 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7201526e0 0x7fe7200cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71ed9d640 1 -- 172.21.15.204:0/4013619498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7200c3090 msgr2=0x7fe720157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71ed9d640 1 --2- 172.21.15.204:0/4013619498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7200c3090 0x7fe720157fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71ed9d640 1 -- 172.21.15.204:0/4013619498 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe720068f70 con 0x7fe720159a80 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe71ed9d640 1 --2- 172.21.15.204:0/4013619498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 0x7fe7201f3f90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe718007ea0 tx=0x7fe718007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe70effd640 1 -- 172.21.15.204:0/4013619498 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718014620 con 0x7fe720159a80 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe70effd640 1 -- 172.21.15.204:0/4013619498 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe718019070 con 0x7fe720159a80 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe70effd640 1 -- 172.21.15.204:0/4013619498 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718014a20 con 0x7fe720159a80 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 -- 172.21.15.204:0/4013619498 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe720156310 con 0x7fe720159a80 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 -- 172.21.15.204:0/4013619498 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7201565b0 con 0x7fe720159a80 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe70effd640 1 -- 172.21.15.204:0/4013619498 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe718014bc0 con 0x7fe720159a80 2024-05-27T00:15:02.457 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe70effd640 1 -- 172.21.15.204:0/4013619498 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fe71801d9b0 con 0x7fe720159a80 2024-05-27T00:15:02.458 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.453+0000 7fe724c28640 1 -- 172.21.15.204:0/4013619498 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe720075cb0 con 0x7fe720159a80 2024-05-27T00:15:02.458 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.457+0000 7fe70effd640 1 -- 172.21.15.204:0/4013619498 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe71801dbb0 con 0x7fe720159a80 2024-05-27T00:15:02.463 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:02.462+0000 7fc043f7e180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T00:15:02.527 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.525+0000 7fe724c28640 1 -- 172.21.15.204:0/4013619498 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "aeda16ff-23e1-4f54-8053-b375e68393b2", "id": 6} v 0) -- 0x7fe72006fdd0 con 0x7fe720159a80 2024-05-27T00:15:02.529 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:02.529+0000 7f8a7217b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T00:15:02.532 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.529+0000 7fe70effd640 1 -- 172.21.15.204:0/4013619498 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aeda16ff-23e1-4f54-8053-b375e68393b2", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fe71801dd90 con 0x7fe720159a80 2024-05-27T00:15:02.532 INFO:teuthology.orchestra.run.smithi204.stdout:6 2024-05-27T00:15:02.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.533+0000 7fe724c28640 1 -- 172.21.15.204:0/4013619498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 msgr2=0x7fe7201f3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.533 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.533+0000 7fe724c28640 1 --2- 172.21.15.204:0/4013619498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 0x7fe7201f3f90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe718007ea0 tx=0x7fe718007f90 comp rx=0 tx=0).stop 2024-05-27T00:15:02.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.533+0000 7fe724c28640 1 -- 172.21.15.204:0/4013619498 shutdown_connections 2024-05-27T00:15:02.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.533+0000 7fe724c28640 1 --2- 172.21.15.204:0/4013619498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe720159a80 0x7fe7201f3f90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.533+0000 7fe724c28640 1 --2- 172.21.15.204:0/4013619498 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7201526e0 0x7fe7200cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.533+0000 7fe724c28640 1 --2- 172.21.15.204:0/4013619498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7200c3090 0x7fe720157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.533+0000 7fe724c28640 1 -- 172.21.15.204:0/4013619498 >> 172.21.15.204:0/4013619498 conn(0x7fe72006d680 msgr2=0x7fe720151310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:02.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.533+0000 7fe724c28640 1 -- 172.21.15.204:0/4013619498 shutdown_connections 2024-05-27T00:15:02.534 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.533+0000 7fe724c28640 1 -- 172.21.15.204:0/4013619498 wait complete. 2024-05-27T00:15:02.548 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 5fcb4aee-a2ad-47b3-a75c-d7a531602e85 7 2024-05-27T00:15:02.553 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:02.550+0000 7fc043f7e180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T00:15:02.593 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:02.593+0000 7f8a7217b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T00:15:02.615 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:02.614+0000 7fc043f7e180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T00:15:02.712 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.709+0000 7fa45ca33640 1 Processor -- start 2024-05-27T00:15:02.712 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.709+0000 7fa45ca33640 1 -- start start 2024-05-27T00:15:02.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45ca33640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa4581518f0 0x7fa458157c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45ca33640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa458151cd0 con 0x7fa45806ca80 2024-05-27T00:15:02.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45ca33640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa458072f20 con 0x7fa4580c8c90 2024-05-27T00:15:02.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45ca33640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa458072120 con 0x7fa4581518f0 2024-05-27T00:15:02.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa456d9d640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa4580c8c90 0x7fa458161ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:43376/0 (socket says 172.21.15.204:43376) 2024-05-27T00:15:02.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa456d9d640 1 -- 172.21.15.204:0/2910668327 learned_addr learned my addr 172.21.15.204:0/2910668327 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-05-27T00:15:02.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45659c640 1 --2- 172.21.15.204:0/2910668327 >> v2:172.21.15.98:3300/0 conn(0x7fa4581518f0 0x7fa458157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2698701875 0 0) 0x7fa458072f20 con 0x7fa4580c8c90 2024-05-27T00:15:02.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa444005740 con 0x7fa4580c8c90 2024-05-27T00:15:02.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2632833038 0 0) 0x7fa444005740 con 0x7fa4580c8c90 2024-05-27T00:15:02.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 >> v2:172.21.15.98:3300/0 conn(0x7fa4581518f0 msgr2=0x7fa458157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:02.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 --2- 172.21.15.204:0/2910668327 >> v2:172.21.15.98:3300/0 conn(0x7fa4581518f0 0x7fa458157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 >> v1:172.21.15.98:6789/0 conn(0x7fa45806ca80 legacy=0x7fa45806f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa458073200 con 0x7fa4580c8c90 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fa450001a00 con 0x7fa4580c8c90 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 >> v1:172.21.15.204:6789/0 conn(0x7fa4580c8c90 legacy=0x7fa458161ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa444005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4440065d0 0x7fa444006aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4440072a0 0x7fa44400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa44400e1b0 con 0x7fa4440065d0 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa44400e360 con 0x7fa4440072a0 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa44400e560 con 0x7fa444005bd0 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45659c640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa444005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa456d9d640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4440072a0 0x7fa44400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa44ffff640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4440065d0 0x7fa444006aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45659c640 1 -- 172.21.15.204:0/2910668327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4440072a0 msgr2=0x7fa44400dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:02.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45659c640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4440072a0 0x7fa44400dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45659c640 1 -- 172.21.15.204:0/2910668327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4440065d0 msgr2=0x7fa444006aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:02.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45659c640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4440065d0 0x7fa444006aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45659c640 1 -- 172.21.15.204:0/2910668327 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa444005740 con 0x7fa444005bd0 2024-05-27T00:15:02.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa456d9d640 1 -- 172.21.15.204:0/2910668327 reap_dead start 2024-05-27T00:15:02.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa44ffff640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4440065d0 0x7fa444006aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:02.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa45659c640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa444005fb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa448009ae0 tx=0x7fa44801ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ba39c273635c51c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:02.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4480b2070 con 0x7fa444005bd0 2024-05-27T00:15:02.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4480a5680 con 0x7fa444005bd0 2024-05-27T00:15:02.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.713+0000 7fa455d9b640 1 -- 172.21.15.204:0/2910668327 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4480096b0 con 0x7fa444005bd0 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- 172.21.15.204:0/2910668327 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 msgr2=0x7fa444005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa444005fb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa448009ae0 tx=0x7fa44801ca90 comp rx=0 tx=0).stop 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- 172.21.15.204:0/2910668327 shutdown_connections 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 --2- 172.21.15.204:0/2910668327 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa444005fb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- 172.21.15.204:0/2910668327 >> 172.21.15.204:0/2910668327 conn(0x7fa45806e6b0 msgr2=0x7fa4580ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- 172.21.15.204:0/2910668327 shutdown_connections 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- 172.21.15.204:0/2910668327 wait complete. 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 Processor -- start 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- start start 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa45814c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4581e35b0 0x7fa45814f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa45814ceb0 0x7fa45814f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4580737c0 con 0x7fa4581e35b0 2024-05-27T00:15:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa458073aa0 con 0x7fa45814ceb0 2024-05-27T00:15:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa458157720 con 0x7fa444005bd0 2024-05-27T00:15:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45659c640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa45814c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45659c640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa45814c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.204:35706/0 (socket says 172.21.15.204:35706) 2024-05-27T00:15:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45659c640 1 -- 172.21.15.204:0/2010105212 learned_addr learned my addr 172.21.15.204:0/2010105212 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-27T00:15:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa456d9d640 1 --2- 172.21.15.204:0/2010105212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa45814ceb0 0x7fa45814f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa44ffff640 1 --2- 172.21.15.204:0/2010105212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4581e35b0 0x7fa45814f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:02.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45659c640 1 -- 172.21.15.204:0/2010105212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa45814ceb0 msgr2=0x7fa45814f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45659c640 1 --2- 172.21.15.204:0/2010105212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa45814ceb0 0x7fa45814f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45659c640 1 -- 172.21.15.204:0/2010105212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4581e35b0 msgr2=0x7fa45814f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45659c640 1 --2- 172.21.15.204:0/2010105212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4581e35b0 0x7fa45814f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45659c640 1 -- 172.21.15.204:0/2010105212 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4581ecdb0 con 0x7fa444005bd0 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa44ffff640 1 --2- 172.21.15.204:0/2010105212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4581e35b0 0x7fa45814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45659c640 1 --2- 172.21.15.204:0/2010105212 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa45814c970 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa4480b0040 tx=0x7fa448019830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa44effd640 1 -- 172.21.15.204:0/2010105212 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4480b2040 con 0x7fa444005bd0 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa44effd640 1 -- 172.21.15.204:0/2010105212 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa44800f000 con 0x7fa444005bd0 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- 172.21.15.204:0/2010105212 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa45814dca0 con 0x7fa444005bd0 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa44effd640 1 -- 172.21.15.204:0/2010105212 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4480096b0 con 0x7fa444005bd0 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa45ca33640 1 -- 172.21.15.204:0/2010105212 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa45814d9c0 con 0x7fa444005bd0 2024-05-27T00:15:02.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa44effd640 1 -- 172.21.15.204:0/2010105212 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa4480bf030 con 0x7fa444005bd0 2024-05-27T00:15:02.727 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.717+0000 7fa44effd640 1 -- 172.21.15.204:0/2010105212 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fa448006020 con 0x7fa444005bd0 2024-05-27T00:15:02.727 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.721+0000 7fa45ca33640 1 -- 172.21.15.204:0/2010105212 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa458075cb0 con 0x7fa444005bd0 2024-05-27T00:15:02.727 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.721+0000 7fa44effd640 1 -- 172.21.15.204:0/2010105212 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa4480c6060 con 0x7fa444005bd0 2024-05-27T00:15:02.780 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:02.778+0000 7fc043f7e180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T00:15:02.791 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.789+0000 7fa45ca33640 1 -- 172.21.15.204:0/2010105212 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5fcb4aee-a2ad-47b3-a75c-d7a531602e85", "id": 7} v 0) -- 0x7fa4580c3d60 con 0x7fa444005bd0 2024-05-27T00:15:02.796 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.793+0000 7fa44effd640 1 -- 172.21.15.204:0/2010105212 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5fcb4aee-a2ad-47b3-a75c-d7a531602e85", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fa4480167a0 con 0x7fa444005bd0 2024-05-27T00:15:02.796 INFO:teuthology.orchestra.run.smithi204.stdout:7 2024-05-27T00:15:02.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.793+0000 7fa45ca33640 1 -- 172.21.15.204:0/2010105212 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 msgr2=0x7fa45814c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:02.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.793+0000 7fa45ca33640 1 --2- 172.21.15.204:0/2010105212 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa45814c970 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa4480b0040 tx=0x7fa448019830 comp rx=0 tx=0).stop 2024-05-27T00:15:02.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.793+0000 7fa45ca33640 1 -- 172.21.15.204:0/2010105212 shutdown_connections 2024-05-27T00:15:02.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.793+0000 7fa45ca33640 1 --2- 172.21.15.204:0/2010105212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa45814ceb0 0x7fa45814f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.793+0000 7fa45ca33640 1 --2- 172.21.15.204:0/2010105212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4581e35b0 0x7fa45814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.793+0000 7fa45ca33640 1 --2- 172.21.15.204:0/2010105212 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa444005bd0 0x7fa45814c970 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:02.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.793+0000 7fa45ca33640 1 -- 172.21.15.204:0/2010105212 >> 172.21.15.204:0/2010105212 conn(0x7fa45806e6b0 msgr2=0x7fa45806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:02.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.793+0000 7fa45ca33640 1 -- 172.21.15.204:0/2010105212 shutdown_connections 2024-05-27T00:15:02.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-27T00:15:02.793+0000 7fa45ca33640 1 -- 172.21.15.204:0/2010105212 wait complete. 2024-05-27T00:15:02.812 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-27T00:15:02.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-27T00:15:02.816 INFO:tasks.ceph.osd.0:Started 2024-05-27T00:15:02.816 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-27T00:15:02.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-27T00:15:02.820 INFO:tasks.ceph.osd.1:Started 2024-05-27T00:15:02.820 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-27T00:15:02.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-27T00:15:02.823 INFO:tasks.ceph.osd.2:Started 2024-05-27T00:15:02.823 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-27T00:15:02.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-27T00:15:02.827 INFO:tasks.ceph.osd.3:Started 2024-05-27T00:15:02.828 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-27T00:15:02.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-27T00:15:02.830 INFO:tasks.ceph.osd.4:Started 2024-05-27T00:15:02.830 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-27T00:15:02.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-27T00:15:02.833 INFO:tasks.ceph.osd.5:Started 2024-05-27T00:15:02.834 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-27T00:15:02.834 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-27T00:15:02.836 INFO:tasks.ceph.osd.6:Started 2024-05-27T00:15:02.837 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-27T00:15:02.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-27T00:15:02.839 INFO:tasks.ceph.osd.7:Started 2024-05-27T00:15:02.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:02.857 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:02.854+0000 7fc043f7e180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T00:15:03.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.006+0000 7f5183200640 1 Processor -- start 2024-05-27T00:15:03.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.006+0000 7f5183200640 1 -- start start 2024-05-27T00:15:03.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f5183200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f517c158e60 0x7f517c164000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f5183200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f517c15be40 con 0x7f517c0cb7e0 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f5183200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f517c0c50e0 con 0x7f517c152a50 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f5183200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f517c0c8a00 con 0x7f517c158e60 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f5173e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f517c0cb7e0 0x7f517c158750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52176/0 (socket says 172.21.15.98:52176) 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f5173e00640 1 -- 172.21.15.98:0/164150581 learned_addr learned my addr 172.21.15.98:0/164150581 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f5181200640 1 --2- 172.21.15.98:0/164150581 >> v2:172.21.15.204:3300/0 conn(0x7f517c158e60 0x7f517c164000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f517b400640 1 -- 172.21.15.98:0/164150581 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 214017494 0 0) 0x7f517c15be40 con 0x7f517c0cb7e0 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f517b400640 1 -- 172.21.15.98:0/164150581 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5160003530 con 0x7f517c0cb7e0 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f517b400640 1 -- 172.21.15.98:0/164150581 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3796049570 0 0) 0x7f517c0c50e0 con 0x7f517c152a50 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f517b400640 1 -- 172.21.15.98:0/164150581 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f517c15be40 con 0x7f517c152a50 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f5181200640 1 -- 172.21.15.98:0/164150581 >> v1:172.21.15.204:6789/0 conn(0x7f517c152a50 legacy=0x7f517c0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f5181200640 1 -- 172.21.15.98:0/164150581 >> v1:172.21.15.98:6789/0 conn(0x7f517c0cb7e0 legacy=0x7f517c158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:03.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.010+0000 7f5181200640 1 -- 172.21.15.98:0/164150581 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517c0c8c50 con 0x7f517c158e60 2024-05-27T00:15:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5181200640 1 --2- 172.21.15.98:0/164150581 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f517c158e60 0x7f517c164000 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5174009d90 tx=0x7f517401c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbcfa1bba9cb143f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f517b400640 1 -- 172.21.15.98:0/164150581 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51740b2070 con 0x7f517c158e60 2024-05-27T00:15:03.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f517b400640 1 -- 172.21.15.98:0/164150581 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51740a5970 con 0x7f517c158e60 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f517b400640 1 -- 172.21.15.98:0/164150581 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51740a5d40 con 0x7f517c158e60 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 -- 172.21.15.98:0/164150581 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f517c158e60 msgr2=0x7f517c164000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 --2- 172.21.15.98:0/164150581 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f517c158e60 0x7f517c164000 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5174009d90 tx=0x7f517401c8b0 comp rx=0 tx=0).stop 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 -- 172.21.15.98:0/164150581 shutdown_connections 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 --2- 172.21.15.98:0/164150581 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f517c158e60 0x7f517c164000 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 -- 172.21.15.98:0/164150581 >> 172.21.15.98:0/164150581 conn(0x7f517c072f90 msgr2=0x7f517c0733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 -- 172.21.15.98:0/164150581 shutdown_connections 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 -- 172.21.15.98:0/164150581 wait complete. 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 Processor -- start 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 -- start start 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f517c0cb7e0 0x7f517c1598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:03.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517c152a50 0x7f517c1577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f517c158e60 0x7f517c14b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f517c152e30 con 0x7f517c152a50 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f517c0cb570 con 0x7f517c158e60 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.014+0000 7f5183200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f517c0c9c10 con 0x7f517c0cb7e0 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5173e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517c152a50 0x7f517c1577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f517be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f517c0cb7e0 0x7f517c1598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f517be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f517c0cb7e0 0x7f517c1598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:54934/0 (socket says 172.21.15.98:54934) 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f517be00640 1 -- 172.21.15.98:0/2338510230 learned_addr learned my addr 172.21.15.98:0/2338510230 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5181200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f517c158e60 0x7f517c14b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f517be00640 1 -- 172.21.15.98:0/2338510230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f517c158e60 msgr2=0x7f517c14b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f517be00640 1 --2- 172.21.15.98:0/2338510230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f517c158e60 0x7f517c14b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f517be00640 1 -- 172.21.15.98:0/2338510230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517c152a50 msgr2=0x7f517c1577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:03.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f517be00640 1 --2- 172.21.15.98:0/2338510230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517c152a50 0x7f517c1577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f517be00640 1 -- 172.21.15.98:0/2338510230 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517c0c5e30 con 0x7f517c0cb7e0 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5173e00640 1 --2- 172.21.15.98:0/2338510230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517c152a50 0x7f517c1577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f517be00640 1 --2- 172.21.15.98:0/2338510230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f517c0cb7e0 0x7f517c1598b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f516c00c4c0 tx=0x7f516c00c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5178c00640 1 -- 172.21.15.98:0/2338510230 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c0052d0 con 0x7f517c0cb7e0 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5178c00640 1 -- 172.21.15.98:0/2338510230 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f516c005470 con 0x7f517c0cb7e0 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5183200640 1 -- 172.21.15.98:0/2338510230 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f517c15abf0 con 0x7f517c0cb7e0 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5178c00640 1 -- 172.21.15.98:0/2338510230 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c012870 con 0x7f517c0cb7e0 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5183200640 1 -- 172.21.15.98:0/2338510230 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f517c0b3bd0 con 0x7f517c0cb7e0 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5178c00640 1 -- 172.21.15.98:0/2338510230 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f516c012a10 con 0x7f517c0cb7e0 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5178c00640 1 -- 172.21.15.98:0/2338510230 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f516c012d00 con 0x7f517c0cb7e0 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.018+0000 7f5183200640 1 -- 172.21.15.98:0/2338510230 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f517c075cb0 con 0x7f517c0cb7e0 2024-05-27T00:15:03.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.022+0000 7f5178c00640 1 -- 172.21.15.98:0/2338510230 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f516c003600 con 0x7f517c0cb7e0 2024-05-27T00:15:03.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5183200640 1 -- 172.21.15.98:0/2338510230 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f517c0ca710 con 0x7f517c0cb7e0 2024-05-27T00:15:03.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5178c00640 1 -- 172.21.15.98:0/2338510230 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f516c0037e0 con 0x7f517c0cb7e0 2024-05-27T00:15:03.080 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:03.080 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:02.792649+0000","last_up_change":"0.000000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5183200640 1 -- 172.21.15.98:0/2338510230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f517c0cb7e0 msgr2=0x7f517c1598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5183200640 1 --2- 172.21.15.98:0/2338510230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f517c0cb7e0 0x7f517c1598b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f516c00c4c0 tx=0x7f516c00c980 comp rx=0 tx=0).stop 2024-05-27T00:15:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5183200640 1 -- 172.21.15.98:0/2338510230 shutdown_connections 2024-05-27T00:15:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5183200640 1 --2- 172.21.15.98:0/2338510230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f517c158e60 0x7f517c14b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5183200640 1 --2- 172.21.15.98:0/2338510230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f517c152a50 0x7f517c1577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5183200640 1 --2- 172.21.15.98:0/2338510230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f517c0cb7e0 0x7f517c1598b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5183200640 1 -- 172.21.15.98:0/2338510230 >> 172.21.15.98:0/2338510230 conn(0x7f517c072f90 msgr2=0x7f517c073370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5183200640 1 -- 172.21.15.98:0/2338510230 shutdown_connections 2024-05-27T00:15:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:03.078+0000 7f5183200640 1 -- 172.21.15.98:0/2338510230 wait complete. 2024-05-27T00:15:03.099 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-27T00:15:03.099 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-27T00:15:03.226 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-05-27T00:15:03.225+0000 7f43c7d0b7c0 -1 Falling back to public interface 2024-05-27T00:15:03.230 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-05-27T00:15:03.229+0000 7ff17ff8b7c0 -1 Falling back to public interface 2024-05-27T00:15:03.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-05-27T00:15:03.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-05-27T00:15:03.246 INFO:tasks.ceph.mgr.x.smithi204.stderr: from numpy import show_config as show_numpy_config 2024-05-27T00:15:03.270 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-05-27T00:15:03.269+0000 7f0cb86557c0 -1 Falling back to public interface 2024-05-27T00:15:03.285 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:03.285+0000 7f8a7217b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T00:15:03.424 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:03.421+0000 7f8a7217b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T00:15:03.467 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-05-27T00:15:03.465+0000 7fb604f527c0 -1 Falling back to public interface 2024-05-27T00:15:03.479 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-05-27T00:15:03.478+0000 7fc3b14b87c0 -1 Falling back to public interface 2024-05-27T00:15:03.487 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-05-27T00:15:03.486+0000 7fda453437c0 -1 Falling back to public interface 2024-05-27T00:15:03.503 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-05-27T00:15:03.502+0000 7fc36a5517c0 -1 Falling back to public interface 2024-05-27T00:15:03.503 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-05-27T00:15:03.502+0000 7f8afbede7c0 -1 Falling back to public interface 2024-05-27T00:15:03.536 INFO:tasks.ceph.mgr.y.smithi098.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-05-27T00:15:03.536 INFO:tasks.ceph.mgr.y.smithi098.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-05-27T00:15:03.536 INFO:tasks.ceph.mgr.y.smithi098.stderr: from numpy import show_config as show_numpy_config 2024-05-27T00:15:03.576 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:03.574+0000 7fc043f7e180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T00:15:03.581 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:03.581+0000 7f8a7217b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T00:15:03.644 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:03.641+0000 7f8a7217b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T00:15:03.723 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:03.722+0000 7fc043f7e180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T00:15:03.777 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:03.773+0000 7f8a7217b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T00:15:03.901 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:03.898+0000 7fc043f7e180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T00:15:03.904 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:03.976 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:03.974+0000 7fc043f7e180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T00:15:04.031 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b74600640 1 Processor -- start 2024-05-27T00:15:04.031 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b74600640 1 -- start start 2024-05-27T00:15:04.033 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b74600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b6c152cb0 0x7f0b6c158b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:04.033 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b74600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b6c0c9340 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.033 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b74600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b6c15c840 con 0x7f0b6c073180 2024-05-27T00:15:04.033 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b74600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0b6c0c95e0 con 0x7f0b6c152cb0 2024-05-27T00:15:04.033 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b72600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b6c152cb0 0x7f0b6c158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:04.033 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b71200640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0b6c0c79c0 0x7f0b6c1525a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57112/0 (socket says 172.21.15.98:57112) 2024-05-27T00:15:04.033 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b71200640 1 -- 172.21.15.98:0/2010234707 learned_addr learned my addr 172.21.15.98:0/2010234707 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:04.033 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b72600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b6c152cb0 0x7f0b6c158b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54510/0 (socket says 172.21.15.98:54510) 2024-05-27T00:15:04.034 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b72600640 1 -- 172.21.15.98:0/2010234707 >> v1:172.21.15.204:6789/0 conn(0x7f0b6c073180 legacy=0x7f0b6c073560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:04.034 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b72600640 1 -- 172.21.15.98:0/2010234707 >> v1:172.21.15.98:6789/0 conn(0x7f0b6c0c79c0 legacy=0x7f0b6c1525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:04.034 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.030+0000 7f0b72600640 1 -- 172.21.15.98:0/2010234707 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6c15cb20 con 0x7f0b6c152cb0 2024-05-27T00:15:04.036 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b72600640 1 --2- 172.21.15.98:0/2010234707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c152cb0 0x7f0b6c158b70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009d90 tx=0x7f0b6401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48da7ddcdda2b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:04.036 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b6be00640 1 -- 172.21.15.98:0/2010234707 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b640b2070 con 0x7f0b6c152cb0 2024-05-27T00:15:04.036 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b6be00640 1 -- 172.21.15.98:0/2010234707 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b640a5940 con 0x7f0b6c152cb0 2024-05-27T00:15:04.036 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b6be00640 1 -- 172.21.15.98:0/2010234707 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b640a5d10 con 0x7f0b6c152cb0 2024-05-27T00:15:04.037 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b74600640 1 -- 172.21.15.98:0/2010234707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c152cb0 msgr2=0x7f0b6c158b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:04.037 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b74600640 1 --2- 172.21.15.98:0/2010234707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c152cb0 0x7f0b6c158b70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009d90 tx=0x7f0b6401c970 comp rx=0 tx=0).stop 2024-05-27T00:15:04.037 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b74600640 1 -- 172.21.15.98:0/2010234707 shutdown_connections 2024-05-27T00:15:04.037 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b74600640 1 --2- 172.21.15.98:0/2010234707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c152cb0 0x7f0b6c158b70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:04.037 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b74600640 1 -- 172.21.15.98:0/2010234707 >> 172.21.15.98:0/2010234707 conn(0x7f0b6c0c9ac0 msgr2=0x7f0b6c0c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:04.037 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b74600640 1 -- 172.21.15.98:0/2010234707 shutdown_connections 2024-05-27T00:15:04.037 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.034+0000 7f0b74600640 1 -- 172.21.15.98:0/2010234707 wait complete. 2024-05-27T00:15:04.038 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b74600640 1 Processor -- start 2024-05-27T00:15:04.038 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b74600640 1 -- start start 2024-05-27T00:15:04.040 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b74600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c073180 0x7f0b6c1478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:04.040 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b74600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c0c79c0 0x7f0b6c0c2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:04.040 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b74600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c152cb0 0x7f0b6c151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b74600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6c1667f0 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b74600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6c0c7da0 con 0x7f0b6c073180 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b74600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6c166970 con 0x7f0b6c152cb0 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b71200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c0c79c0 0x7f0b6c0c2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b71200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c0c79c0 0x7f0b6c0c2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54526/0 (socket says 172.21.15.98:54526) 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b71200640 1 -- 172.21.15.98:0/2488320813 learned_addr learned my addr 172.21.15.98:0/2488320813 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b72600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c152cb0 0x7f0b6c151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b71c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c073180 0x7f0b6c1478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b71200640 1 -- 172.21.15.98:0/2488320813 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c152cb0 msgr2=0x7f0b6c151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b71200640 1 --2- 172.21.15.98:0/2488320813 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c152cb0 0x7f0b6c151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b71200640 1 -- 172.21.15.98:0/2488320813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c073180 msgr2=0x7f0b6c1478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b71200640 1 --2- 172.21.15.98:0/2488320813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c073180 0x7f0b6c1478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b71200640 1 -- 172.21.15.98:0/2488320813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6c156720 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.041 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b71200640 1 --2- 172.21.15.98:0/2488320813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c0c79c0 0x7f0b6c0c2240 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800c530 tx=0x7f0b5800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:04.042 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b69600640 1 -- 172.21.15.98:0/2488320813 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58017070 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.043 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b69600640 1 -- 172.21.15.98:0/2488320813 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b58005cf0 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.043 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b69600640 1 -- 172.21.15.98:0/2488320813 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58007270 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.043 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b74600640 1 -- 172.21.15.98:0/2488320813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b6c147df0 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.043 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b74600640 1 -- 172.21.15.98:0/2488320813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b6c148060 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.043 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b69600640 1 -- 172.21.15.98:0/2488320813 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0b5801a070 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.043 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.038+0000 7f0b69600640 1 -- 172.21.15.98:0/2488320813 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f0b5801cb70 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.043 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.042+0000 7f0b74600640 1 -- 172.21.15.98:0/2488320813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b6c073ea0 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.044 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.042+0000 7f0b69600640 1 -- 172.21.15.98:0/2488320813 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0b58007410 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.101 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.098+0000 7f0b74600640 1 -- 172.21.15.98:0/2488320813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0b6c06b520 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.102 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.102+0000 7f0b69600640 1 -- 172.21.15.98:0/2488320813 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f0b580075f0 con 0x7f0b6c0c79c0 2024-05-27T00:15:04.103 INFO:teuthology.misc.health.smithi098.stdout: 2024-05-27T00:15:04.103 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:02.792649+0000","last_up_change":"0.000000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:04.104 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.102+0000 7f0b74600640 1 -- 172.21.15.98:0/2488320813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c0c79c0 msgr2=0x7f0b6c0c2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:04.104 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.102+0000 7f0b74600640 1 --2- 172.21.15.98:0/2488320813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c0c79c0 0x7f0b6c0c2240 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800c530 tx=0x7f0b5800c9f0 comp rx=0 tx=0).stop 2024-05-27T00:15:04.104 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.102+0000 7f0b74600640 1 -- 172.21.15.98:0/2488320813 shutdown_connections 2024-05-27T00:15:04.104 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.102+0000 7f0b74600640 1 --2- 172.21.15.98:0/2488320813 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c152cb0 0x7f0b6c151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:04.104 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.102+0000 7f0b74600640 1 --2- 172.21.15.98:0/2488320813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c0c79c0 0x7f0b6c0c2240 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:04.104 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.102+0000 7f0b74600640 1 --2- 172.21.15.98:0/2488320813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c073180 0x7f0b6c1478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:04.104 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.102+0000 7f0b74600640 1 -- 172.21.15.98:0/2488320813 >> 172.21.15.98:0/2488320813 conn(0x7f0b6c0c9ac0 msgr2=0x7f0b6c1637a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:04.104 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.102+0000 7f0b74600640 1 -- 172.21.15.98:0/2488320813 shutdown_connections 2024-05-27T00:15:04.104 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:04.102+0000 7f0b74600640 1 -- 172.21.15.98:0/2488320813 wait complete. 2024-05-27T00:15:04.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-27T00:15:04.133 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:04.130+0000 7fc043f7e180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T00:15:04.418 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:04.417+0000 7f8a7217b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T00:15:04.495 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:04.493+0000 7f8a7217b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T00:15:04.675 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:04.673+0000 7f8a7217b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T00:15:04.749 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:04.745+0000 7f8a7217b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T00:15:04.821 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:04.818+0000 7fc043f7e180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T00:15:04.908 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:04.906+0000 7fc043f7e180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T00:15:05.038 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:05.037+0000 7f8a7217b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T00:15:05.100 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:05.097+0000 7f8a7217b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T00:15:05.110 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:05.110+0000 7fc043f7e180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T00:15:05.200 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:05.198+0000 7fc043f7e180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T00:15:05.531 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:05.530+0000 7fc043f7e180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T00:15:05.602 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-05-27T00:15:05.602+0000 7fc043f7e180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T00:15:06.138 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:06.137+0000 7f8a195c1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-27T00:15:06.138 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-05-27T00:15:06.137+0000 7f8a195c1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-27T00:15:07.034 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-05-27T00:15:07.033+0000 7ff17ff8b7c0 -1 osd.5 0 log_to_monitors true 2024-05-27T00:15:07.111 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-05-27T00:15:07.109+0000 7f43c7d0b7c0 -1 osd.4 0 log_to_monitors true 2024-05-27T00:15:07.135 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-05-27T00:15:07.133+0000 7f0cb86557c0 -1 osd.7 0 log_to_monitors true 2024-05-27T00:15:07.224 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-05-27T00:15:07.221+0000 7fb604f527c0 -1 osd.6 0 log_to_monitors true 2024-05-27T00:15:07.393 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-05-27T00:15:07.390+0000 7fc3b14b87c0 -1 osd.2 0 log_to_monitors true 2024-05-27T00:15:07.401 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-05-27T00:15:07.398+0000 7fda453437c0 -1 osd.1 0 log_to_monitors true 2024-05-27T00:15:07.448 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-05-27T00:15:07.446+0000 7fc36a5517c0 -1 osd.0 0 log_to_monitors true 2024-05-27T00:15:07.551 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-05-27T00:15:07.550+0000 7f8afbede7c0 -1 osd.3 0 log_to_monitors true 2024-05-27T00:15:10.386 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-05-27T00:15:10.386+0000 7fc3aba00640 -1 osd.2 0 waiting for initial osdmap 2024-05-27T00:15:10.401 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-05-27T00:15:10.398+0000 7fda3f800640 -1 osd.1 0 waiting for initial osdmap 2024-05-27T00:15:10.420 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-05-27T00:15:10.418+0000 7fc3a2400640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T00:15:10.428 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-05-27T00:15:10.426+0000 7fc364a00640 -1 osd.0 0 waiting for initial osdmap 2024-05-27T00:15:10.439 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-05-27T00:15:10.438+0000 7fda36200640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T00:15:10.471 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-05-27T00:15:10.470+0000 7fc35b400640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T00:15:10.564 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-05-27T00:15:10.561+0000 7fb601f0e640 -1 osd.6 0 waiting for initial osdmap 2024-05-27T00:15:10.566 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-05-27T00:15:10.565+0000 7ff17b711640 -1 osd.5 0 waiting for initial osdmap 2024-05-27T00:15:10.590 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-05-27T00:15:10.589+0000 7fb5f8d1f640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T00:15:10.607 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-05-27T00:15:10.605+0000 7f43c3491640 -1 osd.4 0 waiting for initial osdmap 2024-05-27T00:15:10.614 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-05-27T00:15:10.613+0000 7ff173d58640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T00:15:10.642 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-05-27T00:15:10.641+0000 7f43bbad8640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T00:15:10.662 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-05-27T00:15:10.661+0000 7f0cb5611640 -1 osd.7 0 waiting for initial osdmap 2024-05-27T00:15:10.690 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-05-27T00:15:10.689+0000 7f0cac422640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T00:15:10.926 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:10.938 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-05-27T00:15:10.938+0000 7f8af6400640 -1 osd.3 0 waiting for initial osdmap 2024-05-27T00:15:10.979 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-05-27T00:15:10.978+0000 7f8aece00640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T00:15:11.054 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.050+0000 7ffaf0000640 1 Processor -- start 2024-05-27T00:15:11.054 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.050+0000 7ffaf0000640 1 -- start start 2024-05-27T00:15:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaf0000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffae8152aa0 0x7ffae80c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaf0000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 0x7ffae806fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaf0000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae8070a90 con 0x7ffae80c8c90 2024-05-27T00:15:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaf0000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ffae80c38f0 con 0x7ffae8152aa0 2024-05-27T00:15:11.055 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaf0000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffae80c3610 con 0x7ffae806f890 2024-05-27T00:15:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaed600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffae8152aa0 0x7ffae80c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaed600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ffae8152aa0 0x7ffae80c4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54578/0 (socket says 172.21.15.98:54578) 2024-05-27T00:15:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaed600640 1 -- 172.21.15.98:0/2358913989 learned_addr learned my addr 172.21.15.98:0/2358913989 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaee000640 1 --2- 172.21.15.98:0/2358913989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 0x7ffae806fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaed600640 1 -- 172.21.15.98:0/2358913989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 msgr2=0x7ffae806fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaed600640 1 --2- 172.21.15.98:0/2358913989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 0x7ffae806fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaed600640 1 -- 172.21.15.98:0/2358913989 >> v1:172.21.15.204:6789/0 conn(0x7ffae80c8c90 legacy=0x7ffae8158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.054+0000 7ffaed600640 1 -- 172.21.15.98:0/2358913989 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae815bda0 con 0x7ffae8152aa0 2024-05-27T00:15:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaed600640 1 --2- 172.21.15.98:0/2358913989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 0x7ffae80c4ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009870 tx=0x7ffadc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d148ce84c381f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffae7e00640 1 -- 172.21.15.98:0/2358913989 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc0a9620 con 0x7ffae8152aa0 2024-05-27T00:15:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffae7e00640 1 -- 172.21.15.98:0/2358913989 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffadc0a97c0 con 0x7ffae8152aa0 2024-05-27T00:15:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffae7e00640 1 -- 172.21.15.98:0/2358913989 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc0a9b90 con 0x7ffae8152aa0 2024-05-27T00:15:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaf0000640 1 -- 172.21.15.98:0/2358913989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 msgr2=0x7ffae80c4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaf0000640 1 --2- 172.21.15.98:0/2358913989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 0x7ffae80c4ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009870 tx=0x7ffadc01c560 comp rx=0 tx=0).stop 2024-05-27T00:15:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaf0000640 1 -- 172.21.15.98:0/2358913989 shutdown_connections 2024-05-27T00:15:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaf0000640 1 --2- 172.21.15.98:0/2358913989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 0x7ffae806fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaf0000640 1 --2- 172.21.15.98:0/2358913989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 0x7ffae80c4ef0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaf0000640 1 -- 172.21.15.98:0/2358913989 >> 172.21.15.98:0/2358913989 conn(0x7ffae8062490 msgr2=0x7ffae8070fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:11.061 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaf0000640 1 -- 172.21.15.98:0/2358913989 shutdown_connections 2024-05-27T00:15:11.061 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaf0000640 1 -- 172.21.15.98:0/2358913989 wait complete. 2024-05-27T00:15:11.061 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaf0000640 1 Processor -- start 2024-05-27T00:15:11.061 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.058+0000 7ffaf0000640 1 -- start start 2024-05-27T00:15:11.066 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaf0000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 0x7ffae813fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:11.066 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaf0000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffae80c8c90 0x7ffae80cea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:11.066 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaf0000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 0x7ffae8150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:11.066 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaf0000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8152e80 con 0x7ffae8152aa0 2024-05-27T00:15:11.066 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaf0000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffae80c5430 con 0x7ffae80c8c90 2024-05-27T00:15:11.066 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaf0000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffae815cb60 con 0x7ffae806f890 2024-05-27T00:15:11.067 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaee000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 0x7ffae8150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:11.067 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaee000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 0x7ffae8150360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54580/0 (socket says 172.21.15.98:54580) 2024-05-27T00:15:11.067 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaee000640 1 -- 172.21.15.98:0/3766061002 learned_addr learned my addr 172.21.15.98:0/3766061002 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:11.067 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaecc00640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffae80c8c90 0x7ffae80cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:11.067 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaed600640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 0x7ffae813fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:11.067 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaee000640 1 -- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 msgr2=0x7ffae813fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:11.067 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaee000640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 0x7ffae813fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaee000640 1 -- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffae80c8c90 msgr2=0x7ffae80cea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaee000640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffae80c8c90 0x7ffae80cea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaee000640 1 -- 172.21.15.98:0/3766061002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae81f1070 con 0x7ffae8152aa0 2024-05-27T00:15:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaed600640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 0x7ffae813fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaee000640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 0x7ffae8150360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffae0007e70 tx=0x7ffae000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffae5600640 1 -- 172.21.15.98:0/3766061002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae0014620 con 0x7ffae8152aa0 2024-05-27T00:15:11.068 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffae5600640 1 -- 172.21.15.98:0/3766061002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae0019070 con 0x7ffae8152aa0 2024-05-27T00:15:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaf0000640 1 -- 172.21.15.98:0/3766061002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae81f1350 con 0x7ffae8152aa0 2024-05-27T00:15:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffae5600640 1 -- 172.21.15.98:0/3766061002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae0014a20 con 0x7ffae8152aa0 2024-05-27T00:15:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaf0000640 1 -- 172.21.15.98:0/3766061002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae81f1910 con 0x7ffae8152aa0 2024-05-27T00:15:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffaf0000640 1 -- 172.21.15.98:0/3766061002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffae8073ea0 con 0x7ffae8152aa0 2024-05-27T00:15:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.062+0000 7ffae5600640 1 -- 172.21.15.98:0/3766061002 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 87389+0+0 (secure 0 0 0) 0x7ffae0014bc0 con 0x7ffae8152aa0 2024-05-27T00:15:11.069 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.066+0000 7ffae5600640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffac8067830 0x7ffac8069cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:11.070 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.066+0000 7ffae5600640 1 -- 172.21.15.98:0/3766061002 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2231+0+0 (secure 0 0 0) 0x7ffae0085670 con 0x7ffae8152aa0 2024-05-27T00:15:11.070 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.066+0000 7ffaed600640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffac8067830 0x7ffac8069cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:11.070 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.066+0000 7ffaed600640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffac8067830 0x7ffac8069cf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0ae000 tx=0x7ffadc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:11.108 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.106+0000 7ffae5600640 1 -- 172.21.15.98:0/3766061002 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7ffae0057010 con 0x7ffae8152aa0 2024-05-27T00:15:11.264 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.262+0000 7ffaf0000640 1 -- 172.21.15.98:0/3766061002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffae80ca8b0 con 0x7ffae8152aa0 2024-05-27T00:15:11.265 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.262+0000 7ffae5600640 1 -- 172.21.15.98:0/3766061002 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+7058 (secure 0 0 0) 0x7ffae0056910 con 0x7ffae8152aa0 2024-05-27T00:15:11.265 INFO:teuthology.misc.health.smithi098.stdout: 2024-05-27T00:15:11.265 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":13,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:09.493758+0000","last_up_change":"0.000000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:11.267 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 -- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffac8067830 msgr2=0x7ffac8069cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:11.267 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffac8067830 0x7ffac8069cf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0ae000 tx=0x7ffadc009040 comp rx=0 tx=0).stop 2024-05-27T00:15:11.267 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 -- 172.21.15.98:0/3766061002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 msgr2=0x7ffae8150360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:11.268 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 0x7ffae8150360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffae0007e70 tx=0x7ffae000e4b0 comp rx=0 tx=0).stop 2024-05-27T00:15:11.268 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 -- 172.21.15.98:0/3766061002 shutdown_connections 2024-05-27T00:15:11.268 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae8152aa0 0x7ffae8150360 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:11.268 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffac8067830 0x7ffac8069cf0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:11.268 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffae80c8c90 0x7ffae80cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:11.268 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 --2- 172.21.15.98:0/3766061002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae806f890 0x7ffae813fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:11.268 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 -- 172.21.15.98:0/3766061002 >> 172.21.15.98:0/3766061002 conn(0x7ffae8062490 msgr2=0x7ffae814f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:11.268 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 -- 172.21.15.98:0/3766061002 shutdown_connections 2024-05-27T00:15:11.268 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:11.266+0000 7ffaf0000640 1 -- 172.21.15.98:0/3766061002 wait complete. 2024-05-27T00:15:11.284 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-27T00:15:13.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:15:13.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:15:18.090 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:18.226 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.222+0000 7f9322400640 1 Processor -- start 2024-05-27T00:15:18.226 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.222+0000 7f9322400640 1 -- start start 2024-05-27T00:15:18.227 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f9322400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f931c0c7fa0 0x7f931c0c9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.227 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f9322400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f931c0c1950 con 0x7f931c0aec20 2024-05-27T00:15:18.227 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f9322400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f931c0c9670 con 0x7f931c0c8380 2024-05-27T00:15:18.227 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f9322400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f931c0c1670 con 0x7f931c0c7fa0 2024-05-27T00:15:18.227 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f931b400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f931c0c7fa0 0x7f931c0c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.227 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f931b400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f931c0c7fa0 0x7f931c0c9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35980/0 (socket says 172.21.15.98:35980) 2024-05-27T00:15:18.227 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f931b400640 1 -- 172.21.15.98:0/3250669351 learned_addr learned my addr 172.21.15.98:0/3250669351 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:18.228 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f931b400640 1 -- 172.21.15.98:0/3250669351 >> v1:172.21.15.204:6789/0 conn(0x7f931c0c8380 legacy=0x7f931c073400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.228 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f931b400640 1 -- 172.21.15.98:0/3250669351 >> v1:172.21.15.98:6789/0 conn(0x7f931c0aec20 legacy=0x7f931c151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.228 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f931b400640 1 -- 172.21.15.98:0/3250669351 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931c0c2bd0 con 0x7f931c0c7fa0 2024-05-27T00:15:18.228 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.226+0000 7f931aa00640 1 -- 172.21.15.98:0/3250669351 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 529795727 0 0) 0x7f931c0c1950 con 0x7f931c0aec20 2024-05-27T00:15:18.230 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f931b400640 1 --2- 172.21.15.98:0/3250669351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c7fa0 0x7f931c0c9130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f930c009870 tx=0x7f930c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b765f7df3153ce6b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:18.231 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f931aa00640 1 -- 172.21.15.98:0/3250669351 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c0b1070 con 0x7f931c0c7fa0 2024-05-27T00:15:18.231 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f931aa00640 1 -- 172.21.15.98:0/3250669351 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f930c0ac600 con 0x7f931c0c7fa0 2024-05-27T00:15:18.231 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f931aa00640 1 -- 172.21.15.98:0/3250669351 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c0ac9d0 con 0x7f931c0c7fa0 2024-05-27T00:15:18.231 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 -- 172.21.15.98:0/3250669351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c7fa0 msgr2=0x7f931c0c9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.231 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 --2- 172.21.15.98:0/3250669351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c7fa0 0x7f931c0c9130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f930c009870 tx=0x7f930c01c4a0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.232 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 -- 172.21.15.98:0/3250669351 shutdown_connections 2024-05-27T00:15:18.232 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 --2- 172.21.15.98:0/3250669351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c7fa0 0x7f931c0c9130 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.232 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 -- 172.21.15.98:0/3250669351 >> 172.21.15.98:0/3250669351 conn(0x7f931c06e120 msgr2=0x7f931c06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:18.232 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 -- 172.21.15.98:0/3250669351 shutdown_connections 2024-05-27T00:15:18.232 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 -- 172.21.15.98:0/3250669351 wait complete. 2024-05-27T00:15:18.233 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 Processor -- start 2024-05-27T00:15:18.233 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 -- start start 2024-05-27T00:15:18.234 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f931c0aec20 0x7f931c154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f931c0c7fa0 0x7f931c1f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.230+0000 7f9322400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c8380 0x7f931c1f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9322400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f931c0c1250 con 0x7f931c0c8380 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9322400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f931c073b10 con 0x7f931c0aec20 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9322400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f931c0c13d0 con 0x7f931c0c7fa0 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f931be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c8380 0x7f931c1f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f931be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c8380 0x7f931c1f77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35988/0 (socket says 172.21.15.98:35988) 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f931be00640 1 -- 172.21.15.98:0/1574964610 learned_addr learned my addr 172.21.15.98:0/1574964610 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f931b400640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f931c0aec20 0x7f931c154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9312a00640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f931c0c7fa0 0x7f931c1f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f931be00640 1 -- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f931c0c7fa0 msgr2=0x7f931c1f72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f931be00640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f931c0c7fa0 0x7f931c1f72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f931be00640 1 -- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f931c0aec20 msgr2=0x7f931c154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.235 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f931be00640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f931c0aec20 0x7f931c154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.236 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f931be00640 1 -- 172.21.15.98:0/1574964610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931c152d80 con 0x7f931c0c8380 2024-05-27T00:15:18.237 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f931be00640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c8380 0x7f931c1f77f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f931400c7b0 tx=0x7f931400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:18.237 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9313e00640 1 -- 172.21.15.98:0/1574964610 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931400ce60 con 0x7f931c0c8380 2024-05-27T00:15:18.237 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9313e00640 1 -- 172.21.15.98:0/1574964610 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9314003db0 con 0x7f931c0c8380 2024-05-27T00:15:18.237 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9313e00640 1 -- 172.21.15.98:0/1574964610 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93140052b0 con 0x7f931c0c8380 2024-05-27T00:15:18.237 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9322400640 1 -- 172.21.15.98:0/1574964610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f931c0cc2a0 con 0x7f931c0c8380 2024-05-27T00:15:18.237 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9322400640 1 -- 172.21.15.98:0/1574964610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f931c0cc860 con 0x7f931c0c8380 2024-05-27T00:15:18.237 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9322400640 1 -- 172.21.15.98:0/1574964610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f931c073ea0 con 0x7f931c0c8380 2024-05-27T00:15:18.237 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.234+0000 7f9313e00640 1 -- 172.21.15.98:0/1574964610 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9314005450 con 0x7f931c0c8380 2024-05-27T00:15:18.238 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.238+0000 7f9313e00640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92f80678c0 0x7f92f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.238 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.238+0000 7f9313e00640 1 -- 172.21.15.98:0/1574964610 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f9314085860 con 0x7f931c0c8380 2024-05-27T00:15:18.239 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.238+0000 7f931b400640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92f80678c0 0x7f92f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.239 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.238+0000 7f931b400640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92f80678c0 0x7f92f8069d80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f930c0ae000 tx=0x7f930c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:18.243 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.242+0000 7f9313e00640 1 -- 172.21.15.98:0/1574964610 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f93140565c0 con 0x7f931c0c8380 2024-05-27T00:15:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.398+0000 7f9322400640 1 -- 172.21.15.98:0/1574964610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f931c162e30 con 0x7f931c0c8380 2024-05-27T00:15:18.401 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.398+0000 7f9313e00640 1 -- 172.21.15.98:0/1574964610 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12860 (secure 0 0 0) 0x7f9314055ec0 con 0x7f931c0c8380 2024-05-27T00:15:18.401 INFO:teuthology.misc.health.smithi098.stdout: 2024-05-27T00:15:18.401 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":18,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:15.428401+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:18.403 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 -- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92f80678c0 msgr2=0x7f92f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.403 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92f80678c0 0x7f92f8069d80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f930c0ae000 tx=0x7f930c0a7000 comp rx=0 tx=0).stop 2024-05-27T00:15:18.403 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 -- 172.21.15.98:0/1574964610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c8380 msgr2=0x7f931c1f77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.403 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c8380 0x7f931c1f77f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f931400c7b0 tx=0x7f931400cc70 comp rx=0 tx=0).stop 2024-05-27T00:15:18.403 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 -- 172.21.15.98:0/1574964610 shutdown_connections 2024-05-27T00:15:18.403 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f931c0c8380 0x7f931c1f77f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.403 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f931c0c7fa0 0x7f931c1f72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.403 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92f80678c0 0x7f92f8069d80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.403 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 --2- 172.21.15.98:0/1574964610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f931c0aec20 0x7f931c154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.403 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 -- 172.21.15.98:0/1574964610 >> 172.21.15.98:0/1574964610 conn(0x7f931c06e120 msgr2=0x7f931c071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:18.404 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 -- 172.21.15.98:0/1574964610 shutdown_connections 2024-05-27T00:15:18.404 INFO:teuthology.misc.health.smithi098.stderr:2024-05-27T00:15:18.402+0000 7f9322400640 1 -- 172.21.15.98:0/1574964610 wait complete. 2024-05-27T00:15:18.420 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-27T00:15:18.420 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-27T00:15:18.421 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-27T00:15:18.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-27T00:15:18.424 INFO:tasks.ceph.mds.a:Started 2024-05-27T00:15:18.424 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-27T00:15:18.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-27T00:15:18.427 INFO:tasks.ceph.mds.c:Started 2024-05-27T00:15:18.427 INFO:tasks.ceph.mds.e:Restarting daemon 2024-05-27T00:15:18.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-05-27T00:15:18.430 INFO:tasks.ceph.mds.e:Started 2024-05-27T00:15:18.430 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-27T00:15:18.430 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-27T00:15:18.433 INFO:tasks.ceph.mds.b:Started 2024-05-27T00:15:18.433 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-27T00:15:18.433 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-27T00:15:18.435 INFO:tasks.ceph.mds.d:Started 2024-05-27T00:15:18.435 INFO:tasks.ceph.mds.f:Restarting daemon 2024-05-27T00:15:18.435 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-05-27T00:15:18.438 INFO:tasks.ceph.mds.f:Started 2024-05-27T00:15:18.438 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-05-27T00:15:18.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:18.513 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-05-27T00:15:18.513 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-05-27T00:15:18.514 INFO:tasks.ceph.mds.f.smithi204.stdout:starting mds.f at 2024-05-27T00:15:18.527 INFO:tasks.ceph.mds.e.smithi098.stdout:starting mds.e at 2024-05-27T00:15:18.527 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2024-05-27T00:15:18.544 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2024-05-27T00:15:18.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.598+0000 7f17c7400640 1 Processor -- start 2024-05-27T00:15:18.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.598+0000 7f17c7400640 1 -- start start 2024-05-27T00:15:18.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.598+0000 7f17c7400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f17c00cb7e0 0x7f17c0158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17c7400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c015be40 con 0x7f17c00b3ed0 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17c7400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c00c50e0 con 0x7f17c0152a50 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17c7400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f17c00c8a00 con 0x7f17c00cb7e0 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17c5400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f17c00b3ed0 0x7f17c0162ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43298/0 (socket says 172.21.15.98:43298) 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17c5400640 1 -- 172.21.15.98:0/2925929537 learned_addr learned my addr 172.21.15.98:0/2925929537 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17b7e00640 1 --2- 172.21.15.98:0/2925929537 >> v2:172.21.15.204:3300/0 conn(0x7f17c00cb7e0 0x7f17c0158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330053346 0 0) 0x7f17c015be40 con 0x7f17c00b3ed0 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17a4003530 con 0x7f17c00b3ed0 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1018148199 0 0) 0x7f17c00c50e0 con 0x7f17c0152a50 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17c015be40 con 0x7f17c0152a50 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3862799494 0 0) 0x7f17a4003530 con 0x7f17c00b3ed0 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 >> v2:172.21.15.204:3300/0 conn(0x7f17c00cb7e0 msgr2=0x7f17c0158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 --2- 172.21.15.98:0/2925929537 >> v2:172.21.15.204:3300/0 conn(0x7f17c00cb7e0 0x7f17c0158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 >> v1:172.21.15.204:6789/0 conn(0x7f17c0152a50 legacy=0x7f17c00709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c00c8c50 con 0x7f17c00b3ed0 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f17b8001a00 con 0x7f17c00b3ed0 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 >> v1:172.21.15.98:6789/0 conn(0x7f17c00b3ed0 legacy=0x7f17c0162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a4003b50 0x7f17a4003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 0x7f17a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17a40051d0 0x7f17a400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17c00c8a00 con 0x7f17a40051d0 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17a400c100 con 0x7f17a4003b50 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f17a400c300 con 0x7f17a4004810 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17c5400640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17a40051d0 0x7f17a400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17b7e00640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 0x7f17a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17c4a00640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a4003b50 0x7f17a4003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17b7e00640 1 -- 172.21.15.98:0/2925929537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a4003b50 msgr2=0x7f17a4003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:18.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17b7e00640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a4003b50 0x7f17a4003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17b7e00640 1 -- 172.21.15.98:0/2925929537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17a40051d0 msgr2=0x7f17a400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:18.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17b7e00640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17a40051d0 0x7f17a400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17b7e00640 1 -- 172.21.15.98:0/2925929537 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a4003530 con 0x7f17a4004810 2024-05-27T00:15:18.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.602+0000 7f17c5400640 1 -- 172.21.15.98:0/2925929537 reap_dead start 2024-05-27T00:15:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17b7e00640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 0x7f17a4004c90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009ab0 tx=0x7f17a801ca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f64bb73070a4944b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a80b2070 con 0x7f17a4004810 2024-05-27T00:15:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a80a5640 con 0x7f17a4004810 2024-05-27T00:15:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17bfe00640 1 -- 172.21.15.98:0/2925929537 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a80096b0 con 0x7f17a4004810 2024-05-27T00:15:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17c7400640 1 -- 172.21.15.98:0/2925929537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 msgr2=0x7f17a4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17c7400640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 0x7f17a4004c90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009ab0 tx=0x7f17a801ca50 comp rx=0 tx=0).stop 2024-05-27T00:15:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17c7400640 1 -- 172.21.15.98:0/2925929537 shutdown_connections 2024-05-27T00:15:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17c7400640 1 --2- 172.21.15.98:0/2925929537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 0x7f17a4004c90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17c7400640 1 -- 172.21.15.98:0/2925929537 >> 172.21.15.98:0/2925929537 conn(0x7f17c0072f90 msgr2=0x7f17c00733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17c7400640 1 -- 172.21.15.98:0/2925929537 shutdown_connections 2024-05-27T00:15:18.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17c7400640 1 -- 172.21.15.98:0/2925929537 wait complete. 2024-05-27T00:15:18.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17c7400640 1 Processor -- start 2024-05-27T00:15:18.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.606+0000 7f17c7400640 1 -- start start 2024-05-27T00:15:18.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 0x7f17c00cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c01e4f60 0x7f17c01e3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c01f3ae0 0x7f17c01f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c7400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17c00c94f0 con 0x7f17c01e4f60 2024-05-27T00:15:18.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c7400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17c00c9210 con 0x7f17c01f3ae0 2024-05-27T00:15:18.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c7400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f17c00cb570 con 0x7f17a4004810 2024-05-27T00:15:18.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17b7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c01e4f60 0x7f17c01e3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17b7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c01e4f60 0x7f17c01e3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36050/0 (socket says 172.21.15.98:36050) 2024-05-27T00:15:18.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17b7e00640 1 -- 172.21.15.98:0/1030806799 learned_addr learned my addr 172.21.15.98:0/1030806799 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:18.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c5400640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c01f3ae0 0x7f17c01f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c4a00640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 0x7f17c00cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17b7e00640 1 -- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 msgr2=0x7f17c00cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17b7e00640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 0x7f17c00cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17b7e00640 1 -- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c01f3ae0 msgr2=0x7f17c01f3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17b7e00640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c01f3ae0 0x7f17c01f3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17b7e00640 1 -- 172.21.15.98:0/1030806799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c00c9f20 con 0x7f17c01e4f60 2024-05-27T00:15:18.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17b7e00640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c01e4f60 0x7f17c01e3d80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17a801c880 tx=0x7f17a80197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:18.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17bd600640 1 -- 172.21.15.98:0/1030806799 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a80b2040 con 0x7f17c01e4f60 2024-05-27T00:15:18.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17bd600640 1 -- 172.21.15.98:0/1030806799 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a8004020 con 0x7f17c01e4f60 2024-05-27T00:15:18.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17bd600640 1 -- 172.21.15.98:0/1030806799 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a8004410 con 0x7f17c01e4f60 2024-05-27T00:15:18.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c7400640 1 -- 172.21.15.98:0/1030806799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c00cc500 con 0x7f17c01e4f60 2024-05-27T00:15:18.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c7400640 1 -- 172.21.15.98:0/1030806799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c01e2fc0 con 0x7f17c01e4f60 2024-05-27T00:15:18.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.610+0000 7f17c7400640 1 -- 172.21.15.98:0/1030806799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17c0075cb0 con 0x7f17c01e4f60 2024-05-27T00:15:18.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.614+0000 7f17bd600640 1 -- 172.21.15.98:0/1030806799 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f17a80bf060 con 0x7f17c01e4f60 2024-05-27T00:15:18.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.614+0000 7f17bd600640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678c0 0x7f179c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.614+0000 7f17bd600640 1 -- 172.21.15.98:0/1030806799 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f17a811f600 con 0x7f17c01e4f60 2024-05-27T00:15:18.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.614+0000 7f17c4a00640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678c0 0x7f179c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.614+0000 7f17c4a00640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678c0 0x7f179c069d80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f17b0000f70 tx=0x7f17b0000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:18.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.618+0000 7f17bd600640 1 -- 172.21.15.98:0/1030806799 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f17a8016760 con 0x7f17c01e4f60 2024-05-27T00:15:18.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.774+0000 7f17c7400640 1 -- 172.21.15.98:0/1030806799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f17c00736f0 con 0x7f17c01e4f60 2024-05-27T00:15:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.774+0000 7f17bd600640 1 -- 172.21.15.98:0/1030806799 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12860 (secure 0 0 0) 0x7f17a80f0360 con 0x7f17c01e4f60 2024-05-27T00:15:18.777 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:18.777 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:15.428401+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 -- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678c0 msgr2=0x7f179c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678c0 0x7f179c069d80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f17b0000f70 tx=0x7f17b0000ee0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 -- 172.21.15.98:0/1030806799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c01e4f60 msgr2=0x7f17c01e3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c01e4f60 0x7f17c01e3d80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17a801c880 tx=0x7f17a80197f0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 -- 172.21.15.98:0/1030806799 shutdown_connections 2024-05-27T00:15:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678c0 0x7f179c069d80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c01f3ae0 0x7f17c01f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17c01e4f60 0x7f17c01e3d80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 --2- 172.21.15.98:0/1030806799 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17a4004810 0x7f17c00cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 -- 172.21.15.98:0/1030806799 >> 172.21.15.98:0/1030806799 conn(0x7f17c0072f90 msgr2=0x7f17c00c7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 -- 172.21.15.98:0/1030806799 shutdown_connections 2024-05-27T00:15:18.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.778+0000 7f17c7400640 1 -- 172.21.15.98:0/1030806799 wait complete. 2024-05-27T00:15:18.797 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T00:15:13.211703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T00:15:18.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T00:15:18.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aaf200640 1 Processor -- start 2024-05-27T00:15:18.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aaf200640 1 -- start start 2024-05-27T00:15:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aaf200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9aa806c280 0x7f9aa80c9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aaf200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aa80c7c50 con 0x7f9aa8067390 2024-05-27T00:15:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aaf200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aa81587b0 con 0x7f9aa8067770 2024-05-27T00:15:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aaf200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9aa80c9780 con 0x7f9aa806c280 2024-05-27T00:15:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aad200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9aa806c280 0x7f9aa80c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aa7e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9aa8067390 0x7f9aa815c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43308/0 (socket says 172.21.15.98:43308) 2024-05-27T00:15:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aa7e00640 1 -- 172.21.15.98:0/3240858539 learned_addr learned my addr 172.21.15.98:0/3240858539 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aad200640 1 -- 172.21.15.98:0/3240858539 >> v1:172.21.15.204:6789/0 conn(0x7f9aa8067770 legacy=0x7f9aa806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aad200640 1 -- 172.21.15.98:0/3240858539 >> v1:172.21.15.98:6789/0 conn(0x7f9aa8067390 legacy=0x7f9aa815c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.934+0000 7f9aad200640 1 -- 172.21.15.98:0/3240858539 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa8158a90 con 0x7f9aa806c280 2024-05-27T00:15:18.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aad200640 1 --2- 172.21.15.98:0/3240858539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 0x7f9aa80c9240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00097c0 tx=0x7f9aa001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2bdcdc4cb4115b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:18.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aa6a00640 1 -- 172.21.15.98:0/3240858539 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00b5070 con 0x7f9aa806c280 2024-05-27T00:15:18.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aa6a00640 1 -- 172.21.15.98:0/3240858539 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa00a8990 con 0x7f9aa806c280 2024-05-27T00:15:18.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aa6a00640 1 -- 172.21.15.98:0/3240858539 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00a8d60 con 0x7f9aa806c280 2024-05-27T00:15:18.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aaf200640 1 -- 172.21.15.98:0/3240858539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 msgr2=0x7f9aa80c9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aaf200640 1 --2- 172.21.15.98:0/3240858539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 0x7f9aa80c9240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00097c0 tx=0x7f9aa001ec90 comp rx=0 tx=0).stop 2024-05-27T00:15:18.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aaf200640 1 -- 172.21.15.98:0/3240858539 shutdown_connections 2024-05-27T00:15:18.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aaf200640 1 --2- 172.21.15.98:0/3240858539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 0x7f9aa80c9240 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aaf200640 1 -- 172.21.15.98:0/3240858539 >> 172.21.15.98:0/3240858539 conn(0x7f9aa806e6b0 msgr2=0x7f9aa80ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:18.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aaf200640 1 -- 172.21.15.98:0/3240858539 shutdown_connections 2024-05-27T00:15:18.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.938+0000 7f9aaf200640 1 -- 172.21.15.98:0/3240858539 wait complete. 2024-05-27T00:15:18.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aaf200640 1 Processor -- start 2024-05-27T00:15:18.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aaf200640 1 -- start start 2024-05-27T00:15:18.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aaf200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aa8067390 0x7f9aa80b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aaf200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9aa8067770 0x7f9aa80afb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aaf200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 0x7f9aa80c3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aaf200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa8159330 con 0x7f9aa806c280 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aaf200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa8159050 con 0x7f9aa8067390 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aaf200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa81522e0 con 0x7f9aa8067770 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aad200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 0x7f9aa80c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aad200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 0x7f9aa80c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36064/0 (socket says 172.21.15.98:36064) 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aad200640 1 -- 172.21.15.98:0/3554524475 learned_addr learned my addr 172.21.15.98:0/3554524475 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aa7e00640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aa8067390 0x7f9aa80b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aa7400640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9aa8067770 0x7f9aa80afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aad200640 1 -- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9aa8067770 msgr2=0x7f9aa80afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aad200640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9aa8067770 0x7f9aa80afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aad200640 1 -- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aa8067390 msgr2=0x7f9aa80b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:18.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aad200640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aa8067390 0x7f9aa80b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:18.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aad200640 1 -- 172.21.15.98:0/3554524475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa815a260 con 0x7f9aa806c280 2024-05-27T00:15:18.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aa7e00640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aa8067390 0x7f9aa80b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:18.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aad200640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 0x7f9aa80c3840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9aa001b8a0 tx=0x7f9aa001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:18.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9a9fe00640 1 -- 172.21.15.98:0/3554524475 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00bf070 con 0x7f9aa806c280 2024-05-27T00:15:18.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9a9fe00640 1 -- 172.21.15.98:0/3554524475 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa00b0800 con 0x7f9aa806c280 2024-05-27T00:15:18.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9a9fe00640 1 -- 172.21.15.98:0/3554524475 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00b5040 con 0x7f9aa806c280 2024-05-27T00:15:18.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aaf200640 1 -- 172.21.15.98:0/3554524475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa815a500 con 0x7f9aa806c280 2024-05-27T00:15:18.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.942+0000 7f9aaf200640 1 -- 172.21.15.98:0/3554524475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa815aa60 con 0x7f9aa806c280 2024-05-27T00:15:18.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.946+0000 7f9aaf200640 1 -- 172.21.15.98:0/3554524475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa8075cb0 con 0x7f9aa806c280 2024-05-27T00:15:18.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.946+0000 7f9a9fe00640 1 -- 172.21.15.98:0/3554524475 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9aa00b0bd0 con 0x7f9aa806c280 2024-05-27T00:15:18.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.946+0000 7f9a9fe00640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9a880678c0 0x7f9a88069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:18.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.946+0000 7f9a9fe00640 1 -- 172.21.15.98:0/3554524475 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f9aa01222c0 con 0x7f9aa806c280 2024-05-27T00:15:18.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.946+0000 7f9aa7e00640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9a880678c0 0x7f9a88069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:18.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.946+0000 7f9aa7e00640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9a880678c0 0x7f9a88069d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a980012f0 tx=0x7f9a98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:18.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:18.950+0000 7f9a9fe00640 1 -- 172.21.15.98:0/3554524475 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9aa00187f0 con 0x7f9aa806c280 2024-05-27T00:15:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.106+0000 7f9aaf200640 1 -- 172.21.15.98:0/3554524475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9aa8068950 con 0x7f9aa806c280 2024-05-27T00:15:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.106+0000 7f9a9fe00640 1 -- 172.21.15.98:0/3554524475 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) ==== 93+0+10 (secure 0 0 0) 0x7f9aa00f3150 con 0x7f9aa806c280 2024-05-27T00:15:19.109 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 -- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9a880678c0 msgr2=0x7f9a88069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9a880678c0 0x7f9a88069d80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9a980012f0 tx=0x7f9a98009040 comp rx=0 tx=0).stop 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 -- 172.21.15.98:0/3554524475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 msgr2=0x7f9aa80c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 0x7f9aa80c3840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9aa001b8a0 tx=0x7f9aa001ba30 comp rx=0 tx=0).stop 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 -- 172.21.15.98:0/3554524475 shutdown_connections 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aa806c280 0x7f9aa80c3840 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9aa8067770 0x7f9aa80afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9a880678c0 0x7f9a88069d80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 --2- 172.21.15.98:0/3554524475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aa8067390 0x7f9aa80b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 -- 172.21.15.98:0/3554524475 >> 172.21.15.98:0/3554524475 conn(0x7f9aa806e6b0 msgr2=0x7f9aa80caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 -- 172.21.15.98:0/3554524475 shutdown_connections 2024-05-27T00:15:19.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.110+0000 7f9aaf200640 1 -- 172.21.15.98:0/3554524475 wait complete. 2024-05-27T00:15:19.130 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:15:19.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f11400640 1 Processor -- start 2024-05-27T00:15:19.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f11400640 1 -- start start 2024-05-27T00:15:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f11400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f0c152850 0x7f2f0c0c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f11400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f0c15bf40 0x7f2f0c15c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f11400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f0c0ca440 con 0x7f2f0c0c9f30 2024-05-27T00:15:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f11400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2f0c0c58f0 con 0x7f2f0c15bf40 2024-05-27T00:15:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f11400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2f0c15bab0 con 0x7f2f0c152850 2024-05-27T00:15:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f10a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f0c15bf40 0x7f2f0c15c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f03e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2f0c0c9f30 0x7f2f0c070660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43310/0 (socket says 172.21.15.98:43310) 2024-05-27T00:15:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f10a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f0c15bf40 0x7f2f0c15c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36070/0 (socket says 172.21.15.98:36070) 2024-05-27T00:15:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f03e00640 1 -- 172.21.15.98:0/2540860331 learned_addr learned my addr 172.21.15.98:0/2540860331 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f0be00640 1 --2- 172.21.15.98:0/2540860331 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f0c152850 0x7f2f0c0c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f10a00640 1 -- 172.21.15.98:0/2540860331 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f0c152850 msgr2=0x7f2f0c0c53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f10a00640 1 --2- 172.21.15.98:0/2540860331 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f0c152850 0x7f2f0c0c53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f10a00640 1 -- 172.21.15.98:0/2540860331 >> v1:172.21.15.98:6789/0 conn(0x7f2f0c0c9f30 legacy=0x7f2f0c070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f0b400640 1 -- 172.21.15.98:0/2540860331 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2026960584 0 0) 0x7f2f0c0ca440 con 0x7f2f0c0c9f30 2024-05-27T00:15:19.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.266+0000 7f2f10a00640 1 -- 172.21.15.98:0/2540860331 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f0c15c8d0 con 0x7f2f0c15bf40 2024-05-27T00:15:19.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f10a00640 1 --2- 172.21.15.98:0/2540860331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c15bf40 0x7f2f0c15c390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009d90 tx=0x7f2f0401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d3dee962514ebc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f0b400640 1 -- 172.21.15.98:0/2540860331 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f040b4070 con 0x7f2f0c15bf40 2024-05-27T00:15:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f0b400640 1 -- 172.21.15.98:0/2540860331 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f040af600 con 0x7f2f0c15bf40 2024-05-27T00:15:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f0b400640 1 -- 172.21.15.98:0/2540860331 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f040af9d0 con 0x7f2f0c15bf40 2024-05-27T00:15:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f11400640 1 -- 172.21.15.98:0/2540860331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c15bf40 msgr2=0x7f2f0c15c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f11400640 1 --2- 172.21.15.98:0/2540860331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c15bf40 0x7f2f0c15c390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009d90 tx=0x7f2f0401f150 comp rx=0 tx=0).stop 2024-05-27T00:15:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f11400640 1 -- 172.21.15.98:0/2540860331 shutdown_connections 2024-05-27T00:15:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f11400640 1 --2- 172.21.15.98:0/2540860331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c15bf40 0x7f2f0c15c390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f11400640 1 --2- 172.21.15.98:0/2540860331 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f0c152850 0x7f2f0c0c53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f11400640 1 -- 172.21.15.98:0/2540860331 >> 172.21.15.98:0/2540860331 conn(0x7f2f0c0cdeb0 msgr2=0x7f2f0c0ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:19.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f11400640 1 -- 172.21.15.98:0/2540860331 shutdown_connections 2024-05-27T00:15:19.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f11400640 1 -- 172.21.15.98:0/2540860331 wait complete. 2024-05-27T00:15:19.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f11400640 1 Processor -- start 2024-05-27T00:15:19.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.270+0000 7f2f11400640 1 -- start start 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f11400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f0c0c9f30 0x7f2f0c154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f11400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c152850 0x7f2f0c0cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f11400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f0c15bf40 0x7f2f0c1eeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f11400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f0c06f870 con 0x7f2f0c152850 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f11400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2f0c06f590 con 0x7f2f0c15bf40 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f11400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2f0c06fb50 con 0x7f2f0c0c9f30 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f03e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c152850 0x7f2f0c0cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f03e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c152850 0x7f2f0c0cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36072/0 (socket says 172.21.15.98:36072) 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f03e00640 1 -- 172.21.15.98:0/1077565536 learned_addr learned my addr 172.21.15.98:0/1077565536 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f10a00640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f0c15bf40 0x7f2f0c1eeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:19.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f0be00640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f0c0c9f30 0x7f2f0c154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f03e00640 1 -- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f0c0c9f30 msgr2=0x7f2f0c154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f03e00640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f0c0c9f30 0x7f2f0c154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f03e00640 1 -- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f0c15bf40 msgr2=0x7f2f0c1eeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f03e00640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f0c15bf40 0x7f2f0c1eeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f03e00640 1 -- 172.21.15.98:0/1077565536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f0c1547c0 con 0x7f2f0c152850 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f03e00640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c152850 0x7f2f0c0cd090 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c520 tx=0x7f2ef400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f08c00640 1 -- 172.21.15.98:0/1077565536 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4018070 con 0x7f2f0c152850 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f08c00640 1 -- 172.21.15.98:0/1077565536 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ef40074e0 con 0x7f2f0c152850 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f11400640 1 -- 172.21.15.98:0/1077565536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f0c153210 con 0x7f2f0c152850 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f08c00640 1 -- 172.21.15.98:0/1077565536 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef40049f0 con 0x7f2f0c152850 2024-05-27T00:15:19.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f11400640 1 -- 172.21.15.98:0/1077565536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f0c1537d0 con 0x7f2f0c152850 2024-05-27T00:15:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f11400640 1 -- 172.21.15.98:0/1077565536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f0c075c40 con 0x7f2f0c152850 2024-05-27T00:15:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.274+0000 7f2f08c00640 1 -- 172.21.15.98:0/1077565536 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2ef401b070 con 0x7f2f0c152850 2024-05-27T00:15:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.278+0000 7f2f08c00640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ee80678c0 0x7f2ee8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.278+0000 7f2f08c00640 1 -- 172.21.15.98:0/1077565536 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f2ef4084d80 con 0x7f2f0c152850 2024-05-27T00:15:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.278+0000 7f2f0be00640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ee80678c0 0x7f2ee8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.278+0000 7f2f0be00640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ee80678c0 0x7f2ee8069d80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00b9d0 tx=0x7f2efc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.282+0000 7f2f08c00640 1 -- 172.21.15.98:0/1077565536 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f2ef4055520 con 0x7f2f0c152850 2024-05-27T00:15:19.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 -- 172.21.15.98:0/1077565536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f0c15b890 con 0x7f2f0c152850 2024-05-27T00:15:19.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f08c00640 1 -- 172.21.15.98:0/1077565536 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f2ef4059820 con 0x7f2f0c152850 2024-05-27T00:15:19.472 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:19.472 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T00:15:19.473 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 2 2024-05-27T00:15:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 -- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ee80678c0 msgr2=0x7f2ee8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ee80678c0 0x7f2ee8069d80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00b9d0 tx=0x7f2efc008040 comp rx=0 tx=0).stop 2024-05-27T00:15:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 -- 172.21.15.98:0/1077565536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c152850 msgr2=0x7f2f0c0cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c152850 0x7f2f0c0cd090 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c520 tx=0x7f2ef400c9e0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 -- 172.21.15.98:0/1077565536 shutdown_connections 2024-05-27T00:15:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ee80678c0 0x7f2ee8069d80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f0c15bf40 0x7f2f0c1eeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0c152850 0x7f2f0c0cd090 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 --2- 172.21.15.98:0/1077565536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f0c0c9f30 0x7f2f0c154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.470+0000 7f2f11400640 1 -- 172.21.15.98:0/1077565536 >> 172.21.15.98:0/1077565536 conn(0x7f2f0c0cdeb0 msgr2=0x7f2f0c14feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.474+0000 7f2f11400640 1 -- 172.21.15.98:0/1077565536 shutdown_connections 2024-05-27T00:15:19.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:19.474+0000 7f2f11400640 1 -- 172.21.15.98:0/1077565536 wait complete. 2024-05-27T00:15:21.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:15:21.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb547000640 1 Processor -- start 2024-05-27T00:15:21.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb547000640 1 -- start start 2024-05-27T00:15:21.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb547000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5400c4fd0 0x7fb54006f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:21.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb547000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb5400ce370 0x7fb540165140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:21.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb547000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb54006cfb0 con 0x7fb5400cdf90 2024-05-27T00:15:21.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb547000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb54006d500 con 0x7fb5400ce370 2024-05-27T00:15:21.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb547000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb54006ef50 con 0x7fb5400c4fd0 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb53fe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb5400cdf90 0x7fb5400c8fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43320/0 (socket says 172.21.15.98:43320) 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb53fe00640 1 -- 172.21.15.98:0/4268912163 learned_addr learned my addr 172.21.15.98:0/4268912163 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb537e00640 1 --2- 172.21.15.98:0/4268912163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5400c4fd0 0x7fb54006f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb545000640 1 --2- 172.21.15.98:0/4268912163 >> v2:172.21.15.204:3300/0 conn(0x7fb5400ce370 0x7fb540165140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb53f400640 1 -- 172.21.15.98:0/4268912163 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1246468260 0 0) 0x7fb54006cfb0 con 0x7fb5400cdf90 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb53f400640 1 -- 172.21.15.98:0/4268912163 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb524003530 con 0x7fb5400cdf90 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb53f400640 1 -- 172.21.15.98:0/4268912163 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1613371478 0 0) 0x7fb524003530 con 0x7fb5400cdf90 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb545000640 1 -- 172.21.15.98:0/4268912163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5400c4fd0 msgr2=0x7fb54006f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb545000640 1 --2- 172.21.15.98:0/4268912163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5400c4fd0 0x7fb54006f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb545000640 1 -- 172.21.15.98:0/4268912163 >> v1:172.21.15.98:6789/0 conn(0x7fb5400cdf90 legacy=0x7fb5400c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:21.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.642+0000 7fb545000640 1 -- 172.21.15.98:0/4268912163 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5400ca840 con 0x7fb5400ce370 2024-05-27T00:15:21.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.646+0000 7fb545000640 1 --2- 172.21.15.98:0/4268912163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5400ce370 0x7fb540165140 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb538009d90 tx=0x7fb53801f080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ccb184420212506 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:21.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.646+0000 7fb53f400640 1 -- 172.21.15.98:0/4268912163 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5380b5070 con 0x7fb5400ce370 2024-05-27T00:15:21.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.646+0000 7fb53f400640 1 -- 172.21.15.98:0/4268912163 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5380a8bd0 con 0x7fb5400ce370 2024-05-27T00:15:21.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.646+0000 7fb53f400640 1 -- 172.21.15.98:0/4268912163 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530001680 con 0x7fb5400ce370 2024-05-27T00:15:21.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.646+0000 7fb547000640 1 -- 172.21.15.98:0/4268912163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5400ce370 msgr2=0x7fb540165140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:21.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.646+0000 7fb547000640 1 --2- 172.21.15.98:0/4268912163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5400ce370 0x7fb540165140 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb538009d90 tx=0x7fb53801f080 comp rx=0 tx=0).stop 2024-05-27T00:15:21.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.646+0000 7fb547000640 1 -- 172.21.15.98:0/4268912163 shutdown_connections 2024-05-27T00:15:21.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.646+0000 7fb547000640 1 --2- 172.21.15.98:0/4268912163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5400ce370 0x7fb540165140 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:21.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.646+0000 7fb547000640 1 --2- 172.21.15.98:0/4268912163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5400c4fd0 0x7fb54006f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:21.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.646+0000 7fb547000640 1 -- 172.21.15.98:0/4268912163 >> 172.21.15.98:0/4268912163 conn(0x7fb54006d680 msgr2=0x7fb54006da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:21.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 -- 172.21.15.98:0/4268912163 shutdown_connections 2024-05-27T00:15:21.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 -- 172.21.15.98:0/4268912163 wait complete. 2024-05-27T00:15:21.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 Processor -- start 2024-05-27T00:15:21.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 -- start start 2024-05-27T00:15:21.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5400c4fd0 0x7fb540156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:21.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5400cdf90 0x7fb540150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:21.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5400ce370 0x7fb540150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:21.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400ce7f0 con 0x7fb5400cdf90 2024-05-27T00:15:21.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb54006b610 con 0x7fb5400ce370 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb54006d220 con 0x7fb5400c4fd0 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb537e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5400cdf90 0x7fb540150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb537e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5400cdf90 0x7fb540150590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36078/0 (socket says 172.21.15.98:36078) 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb537e00640 1 -- 172.21.15.98:0/550241250 learned_addr learned my addr 172.21.15.98:0/550241250 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb545000640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5400ce370 0x7fb540150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb53fe00640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5400c4fd0 0x7fb540156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb537e00640 1 -- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5400c4fd0 msgr2=0x7fb540156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb537e00640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5400c4fd0 0x7fb540156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb537e00640 1 -- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5400ce370 msgr2=0x7fb540150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb537e00640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5400ce370 0x7fb540150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb537e00640 1 -- 172.21.15.98:0/550241250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5401524f0 con 0x7fb5400cdf90 2024-05-27T00:15:21.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb537e00640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5400cdf90 0x7fb540150590 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e900 tx=0x7fb52800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:21.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb53cc00640 1 -- 172.21.15.98:0/550241250 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528017070 con 0x7fb5400cdf90 2024-05-27T00:15:21.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb53cc00640 1 -- 172.21.15.98:0/550241250 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb52800ca00 con 0x7fb5400cdf90 2024-05-27T00:15:21.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb53cc00640 1 -- 172.21.15.98:0/550241250 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52800cdf0 con 0x7fb5400cdf90 2024-05-27T00:15:21.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 -- 172.21.15.98:0/550241250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5401527d0 con 0x7fb5400cdf90 2024-05-27T00:15:21.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.650+0000 7fb547000640 1 -- 172.21.15.98:0/550241250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb540152d90 con 0x7fb5400cdf90 2024-05-27T00:15:21.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.654+0000 7fb547000640 1 -- 172.21.15.98:0/550241250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb540075cb0 con 0x7fb5400cdf90 2024-05-27T00:15:21.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.654+0000 7fb53cc00640 1 -- 172.21.15.98:0/550241250 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb52801a070 con 0x7fb5400cdf90 2024-05-27T00:15:21.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.654+0000 7fb53cc00640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb51c0678f0 0x7fb51c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:21.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.654+0000 7fb53cc00640 1 -- 172.21.15.98:0/550241250 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fb528085d70 con 0x7fb5400cdf90 2024-05-27T00:15:21.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.654+0000 7fb53fe00640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb51c0678f0 0x7fb51c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:21.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.654+0000 7fb53fe00640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb51c0678f0 0x7fb51c069db0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb530007a10 tx=0x7fb530009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:21.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.658+0000 7fb53cc00640 1 -- 172.21.15.98:0/550241250 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb528056b60 con 0x7fb5400cdf90 2024-05-27T00:15:21.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.846+0000 7fb547000640 1 -- 172.21.15.98:0/550241250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb54006f540 con 0x7fb5400cdf90 2024-05-27T00:15:21.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.846+0000 7fb53cc00640 1 -- 172.21.15.98:0/550241250 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5521 (secure 0 0 0) 0x7fb528056460 con 0x7fb5400cdf90 2024-05-27T00:15:21.850 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-27T00:15:21.850 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2024-05-27T00:15:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 -- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb51c0678f0 msgr2=0x7fb51c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb51c0678f0 0x7fb51c069db0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb530007a10 tx=0x7fb530009040 comp rx=0 tx=0).stop 2024-05-27T00:15:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 -- 172.21.15.98:0/550241250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5400cdf90 msgr2=0x7fb540150590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5400cdf90 0x7fb540150590 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e900 tx=0x7fb52800edc0 comp rx=0 tx=0).stop 2024-05-27T00:15:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 -- 172.21.15.98:0/550241250 shutdown_connections 2024-05-27T00:15:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb51c0678f0 0x7fb51c069db0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5400ce370 0x7fb540150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:21.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5400cdf90 0x7fb540150590 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:21.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 --2- 172.21.15.98:0/550241250 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5400c4fd0 0x7fb540156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:21.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 -- 172.21.15.98:0/550241250 >> 172.21.15.98:0/550241250 conn(0x7fb54006d680 msgr2=0x7fb5400c3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:21.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 -- 172.21.15.98:0/550241250 shutdown_connections 2024-05-27T00:15:21.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:21.850+0000 7fb547000640 1 -- 172.21.15.98:0/550241250 wait complete. 2024-05-27T00:15:21.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.002+0000 7f277f000640 1 Processor -- start 2024-05-27T00:15:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.002+0000 7f277f000640 1 -- start start 2024-05-27T00:15:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277f000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27780b3260 0x7f27780c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277f000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2778070660 0x7f27780c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277f000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f27780c36c0 0x7f27780c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277f000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2778071010 con 0x7f27780b3260 2024-05-27T00:15:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277f000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f27780c88c0 con 0x7f27780c36c0 2024-05-27T00:15:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277f000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f27780c8b80 con 0x7f2778070660 2024-05-27T00:15:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277dc00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27780b3260 0x7f27780c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277dc00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27780b3260 0x7f27780c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36092/0 (socket says 172.21.15.98:36092) 2024-05-27T00:15:22.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277dc00640 1 -- 172.21.15.98:0/2049253122 learned_addr learned my addr 172.21.15.98:0/2049253122 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277e600640 1 --2- 172.21.15.98:0/2049253122 >> v2:172.21.15.204:3300/0 conn(0x7f27780c36c0 0x7f27780c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277d200640 1 --2- 172.21.15.98:0/2049253122 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2778070660 0x7f27780c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277dc00640 1 -- 172.21.15.98:0/2049253122 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2778070660 msgr2=0x7f27780c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277dc00640 1 --2- 172.21.15.98:0/2049253122 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2778070660 0x7f27780c3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277dc00640 1 -- 172.21.15.98:0/2049253122 >> v2:172.21.15.204:3300/0 conn(0x7f27780c36c0 msgr2=0x7f27780c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277dc00640 1 --2- 172.21.15.98:0/2049253122 >> v2:172.21.15.204:3300/0 conn(0x7f27780c36c0 0x7f27780c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277dc00640 1 -- 172.21.15.98:0/2049253122 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27780c8e60 con 0x7f27780b3260 2024-05-27T00:15:22.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.006+0000 7f277e600640 1 --2- 172.21.15.98:0/2049253122 >> v2:172.21.15.204:3300/0 conn(0x7f27780c36c0 0x7f27780c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:22.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277dc00640 1 --2- 172.21.15.98:0/2049253122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780b3260 0x7f27780c7cc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f27780cbd60 tx=0x7f276c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73105aa2211cd719 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:22.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f2777e00640 1 -- 172.21.15.98:0/2049253122 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c0b5070 con 0x7f27780b3260 2024-05-27T00:15:22.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f2777e00640 1 -- 172.21.15.98:0/2049253122 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f276c0a89d0 con 0x7f27780b3260 2024-05-27T00:15:22.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f2777e00640 1 -- 172.21.15.98:0/2049253122 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c0a8da0 con 0x7f27780b3260 2024-05-27T00:15:22.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 -- 172.21.15.98:0/2049253122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780b3260 msgr2=0x7f27780c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 --2- 172.21.15.98:0/2049253122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780b3260 0x7f27780c7cc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f27780cbd60 tx=0x7f276c01ed10 comp rx=0 tx=0).stop 2024-05-27T00:15:22.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 -- 172.21.15.98:0/2049253122 shutdown_connections 2024-05-27T00:15:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 --2- 172.21.15.98:0/2049253122 >> v2:172.21.15.204:3300/0 conn(0x7f27780c36c0 0x7f27780c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 --2- 172.21.15.98:0/2049253122 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2778070660 0x7f27780c3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 --2- 172.21.15.98:0/2049253122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780b3260 0x7f27780c7cc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 -- 172.21.15.98:0/2049253122 >> 172.21.15.98:0/2049253122 conn(0x7f27780cdeb0 msgr2=0x7f27780ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 -- 172.21.15.98:0/2049253122 shutdown_connections 2024-05-27T00:15:22.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 -- 172.21.15.98:0/2049253122 wait complete. 2024-05-27T00:15:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 Processor -- start 2024-05-27T00:15:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.010+0000 7f277f000640 1 -- start start 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277f000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2778070660 0x7f27780cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277f000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c36c0 0x7f27781e56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277f000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27781e5c20 0x7f27781dde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277f000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27780c8200 con 0x7f27780c36c0 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277f000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f27780c9420 con 0x7f2778070660 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277f000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f27780c9940 con 0x7f27781e5c20 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c36c0 0x7f27781e56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c36c0 0x7f27781e56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36102/0 (socket says 172.21.15.98:36102) 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277d200640 1 -- 172.21.15.98:0/2044263803 learned_addr learned my addr 172.21.15.98:0/2044263803 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277e600640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27781e5c20 0x7f27781dde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277dc00640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2778070660 0x7f27780cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277d200640 1 -- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27781e5c20 msgr2=0x7f27781dde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277d200640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27781e5c20 0x7f27781dde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277d200640 1 -- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2778070660 msgr2=0x7f27780cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277d200640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2778070660 0x7f27780cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277d200640 1 -- 172.21.15.98:0/2044263803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27781e3c50 con 0x7f27780c36c0 2024-05-27T00:15:22.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277d200640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c36c0 0x7f27781e56e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f276400d800 tx=0x7f276400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f2775600640 1 -- 172.21.15.98:0/2044263803 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764018030 con 0x7f27780c36c0 2024-05-27T00:15:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f2775600640 1 -- 172.21.15.98:0/2044263803 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27640096a0 con 0x7f27780c36c0 2024-05-27T00:15:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277f000640 1 -- 172.21.15.98:0/2044263803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27781f02b0 con 0x7f27780c36c0 2024-05-27T00:15:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f2775600640 1 -- 172.21.15.98:0/2044263803 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764009870 con 0x7f27780c36c0 2024-05-27T00:15:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277f000640 1 -- 172.21.15.98:0/2044263803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27781f0870 con 0x7f27780c36c0 2024-05-27T00:15:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f277f000640 1 -- 172.21.15.98:0/2044263803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27781e6030 con 0x7f27780c36c0 2024-05-27T00:15:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.014+0000 7f2775600640 1 -- 172.21.15.98:0/2044263803 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f276401b070 con 0x7f27780c36c0 2024-05-27T00:15:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.018+0000 7f2775600640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 0x7f2758069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.018+0000 7f2775600640 1 -- 172.21.15.98:0/2044263803 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f2764085b70 con 0x7f27780c36c0 2024-05-27T00:15:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.018+0000 7f277dc00640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 0x7f2758069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.018+0000 7f277dc00640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 0x7f2758069d80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f276c0b2000 tx=0x7f276c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:22.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.022+0000 7f2775600640 1 -- 172.21.15.98:0/2044263803 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f27781e6030 con 0x7f27780c36c0 2024-05-27T00:15:22.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 -- 172.21.15.98:0/2044263803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27780c5480 con 0x7f27780c36c0 2024-05-27T00:15:22.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f2775600640 1 -- 172.21.15.98:0/2044263803 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12860 (secure 0 0 0) 0x7f2764056320 con 0x7f27780c36c0 2024-05-27T00:15:22.180 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:22.180 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:15.428401+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 -- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 msgr2=0x7f2758069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 0x7f2758069d80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f276c0b2000 tx=0x7f276c009040 comp rx=0 tx=0).stop 2024-05-27T00:15:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 -- 172.21.15.98:0/2044263803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c36c0 msgr2=0x7f27781e56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c36c0 0x7f27781e56e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f276400d800 tx=0x7f276400dcc0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 -- 172.21.15.98:0/2044263803 shutdown_connections 2024-05-27T00:15:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27781e5c20 0x7f27781dde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c36c0 0x7f27781e56e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 0x7f2758069d80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 --2- 172.21.15.98:0/2044263803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2778070660 0x7f27780cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 -- 172.21.15.98:0/2044263803 >> 172.21.15.98:0/2044263803 conn(0x7f27780cdeb0 msgr2=0x7f27780c4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.178+0000 7f277f000640 1 -- 172.21.15.98:0/2044263803 shutdown_connections 2024-05-27T00:15:22.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.182+0000 7f277f000640 1 -- 172.21.15.98:0/2044263803 wait complete. 2024-05-27T00:15:22.201 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T00:15:13.211703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T00:15:22.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T00:15:22.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f5300e00640 1 Processor -- start 2024-05-27T00:15:22.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f5300e00640 1 -- start start 2024-05-27T00:15:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f5300e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52fc152990 0x7f52fc06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f5300e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52fc06ecc0 0x7f52fc0c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f5300e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52fc15bc40 con 0x7f52fc0b1c20 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f5300e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f52fc15c850 con 0x7f52fc06ecc0 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f5300e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f52fc15bf20 con 0x7f52fc152990 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f52f9400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52fc06ecc0 0x7f52fc0c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f52fa800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f52fc0b1c20 0x7f52fc158640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43322/0 (socket says 172.21.15.98:43322) 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f52f9400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f52fc06ecc0 0x7f52fc0c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36112/0 (socket says 172.21.15.98:36112) 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.338+0000 7f52fa800640 1 -- 172.21.15.98:0/928072523 learned_addr learned my addr 172.21.15.98:0/928072523 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f52f9e00640 1 --2- 172.21.15.98:0/928072523 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52fc152990 0x7f52fc06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f52f9400640 1 -- 172.21.15.98:0/928072523 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52fc152990 msgr2=0x7f52fc06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f52f9400640 1 --2- 172.21.15.98:0/928072523 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52fc152990 0x7f52fc06e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f52f9400640 1 -- 172.21.15.98:0/928072523 >> v1:172.21.15.98:6789/0 conn(0x7f52fc0b1c20 legacy=0x7f52fc158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f52f9400640 1 -- 172.21.15.98:0/928072523 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc15c200 con 0x7f52fc06ecc0 2024-05-27T00:15:22.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f52f9400640 1 --2- 172.21.15.98:0/928072523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc06ecc0 0x7f52fc0c7f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f52e80097c0 tx=0x7f52e801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8d0c295a02312a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:22.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f52f8a00640 1 -- 172.21.15.98:0/928072523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e80b0070 con 0x7f52fc06ecc0 2024-05-27T00:15:22.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f52f8a00640 1 -- 172.21.15.98:0/928072523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e80ab600 con 0x7f52fc06ecc0 2024-05-27T00:15:22.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f52f8a00640 1 -- 172.21.15.98:0/928072523 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e80ab9d0 con 0x7f52fc06ecc0 2024-05-27T00:15:22.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f5300e00640 1 -- 172.21.15.98:0/928072523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc06ecc0 msgr2=0x7f52fc0c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.342+0000 7f5300e00640 1 --2- 172.21.15.98:0/928072523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc06ecc0 0x7f52fc0c7f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f52e80097c0 tx=0x7f52e801c4a0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 -- 172.21.15.98:0/928072523 shutdown_connections 2024-05-27T00:15:22.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 --2- 172.21.15.98:0/928072523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc06ecc0 0x7f52fc0c7f20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 --2- 172.21.15.98:0/928072523 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52fc152990 0x7f52fc06e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 -- 172.21.15.98:0/928072523 >> 172.21.15.98:0/928072523 conn(0x7f52fc0cbf00 msgr2=0x7f52fc0cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:22.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 -- 172.21.15.98:0/928072523 shutdown_connections 2024-05-27T00:15:22.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 -- 172.21.15.98:0/928072523 wait complete. 2024-05-27T00:15:22.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 Processor -- start 2024-05-27T00:15:22.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 -- start start 2024-05-27T00:15:22.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc06ecc0 0x7f52fc0c1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52fc0b1c20 0x7f52fc0c1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc152990 0x7f52fc15a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc1659e0 con 0x7f52fc152990 2024-05-27T00:15:22.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc165860 con 0x7f52fc06ecc0 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f52fc165b60 con 0x7f52fc0b1c20 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52fa800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc152990 0x7f52fc15a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52fa800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc152990 0x7f52fc15a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36126/0 (socket says 172.21.15.98:36126) 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52fa800640 1 -- 172.21.15.98:0/2637411547 learned_addr learned my addr 172.21.15.98:0/2637411547 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52f9e00640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc06ecc0 0x7f52fc0c1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52f9400640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52fc0b1c20 0x7f52fc0c1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52fa800640 1 -- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52fc0b1c20 msgr2=0x7f52fc0c1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52fa800640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52fc0b1c20 0x7f52fc0c1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52fa800640 1 -- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc06ecc0 msgr2=0x7f52fc0c1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52fa800640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc06ecc0 0x7f52fc0c1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52fa800640 1 -- 172.21.15.98:0/2637411547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc0c3dd0 con 0x7f52fc152990 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52fa800640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc152990 0x7f52fc15a520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f52f400c400 tx=0x7f52f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52f2000640 1 -- 172.21.15.98:0/2637411547 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4017070 con 0x7f52fc152990 2024-05-27T00:15:22.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52f2000640 1 -- 172.21.15.98:0/2637411547 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f4005cf0 con 0x7f52fc152990 2024-05-27T00:15:22.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f52f2000640 1 -- 172.21.15.98:0/2637411547 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f40072d0 con 0x7f52fc152990 2024-05-27T00:15:22.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 -- 172.21.15.98:0/2637411547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52fc0c9120 con 0x7f52fc152990 2024-05-27T00:15:22.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.346+0000 7f5300e00640 1 -- 172.21.15.98:0/2637411547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52fc0c96e0 con 0x7f52fc152990 2024-05-27T00:15:22.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.350+0000 7f5300e00640 1 -- 172.21.15.98:0/2637411547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52fc075cb0 con 0x7f52fc152990 2024-05-27T00:15:22.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.350+0000 7f52f2000640 1 -- 172.21.15.98:0/2637411547 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f52f401a070 con 0x7f52fc152990 2024-05-27T00:15:22.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.350+0000 7f52f2000640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52dc0678c0 0x7f52dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.350+0000 7f52f2000640 1 -- 172.21.15.98:0/2637411547 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f52f4084ca0 con 0x7f52fc152990 2024-05-27T00:15:22.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.350+0000 7f52f9e00640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52dc0678c0 0x7f52dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.350+0000 7f52f9e00640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52dc0678c0 0x7f52dc069d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f52ec003fd0 tx=0x7f52ec0039a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:22.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.354+0000 7f52f2000640 1 -- 172.21.15.98:0/2637411547 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f52f4055440 con 0x7f52fc152990 2024-05-27T00:15:22.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 -- 172.21.15.98:0/2637411547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f52fc0cc410 con 0x7f52fc152990 2024-05-27T00:15:22.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f52f2000640 1 -- 172.21.15.98:0/2637411547 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) ==== 93+0+10 (secure 0 0 0) 0x7f52f4059740 con 0x7f52fc152990 2024-05-27T00:15:22.515 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-27T00:15:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 -- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52dc0678c0 msgr2=0x7f52dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52dc0678c0 0x7f52dc069d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f52ec003fd0 tx=0x7f52ec0039a0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 -- 172.21.15.98:0/2637411547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc152990 msgr2=0x7f52fc15a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc152990 0x7f52fc15a520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f52f400c400 tx=0x7f52f400c8c0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 -- 172.21.15.98:0/2637411547 shutdown_connections 2024-05-27T00:15:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc152990 0x7f52fc15a520 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52fc0b1c20 0x7f52fc0c1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52dc0678c0 0x7f52dc069d80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 --2- 172.21.15.98:0/2637411547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc06ecc0 0x7f52fc0c1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 -- 172.21.15.98:0/2637411547 >> 172.21.15.98:0/2637411547 conn(0x7f52fc0cbf00 msgr2=0x7f52fc161590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 -- 172.21.15.98:0/2637411547 shutdown_connections 2024-05-27T00:15:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.514+0000 7f5300e00640 1 -- 172.21.15.98:0/2637411547 wait complete. 2024-05-27T00:15:22.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-27T00:15:22.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b0d800640 1 Processor -- start 2024-05-27T00:15:22.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b0d800640 1 -- start start 2024-05-27T00:15:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b0d800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b08158080 0x7f3b0806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b0d800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3b080c9a80 0x7f3b080c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b0d800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b080730e0 0x7f3b08073570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b0d800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3b080c9820 con 0x7f3b08158080 2024-05-27T00:15:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b0d800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f3b0806cb30 con 0x7f3b080c9a80 2024-05-27T00:15:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b0d800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3b080c7b20 con 0x7f3b080730e0 2024-05-27T00:15:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b06a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b08158080 0x7f3b0806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b06a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b08158080 0x7f3b0806b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36142/0 (socket says 172.21.15.98:36142) 2024-05-27T00:15:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.670+0000 7f3b06a00640 1 -- 172.21.15.98:0/2192945066 learned_addr learned my addr 172.21.15.98:0/2192945066 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b07400640 1 --2- 172.21.15.98:0/2192945066 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b080730e0 0x7f3b08073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3afe000640 1 --2- 172.21.15.98:0/2192945066 >> v2:172.21.15.204:3300/0 conn(0x7f3b080c9a80 0x7f3b080c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b06a00640 1 -- 172.21.15.98:0/2192945066 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b080730e0 msgr2=0x7f3b08073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b06a00640 1 --2- 172.21.15.98:0/2192945066 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b080730e0 0x7f3b08073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b06a00640 1 -- 172.21.15.98:0/2192945066 >> v2:172.21.15.204:3300/0 conn(0x7f3b080c9a80 msgr2=0x7f3b080c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b06a00640 1 --2- 172.21.15.98:0/2192945066 >> v2:172.21.15.204:3300/0 conn(0x7f3b080c9a80 0x7f3b080c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b06a00640 1 -- 172.21.15.98:0/2192945066 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0806c570 con 0x7f3b08158080 2024-05-27T00:15:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b06a00640 1 --2- 172.21.15.98:0/2192945066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b08158080 0x7f3b0806b970 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3b080ca6a0 tx=0x7f3af801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba86d25ee8cccaa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b06000640 1 -- 172.21.15.98:0/2192945066 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af80a5670 con 0x7f3b08158080 2024-05-27T00:15:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b06000640 1 -- 172.21.15.98:0/2192945066 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af80a5810 con 0x7f3b08158080 2024-05-27T00:15:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b06000640 1 -- 172.21.15.98:0/2192945066 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af80a5be0 con 0x7f3b08158080 2024-05-27T00:15:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b0d800640 1 -- 172.21.15.98:0/2192945066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b08158080 msgr2=0x7f3b0806b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.674+0000 7f3b0d800640 1 --2- 172.21.15.98:0/2192945066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b08158080 0x7f3b0806b970 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3b080ca6a0 tx=0x7f3af801c9f0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 -- 172.21.15.98:0/2192945066 shutdown_connections 2024-05-27T00:15:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 --2- 172.21.15.98:0/2192945066 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b080730e0 0x7f3b08073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 --2- 172.21.15.98:0/2192945066 >> v2:172.21.15.204:3300/0 conn(0x7f3b080c9a80 0x7f3b080c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 --2- 172.21.15.98:0/2192945066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b08158080 0x7f3b0806b970 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 -- 172.21.15.98:0/2192945066 >> 172.21.15.98:0/2192945066 conn(0x7f3b0806fa70 msgr2=0x7f3b0806fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 -- 172.21.15.98:0/2192945066 shutdown_connections 2024-05-27T00:15:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 -- 172.21.15.98:0/2192945066 wait complete. 2024-05-27T00:15:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 Processor -- start 2024-05-27T00:15:22.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 -- start start 2024-05-27T00:15:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080730e0 0x7f3b081f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b080c9a80 0x7f3b081439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b08158080 0x7f3b081484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b080c41b0 con 0x7f3b080730e0 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3b080c4030 con 0x7f3b080c9a80 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3b0806bef0 con 0x7f3b08158080 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b06a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080730e0 0x7f3b081f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b06a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080730e0 0x7f3b081f76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36156/0 (socket says 172.21.15.98:36156) 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b06a00640 1 -- 172.21.15.98:0/977435431 learned_addr learned my addr 172.21.15.98:0/977435431 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3afe000640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b080c9a80 0x7f3b081439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b06a00640 1 -- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b08158080 msgr2=0x7f3b081484b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b07400640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b08158080 0x7f3b081484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b06a00640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b08158080 0x7f3b081484b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b06a00640 1 -- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b080c9a80 msgr2=0x7f3b081439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b06a00640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b080c9a80 0x7f3b081439f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b06a00640 1 -- 172.21.15.98:0/977435431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b081f3620 con 0x7f3b080730e0 2024-05-27T00:15:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b06a00640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080730e0 0x7f3b081f76f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3af80af040 tx=0x7f3af8019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3aff400640 1 -- 172.21.15.98:0/977435431 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af8009510 con 0x7f3b080730e0 2024-05-27T00:15:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3aff400640 1 -- 172.21.15.98:0/977435431 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af80be070 con 0x7f3b080730e0 2024-05-27T00:15:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3b0d800640 1 -- 172.21.15.98:0/977435431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b081f38e0 con 0x7f3b080730e0 2024-05-27T00:15:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.678+0000 7f3aff400640 1 -- 172.21.15.98:0/977435431 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af80b8870 con 0x7f3b080730e0 2024-05-27T00:15:22.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.682+0000 7f3b0d800640 1 -- 172.21.15.98:0/977435431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b081f3ea0 con 0x7f3b080730e0 2024-05-27T00:15:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.682+0000 7f3b0d800640 1 -- 172.21.15.98:0/977435431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b08075cb0 con 0x7f3b080730e0 2024-05-27T00:15:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.682+0000 7f3aff400640 1 -- 172.21.15.98:0/977435431 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3af80b8a10 con 0x7f3b080730e0 2024-05-27T00:15:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.682+0000 7f3aff400640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3ae40678f0 0x7f3ae4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.682+0000 7f3aff400640 1 -- 172.21.15.98:0/977435431 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f3af8120930 con 0x7f3b080730e0 2024-05-27T00:15:22.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.682+0000 7f3afe000640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3ae40678f0 0x7f3ae4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:22.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.682+0000 7f3afe000640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3ae40678f0 0x7f3ae4069db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3b08147f30 tx=0x7f3af0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.686+0000 7f3aff400640 1 -- 172.21.15.98:0/977435431 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f3af80165e0 con 0x7f3b080730e0 2024-05-27T00:15:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.842+0000 7f3b0d800640 1 -- 172.21.15.98:0/977435431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3b08070810 con 0x7f3b080730e0 2024-05-27T00:15:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3aff400640 1 -- 172.21.15.98:0/977435431 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) ==== 84+0+65 (secure 0 0 0) 0x7f3af80f17c0 con 0x7f3b080730e0 2024-05-27T00:15:22.846 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:22.846 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2024-05-27T00:15:22.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-05-27T00:15:22.846 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2024-05-27T00:15:22.846 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2024-05-27T00:15:22.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-05-27T00:15:22.847 INFO:teuthology.orchestra.run.smithi098.stdout:] 2024-05-27T00:15:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 -- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3ae40678f0 msgr2=0x7f3ae4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3ae40678f0 0x7f3ae4069db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3b08147f30 tx=0x7f3af0008040 comp rx=0 tx=0).stop 2024-05-27T00:15:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 -- 172.21.15.98:0/977435431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080730e0 msgr2=0x7f3b081f76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080730e0 0x7f3b081f76f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3af80af040 tx=0x7f3af8019790 comp rx=0 tx=0).stop 2024-05-27T00:15:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 -- 172.21.15.98:0/977435431 shutdown_connections 2024-05-27T00:15:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b08158080 0x7f3b081484b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3ae40678f0 0x7f3ae4069db0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b080c9a80 0x7f3b081439f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 --2- 172.21.15.98:0/977435431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b080730e0 0x7f3b081f76f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 -- 172.21.15.98:0/977435431 >> 172.21.15.98:0/977435431 conn(0x7f3b0806fa70 msgr2=0x7f3b08071f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 -- 172.21.15.98:0/977435431 shutdown_connections 2024-05-27T00:15:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:22.846+0000 7f3b0d800640 1 -- 172.21.15.98:0/977435431 wait complete. 2024-05-27T00:15:22.867 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-27T00:15:22.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-27T00:15:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4894400640 1 Processor -- start 2024-05-27T00:15:23.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4894400640 1 -- start start 2024-05-27T00:15:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4894400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4894400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f488c0c2b10 0x7f488c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4894400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f488c0c8420 con 0x7f488c1516b0 2024-05-27T00:15:23.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4894400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f488c0c1df0 con 0x7f488c0c2b10 2024-05-27T00:15:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4894400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f488c0c3000 con 0x7f488c0b1e10 2024-05-27T00:15:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4891000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4892400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f488c0c2b10 0x7f488c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4891a00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f488c1516b0 0x7f488c0c1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:51110/0 (socket says 172.21.15.98:51110) 2024-05-27T00:15:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4891000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c06e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:49788/0 (socket says 172.21.15.98:49788) 2024-05-27T00:15:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.006+0000 7f4891a00640 1 -- 172.21.15.98:0/1871706961 learned_addr learned my addr 172.21.15.98:0/1871706961 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:23.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.010+0000 7f4892400640 1 -- 172.21.15.98:0/1871706961 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 msgr2=0x7f488c06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:23.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.010+0000 7f4892400640 1 --2- 172.21.15.98:0/1871706961 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c06e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.010+0000 7f4892400640 1 -- 172.21.15.98:0/1871706961 >> v1:172.21.15.204:6789/0 conn(0x7f488c1516b0 legacy=0x7f488c0c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:23.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.010+0000 7f4892400640 1 -- 172.21.15.98:0/1871706961 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c0c7e60 con 0x7f488c0c2b10 2024-05-27T00:15:23.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.010+0000 7f4891000640 1 --2- 172.21.15.98:0/1871706961 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c06e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:23.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.010+0000 7f4892400640 1 --2- 172.21.15.98:0/1871706961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c0c2b10 0x7f488c06ecc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4884009dc0 tx=0x7f488401c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd5144bccfae6b9c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:23.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.010+0000 7f488be00640 1 -- 172.21.15.98:0/1871706961 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48840b2070 con 0x7f488c0c2b10 2024-05-27T00:15:23.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.010+0000 7f488be00640 1 -- 172.21.15.98:0/1871706961 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48840a5900 con 0x7f488c0c2b10 2024-05-27T00:15:23.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.010+0000 7f488be00640 1 -- 172.21.15.98:0/1871706961 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48840a5cd0 con 0x7f488c0c2b10 2024-05-27T00:15:23.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 -- 172.21.15.98:0/1871706961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c0c2b10 msgr2=0x7f488c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:23.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 --2- 172.21.15.98:0/1871706961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c0c2b10 0x7f488c06ecc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4884009dc0 tx=0x7f488401c950 comp rx=0 tx=0).stop 2024-05-27T00:15:23.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 -- 172.21.15.98:0/1871706961 shutdown_connections 2024-05-27T00:15:23.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 --2- 172.21.15.98:0/1871706961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c0c2b10 0x7f488c06ecc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 --2- 172.21.15.98:0/1871706961 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c06e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 -- 172.21.15.98:0/1871706961 >> 172.21.15.98:0/1871706961 conn(0x7f488c0cbf00 msgr2=0x7f488c0cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:23.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 -- 172.21.15.98:0/1871706961 shutdown_connections 2024-05-27T00:15:23.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 -- 172.21.15.98:0/1871706961 wait complete. 2024-05-27T00:15:23.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 Processor -- start 2024-05-27T00:15:23.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 -- start start 2024-05-27T00:15:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c13b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0c2b10 0x7f488c140c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c1516b0 0x7f488c0cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f488c151c10 con 0x7f488c0c2b10 2024-05-27T00:15:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f488c151a90 con 0x7f488c1516b0 2024-05-27T00:15:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4894400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f488c06cfe0 con 0x7f488c0b1e10 2024-05-27T00:15:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0c2b10 0x7f488c140c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0c2b10 0x7f488c140c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36168/0 (socket says 172.21.15.98:36168) 2024-05-27T00:15:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891000640 1 -- 172.21.15.98:0/2264039458 learned_addr learned my addr 172.21.15.98:0/2264039458 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891a00640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c13b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4892400640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c1516b0 0x7f488c0cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891000640 1 -- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 msgr2=0x7f488c13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891000640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c13b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891000640 1 -- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c1516b0 msgr2=0x7f488c0cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891000640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c1516b0 0x7f488c0cb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891000640 1 -- 172.21.15.98:0/2264039458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c164a10 con 0x7f488c0c2b10 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891a00640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c13b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.014+0000 7f4891000640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0c2b10 0x7f488c140c30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f487800e9d0 tx=0x7f487800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.018+0000 7f4889600640 1 -- 172.21.15.98:0/2264039458 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878017070 con 0x7f488c0c2b10 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.018+0000 7f4889600640 1 -- 172.21.15.98:0/2264039458 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f487800ca60 con 0x7f488c0c2b10 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.018+0000 7f4894400640 1 -- 172.21.15.98:0/2264039458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488c13fbd0 con 0x7f488c0c2b10 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.018+0000 7f4889600640 1 -- 172.21.15.98:0/2264039458 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487800ce50 con 0x7f488c0c2b10 2024-05-27T00:15:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.018+0000 7f4894400640 1 -- 172.21.15.98:0/2264039458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488c140430 con 0x7f488c0c2b10 2024-05-27T00:15:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.018+0000 7f4894400640 1 -- 172.21.15.98:0/2264039458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f488c075cb0 con 0x7f488c0c2b10 2024-05-27T00:15:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.018+0000 7f4889600640 1 -- 172.21.15.98:0/2264039458 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f487801a070 con 0x7f488c0c2b10 2024-05-27T00:15:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.018+0000 7f4889600640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f486c067a60 0x7f486c069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.018+0000 7f4889600640 1 -- 172.21.15.98:0/2264039458 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f48780854d0 con 0x7f488c0c2b10 2024-05-27T00:15:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.018+0000 7f4891a00640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f486c067a60 0x7f486c069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.022+0000 7f4891a00640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f486c067a60 0x7f486c069f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4880000f70 tx=0x7f4880009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.022+0000 7f4889600640 1 -- 172.21.15.98:0/2264039458 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4878055c70 con 0x7f488c0c2b10 2024-05-27T00:15:23.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.178+0000 7f4894400640 1 -- 172.21.15.98:0/2264039458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f488c0721a0 con 0x7f488c0c2b10 2024-05-27T00:15:23.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4889600640 1 -- 172.21.15.98:0/2264039458 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) ==== 138+0+0 (secure 0 0 0) 0x7f4878059f70 con 0x7f488c0c2b10 2024-05-27T00:15:23.567 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 -- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f486c067a60 msgr2=0x7f486c069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f486c067a60 0x7f486c069f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4880000f70 tx=0x7f4880009040 comp rx=0 tx=0).stop 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 -- 172.21.15.98:0/2264039458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0c2b10 msgr2=0x7f488c140c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0c2b10 0x7f488c140c30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f487800e9d0 tx=0x7f487800ee90 comp rx=0 tx=0).stop 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 -- 172.21.15.98:0/2264039458 shutdown_connections 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f486c067a60 0x7f486c069f20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c1516b0 0x7f488c0cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0c2b10 0x7f488c140c30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 --2- 172.21.15.98:0/2264039458 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f488c0b1e10 0x7f488c13b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 -- 172.21.15.98:0/2264039458 >> 172.21.15.98:0/2264039458 conn(0x7f488c0cbf00 msgr2=0x7f488c0730c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 -- 172.21.15.98:0/2264039458 shutdown_connections 2024-05-27T00:15:23.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.565+0000 7f4894400640 1 -- 172.21.15.98:0/2264039458 wait complete. 2024-05-27T00:15:23.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-27T00:15:23.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484e400640 1 Processor -- start 2024-05-27T00:15:23.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484e400640 1 -- start start 2024-05-27T00:15:23.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484e400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f48481511c0 0x7f484815af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484e400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 0x7f4848071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484e400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48480cee70 con 0x7f484806f140 2024-05-27T00:15:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484e400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f484815b450 con 0x7f48481511c0 2024-05-27T00:15:23.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484e400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f484815ddf0 con 0x7f4848070f10 2024-05-27T00:15:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f483fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 0x7f4848071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484d000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f48481511c0 0x7f484815af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f483fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 0x7f4848071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:34532/0 (socket says 172.21.15.98:34532) 2024-05-27T00:15:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f483fe00640 1 -- 172.21.15.98:0/3605753901 learned_addr learned my addr 172.21.15.98:0/3605753901 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484d000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f48481511c0 0x7f484815af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57426/0 (socket says 172.21.15.98:57426) 2024-05-27T00:15:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484d000640 1 -- 172.21.15.98:0/3605753901 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 msgr2=0x7f4848071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:23.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484d000640 1 --2- 172.21.15.98:0/3605753901 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 0x7f4848071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484d000640 1 -- 172.21.15.98:0/3605753901 >> v1:172.21.15.204:6789/0 conn(0x7f484806f140 legacy=0x7f484806f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:23.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.733+0000 7f484d000640 1 -- 172.21.15.98:0/3605753901 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484815e950 con 0x7f48481511c0 2024-05-27T00:15:23.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f484d000640 1 --2- 172.21.15.98:0/3605753901 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48481511c0 0x7f484815af10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48380097c0 tx=0x7f483801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22b8057933ff26e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:23.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f4847e00640 1 -- 172.21.15.98:0/3605753901 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48380b5070 con 0x7f48481511c0 2024-05-27T00:15:23.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f4847e00640 1 -- 172.21.15.98:0/3605753901 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48380a8990 con 0x7f48481511c0 2024-05-27T00:15:23.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f4847e00640 1 -- 172.21.15.98:0/3605753901 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48380a8d60 con 0x7f48481511c0 2024-05-27T00:15:23.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f484e400640 1 -- 172.21.15.98:0/3605753901 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48481511c0 msgr2=0x7f484815af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:23.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f484e400640 1 --2- 172.21.15.98:0/3605753901 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48481511c0 0x7f484815af10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48380097c0 tx=0x7f483801ec90 comp rx=0 tx=0).stop 2024-05-27T00:15:23.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f484e400640 1 -- 172.21.15.98:0/3605753901 shutdown_connections 2024-05-27T00:15:23.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f484e400640 1 --2- 172.21.15.98:0/3605753901 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 0x7f4848071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f484e400640 1 --2- 172.21.15.98:0/3605753901 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48481511c0 0x7f484815af10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f484e400640 1 -- 172.21.15.98:0/3605753901 >> 172.21.15.98:0/3605753901 conn(0x7f48480cdf90 msgr2=0x7f48480ce3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:23.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f484e400640 1 -- 172.21.15.98:0/3605753901 shutdown_connections 2024-05-27T00:15:23.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.737+0000 7f484e400640 1 -- 172.21.15.98:0/3605753901 wait complete. 2024-05-27T00:15:23.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484e400640 1 Processor -- start 2024-05-27T00:15:23.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484e400640 1 -- start start 2024-05-27T00:15:23.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484e400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f484806f140 0x7f4848155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484e400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 0x7f48481594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484e400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48481511c0 0x7f48480c3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484e400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f484815e390 con 0x7f48481511c0 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484e400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f484815e0b0 con 0x7f484806f140 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484e400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f484815e670 con 0x7f4848070f10 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484da00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48481511c0 0x7f48480c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f484806f140 0x7f4848155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484da00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48481511c0 0x7f48480c3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53336/0 (socket says 172.21.15.98:53336) 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484da00640 1 -- 172.21.15.98:0/2359400893 learned_addr learned my addr 172.21.15.98:0/2359400893 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f483fe00640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 0x7f48481594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484da00640 1 -- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 msgr2=0x7f48481594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484da00640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 0x7f48481594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484da00640 1 -- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f484806f140 msgr2=0x7f4848155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484da00640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f484806f140 0x7f4848155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:23.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484da00640 1 -- 172.21.15.98:0/2359400893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48480c4db0 con 0x7f48481511c0 2024-05-27T00:15:23.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484da00640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48481511c0 0x7f48480c3230 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f484000c730 tx=0x7f484000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:23.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f4845600640 1 -- 172.21.15.98:0/2359400893 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4840003b80 con 0x7f48481511c0 2024-05-27T00:15:23.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f4845600640 1 -- 172.21.15.98:0/2359400893 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4840003d20 con 0x7f48481511c0 2024-05-27T00:15:23.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f484e400640 1 -- 172.21.15.98:0/2359400893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48480c5070 con 0x7f48481511c0 2024-05-27T00:15:23.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.741+0000 7f4845600640 1 -- 172.21.15.98:0/2359400893 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484000aa30 con 0x7f48481511c0 2024-05-27T00:15:23.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.745+0000 7f484e400640 1 -- 172.21.15.98:0/2359400893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48481f1910 con 0x7f48481511c0 2024-05-27T00:15:23.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.745+0000 7f484e400640 1 -- 172.21.15.98:0/2359400893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4848158010 con 0x7f48481511c0 2024-05-27T00:15:23.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.745+0000 7f4845600640 1 -- 172.21.15.98:0/2359400893 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f484000abd0 con 0x7f48481511c0 2024-05-27T00:15:23.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.745+0000 7f4845600640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 0x7f4824069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:23.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.745+0000 7f4845600640 1 -- 172.21.15.98:0/2359400893 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5227+0+0 (secure 0 0 0) 0x7f4840085950 con 0x7f48481511c0 2024-05-27T00:15:23.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.745+0000 7f484d000640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 0x7f4824069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:23.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.749+0000 7f484d000640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 0x7f4824069d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48380b2000 tx=0x7f4838008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:23.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.749+0000 7f4845600640 1 -- 172.21.15.98:0/2359400893 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4848158010 con 0x7f48481511c0 2024-05-27T00:15:23.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:23.905+0000 7f484e400640 1 -- 172.21.15.98:0/2359400893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f4848075c40 con 0x7f48481511c0 2024-05-27T00:15:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.573+0000 7f4845600640 1 -- 172.21.15.98:0/2359400893 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) ==== 170+0+0 (secure 0 0 0) 0x7f4840055ef0 con 0x7f48481511c0 2024-05-27T00:15:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2024-05-27T00:15:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 -- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 msgr2=0x7f4824069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 0x7f4824069d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48380b2000 tx=0x7f4838008040 comp rx=0 tx=0).stop 2024-05-27T00:15:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 -- 172.21.15.98:0/2359400893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48481511c0 msgr2=0x7f48480c3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48481511c0 0x7f48480c3230 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f484000c730 tx=0x7f484000cbf0 comp rx=0 tx=0).stop 2024-05-27T00:15:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 -- 172.21.15.98:0/2359400893 shutdown_connections 2024-05-27T00:15:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48481511c0 0x7f48480c3230 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4848070f10 0x7f48481594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:24.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 0x7f4824069d80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 --2- 172.21.15.98:0/2359400893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f484806f140 0x7f4848155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 -- 172.21.15.98:0/2359400893 >> 172.21.15.98:0/2359400893 conn(0x7f48480cdf90 msgr2=0x7f484815cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 -- 172.21.15.98:0/2359400893 shutdown_connections 2024-05-27T00:15:24.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.577+0000 7f484e400640 1 -- 172.21.15.98:0/2359400893 wait complete. 2024-05-27T00:15:24.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-27T00:15:24.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e4400640 1 Processor -- start 2024-05-27T00:15:24.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e4400640 1 -- start start 2024-05-27T00:15:24.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e4400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96dc071a90 0x7f96dc15aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:24.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e4400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f96dc0b4dd0 0x7f96dc0cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:24.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e4400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96dc072dc0 con 0x7f96dc154de0 2024-05-27T00:15:24.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e4400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f96dc1551c0 con 0x7f96dc071a90 2024-05-27T00:15:24.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e4400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f96dc071f60 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e1000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96dc071a90 0x7f96dc15aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:24.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e1a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f96dc154de0 0x7f96dc0b46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43232/0 (socket says 172.21.15.98:43232) 2024-05-27T00:15:24.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e1a00640 1 -- 172.21.15.98:0/3433814002 learned_addr learned my addr 172.21.15.98:0/3433814002 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:24.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e2400640 1 --2- 172.21.15.98:0/3433814002 >> v2:172.21.15.204:3300/0 conn(0x7f96dc0b4dd0 0x7f96dc0cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:24.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96dbe00640 1 -- 172.21.15.98:0/3433814002 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4227214716 0 0) 0x7f96dc072dc0 con 0x7f96dc154de0 2024-05-27T00:15:24.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e1000640 1 -- 172.21.15.98:0/3433814002 >> v2:172.21.15.204:3300/0 conn(0x7f96dc0b4dd0 msgr2=0x7f96dc0cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:24.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e1000640 1 --2- 172.21.15.98:0/3433814002 >> v2:172.21.15.204:3300/0 conn(0x7f96dc0b4dd0 0x7f96dc0cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:24.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e1000640 1 -- 172.21.15.98:0/3433814002 >> v1:172.21.15.98:6789/0 conn(0x7f96dc154de0 legacy=0x7f96dc0b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:24.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.749+0000 7f96e1000640 1 -- 172.21.15.98:0/3433814002 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc072240 con 0x7f96dc071a90 2024-05-27T00:15:24.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e1000640 1 --2- 172.21.15.98:0/3433814002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc071a90 0x7f96dc15aa00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f96dc071e70 tx=0x7f96c801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6682403648aa9bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:24.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96dbe00640 1 -- 172.21.15.98:0/3433814002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c80b5070 con 0x7f96dc071a90 2024-05-27T00:15:24.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96dbe00640 1 -- 172.21.15.98:0/3433814002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96c80a8970 con 0x7f96dc071a90 2024-05-27T00:15:24.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96dbe00640 1 -- 172.21.15.98:0/3433814002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c80a8d40 con 0x7f96dc071a90 2024-05-27T00:15:24.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e4400640 1 -- 172.21.15.98:0/3433814002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc071a90 msgr2=0x7f96dc15aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:24.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e4400640 1 --2- 172.21.15.98:0/3433814002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc071a90 0x7f96dc15aa00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f96dc071e70 tx=0x7f96c801ec70 comp rx=0 tx=0).stop 2024-05-27T00:15:24.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e4400640 1 -- 172.21.15.98:0/3433814002 shutdown_connections 2024-05-27T00:15:24.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e4400640 1 --2- 172.21.15.98:0/3433814002 >> v2:172.21.15.204:3300/0 conn(0x7f96dc0b4dd0 0x7f96dc0cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:24.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e4400640 1 --2- 172.21.15.98:0/3433814002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc071a90 0x7f96dc15aa00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:24.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e4400640 1 -- 172.21.15.98:0/3433814002 >> 172.21.15.98:0/3433814002 conn(0x7f96dc0ccaf0 msgr2=0x7f96dc0ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:24.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e4400640 1 -- 172.21.15.98:0/3433814002 shutdown_connections 2024-05-27T00:15:24.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e4400640 1 -- 172.21.15.98:0/3433814002 wait complete. 2024-05-27T00:15:24.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e4400640 1 Processor -- start 2024-05-27T00:15:24.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.753+0000 7f96e4400640 1 -- start start 2024-05-27T00:15:24.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e4400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96dc071a90 0x7f96dc14f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:24.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e4400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc0b4dd0 0x7f96dc151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:24.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e4400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96dc154de0 0x7f96dc1f1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:24.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e4400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc072ae0 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e4400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc072800 con 0x7f96dc071a90 2024-05-27T00:15:24.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e4400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f96dc0cd900 con 0x7f96dc154de0 2024-05-27T00:15:24.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e1000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc0b4dd0 0x7f96dc151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:24.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e1000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc0b4dd0 0x7f96dc151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53364/0 (socket says 172.21.15.98:53364) 2024-05-27T00:15:24.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e1000640 1 -- 172.21.15.98:0/2682940055 learned_addr learned my addr 172.21.15.98:0/2682940055 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:24.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e1a00640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96dc071a90 0x7f96dc14f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:24.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e2400640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96dc154de0 0x7f96dc1f1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:24.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e1000640 1 -- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96dc154de0 msgr2=0x7f96dc1f1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:24.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e1000640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96dc154de0 0x7f96dc1f1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:24.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e1000640 1 -- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96dc071a90 msgr2=0x7f96dc14f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:24.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e1000640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96dc071a90 0x7f96dc14f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:24.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e1000640 1 -- 172.21.15.98:0/2682940055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc1f4160 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e1000640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc0b4dd0 0x7f96dc151ef0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f96c80b3040 tx=0x7f96c801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:24.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96d9600640 1 -- 172.21.15.98:0/2682940055 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c80bf070 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96d9600640 1 -- 172.21.15.98:0/2682940055 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96c80b07a0 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e4400640 1 -- 172.21.15.98:0/2682940055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96dc1f4400 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96d9600640 1 -- 172.21.15.98:0/2682940055 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c80b5040 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e4400640 1 -- 172.21.15.98:0/2682940055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96dc1f49a0 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.757+0000 7f96e4400640 1 -- 172.21.15.98:0/2682940055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96dc075cb0 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.761+0000 7f96d9600640 1 -- 172.21.15.98:0/2682940055 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f96c80c2070 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.761+0000 7f96d9600640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96bc0678f0 0x7f96bc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:24.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.761+0000 7f96d9600640 1 -- 172.21.15.98:0/2682940055 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5609+0+0 (secure 0 0 0) 0x7f96c8122580 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.761+0000 7f96e1a00640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96bc0678f0 0x7f96bc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:24.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.761+0000 7f96e1a00640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96bc0678f0 0x7f96bc069db0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f96d0000f70 tx=0x7f96d0007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:24.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.765+0000 7f96d9600640 1 -- 172.21.15.98:0/2682940055 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f96c80187d0 con 0x7f96dc0b4dd0 2024-05-27T00:15:24.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:24.953+0000 7f96e4400640 1 -- 172.21.15.98:0/2682940055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f96dc0cec20 con 0x7f96dc0b4dd0 2024-05-27T00:15:25.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.601+0000 7f96d9600640 1 -- 172.21.15.98:0/2682940055 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f96c80f3110 con 0x7f96dc0b4dd0 2024-05-27T00:15:25.603 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 2 and data pool 3 2024-05-27T00:15:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.601+0000 7f96e4400640 1 -- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96bc0678f0 msgr2=0x7f96bc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.601+0000 7f96e4400640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96bc0678f0 0x7f96bc069db0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f96d0000f70 tx=0x7f96d0007b60 comp rx=0 tx=0).stop 2024-05-27T00:15:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.601+0000 7f96e4400640 1 -- 172.21.15.98:0/2682940055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc0b4dd0 msgr2=0x7f96dc151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.601+0000 7f96e4400640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc0b4dd0 0x7f96dc151ef0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f96c80b3040 tx=0x7f96c801ba10 comp rx=0 tx=0).stop 2024-05-27T00:15:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.601+0000 7f96e4400640 1 -- 172.21.15.98:0/2682940055 shutdown_connections 2024-05-27T00:15:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.601+0000 7f96e4400640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96dc154de0 0x7f96dc1f1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.601+0000 7f96e4400640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96dc0b4dd0 0x7f96dc151ef0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.601+0000 7f96e4400640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96bc0678f0 0x7f96bc069db0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.601+0000 7f96e4400640 1 --2- 172.21.15.98:0/2682940055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96dc071a90 0x7f96dc14f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.605+0000 7f96e4400640 1 -- 172.21.15.98:0/2682940055 >> 172.21.15.98:0/2682940055 conn(0x7f96dc0ccaf0 msgr2=0x7f96dc0cb810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.605+0000 7f96e4400640 1 -- 172.21.15.98:0/2682940055 shutdown_connections 2024-05-27T00:15:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.605+0000 7f96e4400640 1 -- 172.21.15.98:0/2682940055 wait complete. 2024-05-27T00:15:25.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:25.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.765+0000 7f8d4f200640 1 Processor -- start 2024-05-27T00:15:25.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.765+0000 7f8d4f200640 1 -- start start 2024-05-27T00:15:25.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.765+0000 7f8d4f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5015b9b0 0x7f8d5015bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.765+0000 7f8d4f200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8d500b3e30 0x7f8d5015c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.765+0000 7f8d4f200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d5006e160 0x7f8d5006e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.765+0000 7f8d4f200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8d500c39c0 con 0x7f8d5006e160 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.765+0000 7f8d4f200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f8d5006eb20 con 0x7f8d500b3e30 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.765+0000 7f8d4f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8d500c3c10 con 0x7f8d5015b9b0 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4e800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d5006e160 0x7f8d5006e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4e800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8d5006e160 0x7f8d5006e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53376/0 (socket says 172.21.15.98:53376) 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4e800640 1 -- 172.21.15.98:0/1802008535 learned_addr learned my addr 172.21.15.98:0/1802008535 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d47e00640 1 --2- 172.21.15.98:0/1802008535 >> v2:172.21.15.204:3300/0 conn(0x7f8d500b3e30 0x7f8d5015c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4de00640 1 --2- 172.21.15.98:0/1802008535 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5015b9b0 0x7f8d5015bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4e800640 1 -- 172.21.15.98:0/1802008535 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5015b9b0 msgr2=0x7f8d5015bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:25.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4e800640 1 --2- 172.21.15.98:0/1802008535 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5015b9b0 0x7f8d5015bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4e800640 1 -- 172.21.15.98:0/1802008535 >> v2:172.21.15.204:3300/0 conn(0x7f8d500b3e30 msgr2=0x7f8d5015c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:25.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4e800640 1 --2- 172.21.15.98:0/1802008535 >> v2:172.21.15.204:3300/0 conn(0x7f8d500b3e30 0x7f8d5015c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4e800640 1 -- 172.21.15.98:0/1802008535 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d500c3ef0 con 0x7f8d5006e160 2024-05-27T00:15:25.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d47e00640 1 --2- 172.21.15.98:0/1802008535 >> v2:172.21.15.204:3300/0 conn(0x7f8d500b3e30 0x7f8d5015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4e800640 1 --2- 172.21.15.98:0/1802008535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d5006e160 0x7f8d5006e5e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8d500705c0 tx=0x7f8d4801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b30811e0e54c08 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:25.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4d400640 1 -- 172.21.15.98:0/1802008535 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d480a57b0 con 0x7f8d5006e160 2024-05-27T00:15:25.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4d400640 1 -- 172.21.15.98:0/1802008535 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d480a5950 con 0x7f8d5006e160 2024-05-27T00:15:25.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.769+0000 7f8d4d400640 1 -- 172.21.15.98:0/1802008535 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d480a5d20 con 0x7f8d5006e160 2024-05-27T00:15:25.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 -- 172.21.15.98:0/1802008535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d5006e160 msgr2=0x7f8d5006e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:25.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 --2- 172.21.15.98:0/1802008535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d5006e160 0x7f8d5006e5e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8d500705c0 tx=0x7f8d4801c930 comp rx=0 tx=0).stop 2024-05-27T00:15:25.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 -- 172.21.15.98:0/1802008535 shutdown_connections 2024-05-27T00:15:25.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 --2- 172.21.15.98:0/1802008535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d5006e160 0x7f8d5006e5e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 --2- 172.21.15.98:0/1802008535 >> v2:172.21.15.204:3300/0 conn(0x7f8d500b3e30 0x7f8d5015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 --2- 172.21.15.98:0/1802008535 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5015b9b0 0x7f8d5015bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 -- 172.21.15.98:0/1802008535 >> 172.21.15.98:0/1802008535 conn(0x7f8d50070a10 msgr2=0x7f8d50070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:25.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 -- 172.21.15.98:0/1802008535 shutdown_connections 2024-05-27T00:15:25.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 -- 172.21.15.98:0/1802008535 wait complete. 2024-05-27T00:15:25.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 Processor -- start 2024-05-27T00:15:25.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 -- start start 2024-05-27T00:15:25.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5006e160 0x7f8d500ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:25.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d500b3e30 0x7f8d5014e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:25.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d5015b9b0 0x7f8d501595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:25.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8d50070110 con 0x7f8d500b3e30 2024-05-27T00:15:25.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8d500c4ec0 con 0x7f8d5015b9b0 2024-05-27T00:15:25.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8d50071530 con 0x7f8d5006e160 2024-05-27T00:15:25.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d47e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d500b3e30 0x7f8d5014e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:25.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d47e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d500b3e30 0x7f8d5014e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53388/0 (socket says 172.21.15.98:53388) 2024-05-27T00:15:25.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d47e00640 1 -- 172.21.15.98:0/329086573 learned_addr learned my addr 172.21.15.98:0/329086573 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:25.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4e800640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d5015b9b0 0x7f8d501595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:25.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d4de00640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5006e160 0x7f8d500ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:25.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d47e00640 1 -- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5006e160 msgr2=0x7f8d500ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:25.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d47e00640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5006e160 0x7f8d500ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d47e00640 1 -- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d5015b9b0 msgr2=0x7f8d501595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d47e00640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d5015b9b0 0x7f8d501595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d47e00640 1 -- 172.21.15.98:0/329086573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d500681f0 con 0x7f8d500b3e30 2024-05-27T00:15:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.773+0000 7f8d47e00640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d500b3e30 0x7f8d5014e900 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d3800d830 tx=0x7f8d3800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.777+0000 7f8d46000640 1 -- 172.21.15.98:0/329086573 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d38018030 con 0x7f8d500b3e30 2024-05-27T00:15:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.777+0000 7f8d46000640 1 -- 172.21.15.98:0/329086573 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d3800be00 con 0x7f8d500b3e30 2024-05-27T00:15:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.777+0000 7f8d46000640 1 -- 172.21.15.98:0/329086573 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d380098b0 con 0x7f8d500b3e30 2024-05-27T00:15:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.777+0000 7f8d4f200640 1 -- 172.21.15.98:0/329086573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d501f3140 con 0x7f8d500b3e30 2024-05-27T00:15:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.777+0000 7f8d4f200640 1 -- 172.21.15.98:0/329086573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d501f3400 con 0x7f8d500b3e30 2024-05-27T00:15:25.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.777+0000 7f8d4f200640 1 -- 172.21.15.98:0/329086573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d50075c40 con 0x7f8d500b3e30 2024-05-27T00:15:25.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.777+0000 7f8d46000640 1 -- 172.21.15.98:0/329086573 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8d3801b070 con 0x7f8d500b3e30 2024-05-27T00:15:25.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.777+0000 7f8d46000640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d2c0678c0 0x7f8d2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:25.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.777+0000 7f8d46000640 1 -- 172.21.15.98:0/329086573 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8d38085e80 con 0x7f8d500b3e30 2024-05-27T00:15:25.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.777+0000 7f8d4de00640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d2c0678c0 0x7f8d2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:25.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.781+0000 7f8d4de00640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d2c0678c0 0x7f8d2c069d80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8d4000b9d0 tx=0x7f8d40008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:25.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.781+0000 7f8d46000640 1 -- 172.21.15.98:0/329086573 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8d38056920 con 0x7f8d500b3e30 2024-05-27T00:15:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.937+0000 7f8d4f200640 1 -- 172.21.15.98:0/329086573 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d50067da0 con 0x7f8d500b3e30 2024-05-27T00:15:25.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d46000640 1 -- 172.21.15.98:0/329086573 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16594 (secure 0 0 0) 0x7f8d38056220 con 0x7f8d500b3e30 2024-05-27T00:15:25.943 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:25.943 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:24.956025+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 -- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d2c0678c0 msgr2=0x7f8d2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d2c0678c0 0x7f8d2c069d80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8d4000b9d0 tx=0x7f8d40008040 comp rx=0 tx=0).stop 2024-05-27T00:15:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 -- 172.21.15.98:0/329086573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d500b3e30 msgr2=0x7f8d5014e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d500b3e30 0x7f8d5014e900 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d3800d830 tx=0x7f8d3800dcf0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 -- 172.21.15.98:0/329086573 shutdown_connections 2024-05-27T00:15:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d2c0678c0 0x7f8d2c069d80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d5015b9b0 0x7f8d501595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d500b3e30 0x7f8d5014e900 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 --2- 172.21.15.98:0/329086573 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8d5006e160 0x7f8d500ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 -- 172.21.15.98:0/329086573 >> 172.21.15.98:0/329086573 conn(0x7f8d50070a10 msgr2=0x7f8d50070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 -- 172.21.15.98:0/329086573 shutdown_connections 2024-05-27T00:15:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:25.941+0000 7f8d4f200640 1 -- 172.21.15.98:0/329086573 wait complete. 2024-05-27T00:15:25.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:26.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd087600640 1 Processor -- start 2024-05-27T00:15:26.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd087600640 1 -- start start 2024-05-27T00:15:26.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd087600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd080154a00 0x7fd0800b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd087600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd080154de0 0x7fd08015b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd087600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd08015b920 con 0x7fd0800d0070 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd087600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd0800b3f20 con 0x7fd080154de0 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd087600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd0800b4200 con 0x7fd080154a00 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd07fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd080154de0 0x7fd08015b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd07fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd080154de0 0x7fd08015b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53402/0 (socket says 172.21.15.98:53402) 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd07fe00640 1 -- 172.21.15.98:0/1202764359 learned_addr learned my addr 172.21.15.98:0/1202764359 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd084c00640 1 --2- 172.21.15.98:0/1202764359 >> v2:172.21.15.204:3300/0 conn(0x7fd080154a00 0x7fd0800b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd07fe00640 1 -- 172.21.15.98:0/1202764359 >> v2:172.21.15.204:3300/0 conn(0x7fd080154a00 msgr2=0x7fd0800b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd07fe00640 1 --2- 172.21.15.98:0/1202764359 >> v2:172.21.15.204:3300/0 conn(0x7fd080154a00 0x7fd0800b2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd07fe00640 1 -- 172.21.15.98:0/1202764359 >> v1:172.21.15.204:6789/0 conn(0x7fd0800d0070 legacy=0x7fd0800cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd07fe00640 1 -- 172.21.15.98:0/1202764359 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0800b44e0 con 0x7fd080154de0 2024-05-27T00:15:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.097+0000 7fd084c00640 1 --2- 172.21.15.98:0/1202764359 >> v2:172.21.15.204:3300/0 conn(0x7fd080154a00 0x7fd0800b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd07fe00640 1 --2- 172.21.15.98:0/1202764359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd080154de0 0x7fd08015b3e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd08006d3e0 tx=0x7fd06c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0d1add26898a57 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd07f400640 1 -- 172.21.15.98:0/1202764359 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0b5070 con 0x7fd080154de0 2024-05-27T00:15:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd07f400640 1 -- 172.21.15.98:0/1202764359 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd06c0a8a10 con 0x7fd080154de0 2024-05-27T00:15:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd07f400640 1 -- 172.21.15.98:0/1202764359 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0a8de0 con 0x7fd080154de0 2024-05-27T00:15:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd087600640 1 -- 172.21.15.98:0/1202764359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd080154de0 msgr2=0x7fd08015b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd087600640 1 --2- 172.21.15.98:0/1202764359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd080154de0 0x7fd08015b3e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd08006d3e0 tx=0x7fd06c01ed10 comp rx=0 tx=0).stop 2024-05-27T00:15:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd087600640 1 -- 172.21.15.98:0/1202764359 shutdown_connections 2024-05-27T00:15:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd087600640 1 --2- 172.21.15.98:0/1202764359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd080154de0 0x7fd08015b3e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd087600640 1 --2- 172.21.15.98:0/1202764359 >> v2:172.21.15.204:3300/0 conn(0x7fd080154a00 0x7fd0800b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd087600640 1 -- 172.21.15.98:0/1202764359 >> 172.21.15.98:0/1202764359 conn(0x7fd080067530 msgr2=0x7fd08006c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd087600640 1 -- 172.21.15.98:0/1202764359 shutdown_connections 2024-05-27T00:15:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.101+0000 7fd087600640 1 -- 172.21.15.98:0/1202764359 wait complete. 2024-05-27T00:15:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 Processor -- start 2024-05-27T00:15:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 -- start start 2024-05-27T00:15:26.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0800d0070 0x7fd0800c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd080154a00 0x7fd0800c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd080154de0 0x7fd0800c6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0800b3620 con 0x7fd0800d0070 2024-05-27T00:15:26.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd0800b4aa0 con 0x7fd080154de0 2024-05-27T00:15:26.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd0800b37a0 con 0x7fd080154a00 2024-05-27T00:15:26.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd084c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0800d0070 0x7fd0800c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd084c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0800d0070 0x7fd0800c8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53410/0 (socket says 172.21.15.98:53410) 2024-05-27T00:15:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd084c00640 1 -- 172.21.15.98:0/2542681681 learned_addr learned my addr 172.21.15.98:0/2542681681 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd07fe00640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd080154a00 0x7fd0800c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd085600640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd080154de0 0x7fd0800c6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd084c00640 1 -- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd080154a00 msgr2=0x7fd0800c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd084c00640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd080154a00 0x7fd0800c64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd084c00640 1 -- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd080154de0 msgr2=0x7fd0800c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd084c00640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd080154de0 0x7fd0800c6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd084c00640 1 -- 172.21.15.98:0/2542681681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0800c7670 con 0x7fd0800d0070 2024-05-27T00:15:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd085600640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd080154de0 0x7fd0800c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd084c00640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0800d0070 0x7fd0800c8460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd070007fa0 tx=0x7fd07000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd07cc00640 1 -- 172.21.15.98:0/2542681681 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07001a070 con 0x7fd0800d0070 2024-05-27T00:15:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd07cc00640 1 -- 172.21.15.98:0/2542681681 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd070003ba0 con 0x7fd0800d0070 2024-05-27T00:15:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 -- 172.21.15.98:0/2542681681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd08015e950 con 0x7fd0800d0070 2024-05-27T00:15:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd07cc00640 1 -- 172.21.15.98:0/2542681681 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd070004030 con 0x7fd0800d0070 2024-05-27T00:15:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 -- 172.21.15.98:0/2542681681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0800c7910 con 0x7fd0800d0070 2024-05-27T00:15:26.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.105+0000 7fd087600640 1 -- 172.21.15.98:0/2542681681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd080075cb0 con 0x7fd0800d0070 2024-05-27T00:15:26.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.109+0000 7fd07cc00640 1 -- 172.21.15.98:0/2542681681 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd07001d070 con 0x7fd0800d0070 2024-05-27T00:15:26.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.109+0000 7fd07cc00640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0600678c0 0x7fd060069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.109+0000 7fd07cc00640 1 -- 172.21.15.98:0/2542681681 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd070085f20 con 0x7fd0800d0070 2024-05-27T00:15:26.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.109+0000 7fd07fe00640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0600678c0 0x7fd060069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.109+0000 7fd07fe00640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0600678c0 0x7fd060069d80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0b2000 tx=0x7fd06c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.113+0000 7fd07cc00640 1 -- 172.21.15.98:0/2542681681 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd070010070 con 0x7fd0800d0070 2024-05-27T00:15:26.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.269+0000 7fd087600640 1 -- 172.21.15.98:0/2542681681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0800b5be0 con 0x7fd0800d0070 2024-05-27T00:15:26.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd07cc00640 1 -- 172.21.15.98:0/2542681681 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16594 (secure 0 0 0) 0x7fd070056400 con 0x7fd0800d0070 2024-05-27T00:15:26.274 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:26.278 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:24.956025+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:26.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 -- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0600678c0 msgr2=0x7fd060069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0600678c0 0x7fd060069d80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0b2000 tx=0x7fd06c0aa070 comp rx=0 tx=0).stop 2024-05-27T00:15:26.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 -- 172.21.15.98:0/2542681681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0800d0070 msgr2=0x7fd0800c8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0800d0070 0x7fd0800c8460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd070007fa0 tx=0x7fd07000d560 comp rx=0 tx=0).stop 2024-05-27T00:15:26.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 -- 172.21.15.98:0/2542681681 shutdown_connections 2024-05-27T00:15:26.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0600678c0 0x7fd060069d80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd080154de0 0x7fd0800c6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd080154a00 0x7fd0800c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 --2- 172.21.15.98:0/2542681681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0800d0070 0x7fd0800c8460 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 -- 172.21.15.98:0/2542681681 >> 172.21.15.98:0/2542681681 conn(0x7fd080067530 msgr2=0x7fd0800ce940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:26.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 -- 172.21.15.98:0/2542681681 shutdown_connections 2024-05-27T00:15:26.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.273+0000 7fd087600640 1 -- 172.21.15.98:0/2542681681 wait complete. 2024-05-27T00:15:26.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-27T00:15:26.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a66600640 1 Processor -- start 2024-05-27T00:15:26.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a66600640 1 -- start start 2024-05-27T00:15:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a66600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7a6015a880 0x7f7a600c9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a66600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a600c7be0 con 0x7f7a600c70b0 2024-05-27T00:15:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a66600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a6015ade0 con 0x7f7a60154ad0 2024-05-27T00:15:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a66600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7a6015de70 con 0x7f7a6015a880 2024-05-27T00:15:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a5ea00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7a6015a880 0x7f7a600c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a5fe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7a600c70b0 0x7f7a60165200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43238/0 (socket says 172.21.15.98:43238) 2024-05-27T00:15:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a5fe00640 1 -- 172.21.15.98:0/660215256 learned_addr learned my addr 172.21.15.98:0/660215256 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a5ea00640 1 -- 172.21.15.98:0/660215256 >> v1:172.21.15.204:6789/0 conn(0x7f7a60154ad0 legacy=0x7f7a6015e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a5ea00640 1 -- 172.21.15.98:0/660215256 >> v1:172.21.15.98:6789/0 conn(0x7f7a600c70b0 legacy=0x7f7a60165200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a5ea00640 1 -- 172.21.15.98:0/660215256 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a6015b0c0 con 0x7f7a6015a880 2024-05-27T00:15:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.433+0000 7f7a5e000640 1 -- 172.21.15.98:0/660215256 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1550134737 0 0) 0x7f7a600c7be0 con 0x7f7a600c70b0 2024-05-27T00:15:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a5ea00640 1 --2- 172.21.15.98:0/660215256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 0x7f7a600c9140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7a500097c0 tx=0x7f7a5001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b150a6c186cf37c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a5e000640 1 -- 172.21.15.98:0/660215256 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a500b0070 con 0x7f7a6015a880 2024-05-27T00:15:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a5e000640 1 -- 172.21.15.98:0/660215256 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a500ab600 con 0x7f7a6015a880 2024-05-27T00:15:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a5e000640 1 -- 172.21.15.98:0/660215256 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a500ab9d0 con 0x7f7a6015a880 2024-05-27T00:15:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a66600640 1 -- 172.21.15.98:0/660215256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 msgr2=0x7f7a600c9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a66600640 1 --2- 172.21.15.98:0/660215256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 0x7f7a600c9140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7a500097c0 tx=0x7f7a5001c4a0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a66600640 1 -- 172.21.15.98:0/660215256 shutdown_connections 2024-05-27T00:15:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a66600640 1 --2- 172.21.15.98:0/660215256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 0x7f7a600c9140 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a66600640 1 -- 172.21.15.98:0/660215256 >> 172.21.15.98:0/660215256 conn(0x7f7a6006dca0 msgr2=0x7f7a6006b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a66600640 1 -- 172.21.15.98:0/660215256 shutdown_connections 2024-05-27T00:15:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.437+0000 7f7a66600640 1 -- 172.21.15.98:0/660215256 wait complete. 2024-05-27T00:15:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a66600640 1 Processor -- start 2024-05-27T00:15:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a66600640 1 -- start start 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a66600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a600c70b0 0x7f7a600c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a66600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a60154ad0 0x7f7a601462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a66600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 0x7f7a601590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a66600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a60154eb0 con 0x7f7a6015a880 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a66600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7a60062d00 con 0x7f7a60154ad0 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a66600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7a60169140 con 0x7f7a600c70b0 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 0x7f7a601590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 0x7f7a601590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53434/0 (socket says 172.21.15.98:53434) 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5fe00640 1 -- 172.21.15.98:0/4115218196 learned_addr learned my addr 172.21.15.98:0/4115218196 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5ea00640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a60154ad0 0x7f7a601462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5f400640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a600c70b0 0x7f7a600c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5fe00640 1 -- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a600c70b0 msgr2=0x7f7a600c52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5fe00640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a600c70b0 0x7f7a600c52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5fe00640 1 -- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a60154ad0 msgr2=0x7f7a601462a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5fe00640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a60154ad0 0x7f7a601462a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5fe00640 1 -- 172.21.15.98:0/4115218196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a601457e0 con 0x7f7a6015a880 2024-05-27T00:15:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a5fe00640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 0x7f7a601590d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7a5800c7d0 tx=0x7f7a5800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a57400640 1 -- 172.21.15.98:0/4115218196 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a58018070 con 0x7f7a6015a880 2024-05-27T00:15:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a57400640 1 -- 172.21.15.98:0/4115218196 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a5800a640 con 0x7f7a6015a880 2024-05-27T00:15:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a57400640 1 -- 172.21.15.98:0/4115218196 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5800aa70 con 0x7f7a6015a880 2024-05-27T00:15:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a66600640 1 -- 172.21.15.98:0/4115218196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a60147020 con 0x7f7a6015a880 2024-05-27T00:15:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.441+0000 7f7a66600640 1 -- 172.21.15.98:0/4115218196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a601475e0 con 0x7f7a6015a880 2024-05-27T00:15:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.445+0000 7f7a66600640 1 -- 172.21.15.98:0/4115218196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a60075cb0 con 0x7f7a6015a880 2024-05-27T00:15:26.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.445+0000 7f7a57400640 1 -- 172.21.15.98:0/4115218196 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7a5801b070 con 0x7f7a6015a880 2024-05-27T00:15:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.445+0000 7f7a57400640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7a3c0678f0 0x7f7a3c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.445+0000 7f7a57400640 1 -- 172.21.15.98:0/4115218196 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7a580851d0 con 0x7f7a6015a880 2024-05-27T00:15:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.445+0000 7f7a5f400640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7a3c0678f0 0x7f7a3c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.445+0000 7f7a5f400640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7a3c0678f0 0x7f7a3c069db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a48000f70 tx=0x7f7a48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.449+0000 7f7a57400640 1 -- 172.21.15.98:0/4115218196 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7a580556b0 con 0x7f7a6015a880 2024-05-27T00:15:26.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:26.641+0000 7f7a66600640 1 -- 172.21.15.98:0/4115218196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f7a600c7790 con 0x7f7a6015a880 2024-05-27T00:15:27.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.609+0000 7f7a57400640 1 -- 172.21.15.98:0/4115218196 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7f7a580599b0 con 0x7f7a6015a880 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 -- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7a3c0678f0 msgr2=0x7f7a3c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7a3c0678f0 0x7f7a3c069db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7a48000f70 tx=0x7f7a48009040 comp rx=0 tx=0).stop 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 -- 172.21.15.98:0/4115218196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 msgr2=0x7f7a601590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 0x7f7a601590d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7a5800c7d0 tx=0x7f7a5800cc90 comp rx=0 tx=0).stop 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 -- 172.21.15.98:0/4115218196 shutdown_connections 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a6015a880 0x7f7a601590d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7a3c0678f0 0x7f7a3c069db0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a60154ad0 0x7f7a601462a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 --2- 172.21.15.98:0/4115218196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a600c70b0 0x7f7a600c52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 -- 172.21.15.98:0/4115218196 >> 172.21.15.98:0/4115218196 conn(0x7f7a6006dca0 msgr2=0x7f7a6006ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:27.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 -- 172.21.15.98:0/4115218196 shutdown_connections 2024-05-27T00:15:27.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.613+0000 7f7a66600640 1 -- 172.21.15.98:0/4115218196 wait complete. 2024-05-27T00:15:27.636 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-05-27T00:15:27.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-27T00:15:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.773+0000 7f5433000640 1 Processor -- start 2024-05-27T00:15:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.773+0000 7f5433000640 1 -- start start 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5433000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f542c0b1c20 0x7f542c0c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5433000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f542c0c8460 0x7f542c158640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5433000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542c15bc40 con 0x7f542c152990 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5433000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f542c15c850 con 0x7f542c0c8460 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5433000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f542c15bf20 con 0x7f542c0b1c20 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f542be00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f542c152990 0x7f542c06e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43240/0 (socket says 172.21.15.98:43240) 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f542be00640 1 -- 172.21.15.98:0/3825209868 learned_addr learned my addr 172.21.15.98:0/3825209868 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5431000640 1 --2- 172.21.15.98:0/3825209868 >> v2:172.21.15.98:3300/0 conn(0x7f542c0c8460 0x7f542c158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5423400640 1 --2- 172.21.15.98:0/3825209868 >> v2:172.21.15.204:3300/0 conn(0x7f542c0b1c20 0x7f542c0c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5431000640 1 -- 172.21.15.98:0/3825209868 >> v2:172.21.15.204:3300/0 conn(0x7f542c0b1c20 msgr2=0x7f542c0c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5431000640 1 --2- 172.21.15.98:0/3825209868 >> v2:172.21.15.204:3300/0 conn(0x7f542c0b1c20 0x7f542c0c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:27.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5431000640 1 -- 172.21.15.98:0/3825209868 >> v1:172.21.15.98:6789/0 conn(0x7f542c152990 legacy=0x7f542c06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:27.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5431000640 1 -- 172.21.15.98:0/3825209868 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c15c200 con 0x7f542c0c8460 2024-05-27T00:15:27.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f542b400640 1 -- 172.21.15.98:0/3825209868 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2450700603 0 0) 0x7f542c15bc40 con 0x7f542c152990 2024-05-27T00:15:27.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.777+0000 7f5431000640 1 --2- 172.21.15.98:0/3825209868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 0x7f542c158640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f542c06e0b0 tx=0x7f542401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9a002d469c08c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f542b400640 1 -- 172.21.15.98:0/3825209868 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54240b2070 con 0x7f542c0c8460 2024-05-27T00:15:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f542b400640 1 -- 172.21.15.98:0/3825209868 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54240a57f0 con 0x7f542c0c8460 2024-05-27T00:15:27.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f542b400640 1 -- 172.21.15.98:0/3825209868 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54240a5bc0 con 0x7f542c0c8460 2024-05-27T00:15:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 -- 172.21.15.98:0/3825209868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 msgr2=0x7f542c158640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 --2- 172.21.15.98:0/3825209868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 0x7f542c158640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f542c06e0b0 tx=0x7f542401c930 comp rx=0 tx=0).stop 2024-05-27T00:15:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 -- 172.21.15.98:0/3825209868 shutdown_connections 2024-05-27T00:15:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 --2- 172.21.15.98:0/3825209868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 0x7f542c158640 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 --2- 172.21.15.98:0/3825209868 >> v2:172.21.15.204:3300/0 conn(0x7f542c0b1c20 0x7f542c0c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 -- 172.21.15.98:0/3825209868 >> 172.21.15.98:0/3825209868 conn(0x7f542c0cbf00 msgr2=0x7f542c0cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 -- 172.21.15.98:0/3825209868 shutdown_connections 2024-05-27T00:15:27.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 -- 172.21.15.98:0/3825209868 wait complete. 2024-05-27T00:15:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 Processor -- start 2024-05-27T00:15:27.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 -- start start 2024-05-27T00:15:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f542c0b1c20 0x7f542c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 0x7f542c0c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:27.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f542c152990 0x7f542c0c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f542c1657d0 con 0x7f542c0c8460 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f542c165650 con 0x7f542c152990 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5433000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f542c165950 con 0x7f542c0b1c20 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5423400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 0x7f542c0c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5423400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 0x7f542c0c4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53448/0 (socket says 172.21.15.98:53448) 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5423400640 1 -- 172.21.15.98:0/1469040672 learned_addr learned my addr 172.21.15.98:0/1469040672 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5431000640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f542c152990 0x7f542c0c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f542be00640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f542c0b1c20 0x7f542c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5423400640 1 -- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f542c0b1c20 msgr2=0x7f542c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5423400640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f542c0b1c20 0x7f542c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5423400640 1 -- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f542c152990 msgr2=0x7f542c0c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5423400640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f542c152990 0x7f542c0c4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:27.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.781+0000 7f5423400640 1 -- 172.21.15.98:0/1469040672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c15a4e0 con 0x7f542c0c8460 2024-05-27T00:15:27.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f5423400640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 0x7f542c0c4900 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f541400d800 tx=0x7f541400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:27.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f5428c00640 1 -- 172.21.15.98:0/1469040672 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541400b660 con 0x7f542c0c8460 2024-05-27T00:15:27.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f5428c00640 1 -- 172.21.15.98:0/1469040672 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5414019070 con 0x7f542c0c8460 2024-05-27T00:15:27.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f5433000640 1 -- 172.21.15.98:0/1469040672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542c0c3b40 con 0x7f542c0c8460 2024-05-27T00:15:27.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f5428c00640 1 -- 172.21.15.98:0/1469040672 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541400ba00 con 0x7f542c0c8460 2024-05-27T00:15:27.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f5433000640 1 -- 172.21.15.98:0/1469040672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f542c0c4100 con 0x7f542c0c8460 2024-05-27T00:15:27.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f5433000640 1 -- 172.21.15.98:0/1469040672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f542c075cb0 con 0x7f542c0c8460 2024-05-27T00:15:27.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f5428c00640 1 -- 172.21.15.98:0/1469040672 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f541400bba0 con 0x7f542c0c8460 2024-05-27T00:15:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f5428c00640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f54080678c0 0x7f5408069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:27.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f5428c00640 1 -- 172.21.15.98:0/1469040672 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5414086180 con 0x7f542c0c8460 2024-05-27T00:15:27.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.785+0000 7f542be00640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f54080678c0 0x7f5408069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:27.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.789+0000 7f542be00640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f54080678c0 0x7f5408069d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f541c000f70 tx=0x7f541c007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:27.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.789+0000 7f5428c00640 1 -- 172.21.15.98:0/1469040672 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f5414056c20 con 0x7f542c0c8460 2024-05-27T00:15:27.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:27.977+0000 7f5433000640 1 -- 172.21.15.98:0/1469040672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f542c161480 con 0x7f542c0c8460 2024-05-27T00:15:28.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5428c00640 1 -- 172.21.15.98:0/1469040672 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) ==== 122+0+0 (secure 0 0 0) 0x7f5414056520 con 0x7f542c0c8460 2024-05-27T00:15:28.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 -- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f54080678c0 msgr2=0x7f5408069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f54080678c0 0x7f5408069d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f541c000f70 tx=0x7f541c007c10 comp rx=0 tx=0).stop 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 -- 172.21.15.98:0/1469040672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 msgr2=0x7f542c0c4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 0x7f542c0c4900 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f541400d800 tx=0x7f541400dcc0 comp rx=0 tx=0).stop 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 -- 172.21.15.98:0/1469040672 shutdown_connections 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f54080678c0 0x7f5408069d80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f542c152990 0x7f542c0c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f542c0c8460 0x7f542c0c4900 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 --2- 172.21.15.98:0/1469040672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f542c0b1c20 0x7f542c15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 -- 172.21.15.98:0/1469040672 >> 172.21.15.98:0/1469040672 conn(0x7f542c0cbf00 msgr2=0x7f542c1515e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 -- 172.21.15.98:0/1469040672 shutdown_connections 2024-05-27T00:15:28.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.613+0000 7f5433000640 1 -- 172.21.15.98:0/1469040672 wait complete. 2024-05-27T00:15:28.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-05-27T00:15:28.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.773+0000 7f87f8600640 1 Processor -- start 2024-05-27T00:15:28.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.773+0000 7f87f8600640 1 -- start start 2024-05-27T00:15:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.773+0000 7f87f8600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 0x7f87f00c1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.773+0000 7f87f8600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f0162180 con 0x7f87f00c7b20 2024-05-27T00:15:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.773+0000 7f87f8600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f015f490 con 0x7f87f0162c20 2024-05-27T00:15:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.773+0000 7f87f8600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f87f015f770 con 0x7f87f0159060 2024-05-27T00:15:28.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.773+0000 7f87f5200640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f87f00c7b20 0x7f87f015eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43248/0 (socket says 172.21.15.98:43248) 2024-05-27T00:15:28.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.773+0000 7f87f5200640 1 -- 172.21.15.98:0/705527208 learned_addr learned my addr 172.21.15.98:0/705527208 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:28.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.773+0000 7f87f5c00640 1 --2- 172.21.15.98:0/705527208 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 0x7f87f00c1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:28.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.777+0000 7f87efe00640 1 -- 172.21.15.98:0/705527208 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 10008987 0 0) 0x7f87f0162180 con 0x7f87f00c7b20 2024-05-27T00:15:28.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.777+0000 7f87efe00640 1 -- 172.21.15.98:0/705527208 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87d8003530 con 0x7f87f00c7b20 2024-05-27T00:15:28.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.777+0000 7f87efe00640 1 -- 172.21.15.98:0/705527208 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3140811223 0 0) 0x7f87f015f490 con 0x7f87f0162c20 2024-05-27T00:15:28.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.777+0000 7f87efe00640 1 -- 172.21.15.98:0/705527208 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f0162180 con 0x7f87f0162c20 2024-05-27T00:15:28.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.777+0000 7f87efe00640 1 -- 172.21.15.98:0/705527208 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2313331590 0 0) 0x7f87d8003530 con 0x7f87f00c7b20 2024-05-27T00:15:28.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.777+0000 7f87f5c00640 1 -- 172.21.15.98:0/705527208 >> v1:172.21.15.204:6789/0 conn(0x7f87f0162c20 legacy=0x7f87f00c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:28.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.777+0000 7f87f5c00640 1 -- 172.21.15.98:0/705527208 >> v1:172.21.15.98:6789/0 conn(0x7f87f00c7b20 legacy=0x7f87f015eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:28.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.777+0000 7f87f5c00640 1 -- 172.21.15.98:0/705527208 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f0162420 con 0x7f87f0159060 2024-05-27T00:15:28.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.777+0000 7f87f5c00640 1 --2- 172.21.15.98:0/705527208 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 0x7f87f00c1510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87e40097f0 tx=0x7f87e401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d363334addef9e00 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:28.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.777+0000 7f87efe00640 1 -- 172.21.15.98:0/705527208 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e40b0070 con 0x7f87f0159060 2024-05-27T00:15:28.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87efe00640 1 -- 172.21.15.98:0/705527208 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87e40ab400 con 0x7f87f0159060 2024-05-27T00:15:28.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87efe00640 1 -- 172.21.15.98:0/705527208 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc001680 con 0x7f87f0159060 2024-05-27T00:15:28.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 -- 172.21.15.98:0/705527208 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 msgr2=0x7f87f00c1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:28.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 --2- 172.21.15.98:0/705527208 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 0x7f87f00c1510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87e40097f0 tx=0x7f87e401c400 comp rx=0 tx=0).stop 2024-05-27T00:15:28.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 -- 172.21.15.98:0/705527208 shutdown_connections 2024-05-27T00:15:28.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 --2- 172.21.15.98:0/705527208 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 0x7f87f00c1510 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:28.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 -- 172.21.15.98:0/705527208 >> 172.21.15.98:0/705527208 conn(0x7f87f00cbf00 msgr2=0x7f87f00cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:28.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 -- 172.21.15.98:0/705527208 shutdown_connections 2024-05-27T00:15:28.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 -- 172.21.15.98:0/705527208 wait complete. 2024-05-27T00:15:28.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 Processor -- start 2024-05-27T00:15:28.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 -- start start 2024-05-27T00:15:28.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c7b20 0x7f87f0153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:28.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 0x7f87f0153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87f0162c20 0x7f87f01fb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87f00c7670 con 0x7f87f00c7b20 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f87f00c74f0 con 0x7f87f0162c20 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f8600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f87f016cc60 con 0x7f87f0159060 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f5c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c7b20 0x7f87f0153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f5c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c7b20 0x7f87f0153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53460/0 (socket says 172.21.15.98:53460) 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f5c00640 1 -- 172.21.15.98:0/3307501957 learned_addr learned my addr 172.21.15.98:0/3307501957 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f5200640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 0x7f87f0153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f6600640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87f0162c20 0x7f87f01fb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f5c00640 1 -- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 msgr2=0x7f87f0153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f5c00640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 0x7f87f0153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f5c00640 1 -- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87f0162c20 msgr2=0x7f87f01fb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:28.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f5c00640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87f0162c20 0x7f87f01fb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f5c00640 1 -- 172.21.15.98:0/3307501957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f0161e10 con 0x7f87f00c7b20 2024-05-27T00:15:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f5200640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 0x7f87f0153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.781+0000 7f87f6600640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87f0162c20 0x7f87f01fb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87f5c00640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c7b20 0x7f87f0153970 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f87e40ae040 tx=0x7f87e40191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87ed600640 1 -- 172.21.15.98:0/3307501957 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e40b9020 con 0x7f87f00c7b20 2024-05-27T00:15:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87ed600640 1 -- 172.21.15.98:0/3307501957 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87e4003820 con 0x7f87f00c7b20 2024-05-27T00:15:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87f8600640 1 -- 172.21.15.98:0/3307501957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f0155110 con 0x7f87f00c7b20 2024-05-27T00:15:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87ed600640 1 -- 172.21.15.98:0/3307501957 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e40b0040 con 0x7f87f00c7b20 2024-05-27T00:15:28.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87f8600640 1 -- 172.21.15.98:0/3307501957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f0153170 con 0x7f87f00c7b20 2024-05-27T00:15:28.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87f8600640 1 -- 172.21.15.98:0/3307501957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87f0075cb0 con 0x7f87f00c7b20 2024-05-27T00:15:28.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87ed600640 1 -- 172.21.15.98:0/3307501957 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f87e40bc020 con 0x7f87f00c7b20 2024-05-27T00:15:28.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87ed600640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87d00678c0 0x7f87d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:28.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87ed600640 1 -- 172.21.15.98:0/3307501957 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f87e411fb50 con 0x7f87f00c7b20 2024-05-27T00:15:28.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.785+0000 7f87f5200640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87d00678c0 0x7f87d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:28.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.789+0000 7f87f5200640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87d00678c0 0x7f87d0069d80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f87f00ca310 tx=0x7f87dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:28.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.789+0000 7f87ed600640 1 -- 172.21.15.98:0/3307501957 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f87e40c3030 con 0x7f87f00c7b20 2024-05-27T00:15:28.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:28.977+0000 7f87f8600640 1 -- 172.21.15.98:0/3307501957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f87f00cc470 con 0x7f87f00c7b20 2024-05-27T00:15:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.617+0000 7f87ed600640 1 -- 172.21.15.98:0/3307501957 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v7) ==== 115+0+0 (secure 0 0 0) 0x7f87e4015ff0 con 0x7f87f00c7b20 2024-05-27T00:15:29.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 -- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87d00678c0 msgr2=0x7f87d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:29.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87d00678c0 0x7f87d0069d80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f87f00ca310 tx=0x7f87dc009040 comp rx=0 tx=0).stop 2024-05-27T00:15:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 -- 172.21.15.98:0/3307501957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c7b20 msgr2=0x7f87f0153970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c7b20 0x7f87f0153970 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f87e40ae040 tx=0x7f87e40191a0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 -- 172.21.15.98:0/3307501957 shutdown_connections 2024-05-27T00:15:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87d00678c0 0x7f87d0069d80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87f0162c20 0x7f87f01fb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87f0159060 0x7f87f0153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 --2- 172.21.15.98:0/3307501957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f00c7b20 0x7f87f0153970 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 -- 172.21.15.98:0/3307501957 >> 172.21.15.98:0/3307501957 conn(0x7f87f00cbf00 msgr2=0x7f87f0072900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 -- 172.21.15.98:0/3307501957 shutdown_connections 2024-05-27T00:15:29.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.621+0000 7f87f8600640 1 -- 172.21.15.98:0/3307501957 wait complete. 2024-05-27T00:15:29.646 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-27T00:15:29.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:15:29.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3fec00640 1 Processor -- start 2024-05-27T00:15:29.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3fec00640 1 -- start start 2024-05-27T00:15:29.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3fec00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3f8070fe0 0x7fc3f8154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:29.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3fec00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3f815a950 con 0x7fc3f8070c00 2024-05-27T00:15:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3fec00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3f815e060 con 0x7fc3f80cb570 2024-05-27T00:15:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3fec00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc3f815e340 con 0x7fc3f8070fe0 2024-05-27T00:15:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3ef400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3f8070fe0 0x7fc3f8154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3ef400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3f8070fe0 0x7fc3f8154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53470/0 (socket says 172.21.15.98:53470) 2024-05-27T00:15:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3ef400640 1 -- 172.21.15.98:0/610193444 learned_addr learned my addr 172.21.15.98:0/610193444 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3f7e00640 1 --1- 172.21.15.98:0/610193444 >> v1:172.21.15.98:6789/0 conn(0x7fc3f8070c00 0x7fc3f80c8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43256/0 (socket says 172.21.15.98:43256) 2024-05-27T00:15:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3ef400640 1 -- 172.21.15.98:0/610193444 >> v1:172.21.15.204:6789/0 conn(0x7fc3f80cb570 legacy=0x7fc3f815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3ef400640 1 -- 172.21.15.98:0/610193444 >> v1:172.21.15.98:6789/0 conn(0x7fc3f8070c00 legacy=0x7fc3f80c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3f7400640 1 -- 172.21.15.98:0/610193444 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728578772 0 0) 0x7fc3f815a950 con 0x7fc3f8070c00 2024-05-27T00:15:29.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.781+0000 7fc3ef400640 1 -- 172.21.15.98:0/610193444 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f815e620 con 0x7fc3f8070fe0 2024-05-27T00:15:29.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3ef400640 1 --2- 172.21.15.98:0/610193444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070fe0 0x7fc3f8154780 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009870 tx=0x7fc3e001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675b6f7faa7e2b9e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3f7400640 1 -- 172.21.15.98:0/610193444 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00b5070 con 0x7fc3f8070fe0 2024-05-27T00:15:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3f7400640 1 -- 172.21.15.98:0/610193444 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e00a8a50 con 0x7fc3f8070fe0 2024-05-27T00:15:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3f7400640 1 -- 172.21.15.98:0/610193444 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00a8e20 con 0x7fc3f8070fe0 2024-05-27T00:15:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3fec00640 1 -- 172.21.15.98:0/610193444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070fe0 msgr2=0x7fc3f8154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:29.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3fec00640 1 --2- 172.21.15.98:0/610193444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070fe0 0x7fc3f8154780 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009870 tx=0x7fc3e001ed50 comp rx=0 tx=0).stop 2024-05-27T00:15:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3fec00640 1 -- 172.21.15.98:0/610193444 shutdown_connections 2024-05-27T00:15:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3fec00640 1 --2- 172.21.15.98:0/610193444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070fe0 0x7fc3f8154780 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3fec00640 1 -- 172.21.15.98:0/610193444 >> 172.21.15.98:0/610193444 conn(0x7fc3f8072f90 msgr2=0x7fc3f80733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3fec00640 1 -- 172.21.15.98:0/610193444 shutdown_connections 2024-05-27T00:15:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3fec00640 1 -- 172.21.15.98:0/610193444 wait complete. 2024-05-27T00:15:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.785+0000 7fc3fec00640 1 Processor -- start 2024-05-27T00:15:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fec00640 1 -- start start 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fec00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070c00 0x7fc3f814df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fec00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3f8070fe0 0x7fc3f81519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fec00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3f80cb570 0x7fc3f8151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fec00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f80c9110 con 0x7fc3f8070c00 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fec00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f815ebe0 con 0x7fc3f80cb570 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fec00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f815b560 con 0x7fc3f8070fe0 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070c00 0x7fc3f814df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070c00 0x7fc3f814df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53484/0 (socket says 172.21.15.98:53484) 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f7e00640 1 -- 172.21.15.98:0/2216305732 learned_addr learned my addr 172.21.15.98:0/2216305732 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fcc00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3f80cb570 0x7fc3f8151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3ef400640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3f8070fe0 0x7fc3f81519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f7e00640 1 -- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3f8070fe0 msgr2=0x7fc3f81519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:29.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f7e00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3f8070fe0 0x7fc3f81519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f7e00640 1 -- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3f80cb570 msgr2=0x7fc3f8151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f7e00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3f80cb570 0x7fc3f8151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f7e00640 1 -- 172.21.15.98:0/2216305732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f8157b70 con 0x7fc3f8070c00 2024-05-27T00:15:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f7e00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070c00 0x7fc3f814df80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c4d0 tx=0x7fc3e800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f4c00640 1 -- 172.21.15.98:0/2216305732 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8018070 con 0x7fc3f8070c00 2024-05-27T00:15:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f4c00640 1 -- 172.21.15.98:0/2216305732 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e80074e0 con 0x7fc3f8070c00 2024-05-27T00:15:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fec00640 1 -- 172.21.15.98:0/2216305732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f814e4c0 con 0x7fc3f8070c00 2024-05-27T00:15:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3f4c00640 1 -- 172.21.15.98:0/2216305732 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8004a30 con 0x7fc3f8070c00 2024-05-27T00:15:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fec00640 1 -- 172.21.15.98:0/2216305732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f81562f0 con 0x7fc3f8070c00 2024-05-27T00:15:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.789+0000 7fc3fec00640 1 -- 172.21.15.98:0/2216305732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f8075cb0 con 0x7fc3f8070c00 2024-05-27T00:15:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.793+0000 7fc3f4c00640 1 -- 172.21.15.98:0/2216305732 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc3e801b070 con 0x7fc3f8070c00 2024-05-27T00:15:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.793+0000 7fc3f4c00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3d40678c0 0x7fc3d4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.793+0000 7fc3f4c00640 1 -- 172.21.15.98:0/2216305732 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc3e8084fd0 con 0x7fc3f8070c00 2024-05-27T00:15:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.793+0000 7fc3ef400640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3d40678c0 0x7fc3d4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.793+0000 7fc3ef400640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3d40678c0 0x7fc3d4069d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00b2000 tx=0x7fc3e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.797+0000 7fc3f4c00640 1 -- 172.21.15.98:0/2216305732 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc3e8055b00 con 0x7fc3f8070c00 2024-05-27T00:15:29.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.981+0000 7fc3fec00640 1 -- 172.21.15.98:0/2216305732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3f80c9720 con 0x7fc3f8070c00 2024-05-27T00:15:29.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3f4c00640 1 -- 172.21.15.98:0/2216305732 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6828 (secure 0 0 0) 0x7fc3e8055400 con 0x7fc3f8070c00 2024-05-27T00:15:29.986 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:29.986 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:28.982518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:15:29.987 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2024-05-27T00:15:29.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 -- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3d40678c0 msgr2=0x7fc3d4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:29.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3d40678c0 0x7fc3d4069d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00b2000 tx=0x7fc3e0008040 comp rx=0 tx=0).stop 2024-05-27T00:15:29.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 -- 172.21.15.98:0/2216305732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070c00 msgr2=0x7fc3f814df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:29.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070c00 0x7fc3f814df80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800c4d0 tx=0x7fc3e800c990 comp rx=0 tx=0).stop 2024-05-27T00:15:29.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 -- 172.21.15.98:0/2216305732 shutdown_connections 2024-05-27T00:15:29.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3d40678c0 0x7fc3d4069d80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3f80cb570 0x7fc3f8151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3f8070fe0 0x7fc3f81519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 --2- 172.21.15.98:0/2216305732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3f8070c00 0x7fc3f814df80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:29.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 -- 172.21.15.98:0/2216305732 >> 172.21.15.98:0/2216305732 conn(0x7fc3f8072f90 msgr2=0x7fc3f815ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:29.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 -- 172.21.15.98:0/2216305732 shutdown_connections 2024-05-27T00:15:29.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:29.985+0000 7fc3fec00640 1 -- 172.21.15.98:0/2216305732 wait complete. 2024-05-27T00:15:30.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:30.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.137+0000 7f5a10c00640 1 Processor -- start 2024-05-27T00:15:30.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.137+0000 7f5a10c00640 1 -- start start 2024-05-27T00:15:30.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a10c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5a0c062720 0x7f5a0c072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a10c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5a0c15e6b0 0x7f5a0c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a10c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a0c0700d0 con 0x7f5a0c154a80 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a10c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5a0c0726a0 con 0x7f5a0c15e6b0 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a10c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f5a0c071a90 con 0x7f5a0c062720 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a0a800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5a0c15e6b0 0x7f5a0c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a09e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5a0c154a80 0x7f5a0c0c6f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43260/0 (socket says 172.21.15.98:43260) 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a09e00640 1 -- 172.21.15.98:0/1547253186 learned_addr learned my addr 172.21.15.98:0/1547253186 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a01400640 1 --2- 172.21.15.98:0/1547253186 >> v2:172.21.15.204:3300/0 conn(0x7f5a0c062720 0x7f5a0c072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a0a800640 1 -- 172.21.15.98:0/1547253186 >> v2:172.21.15.204:3300/0 conn(0x7f5a0c062720 msgr2=0x7f5a0c072b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a0a800640 1 --2- 172.21.15.98:0/1547253186 >> v2:172.21.15.204:3300/0 conn(0x7f5a0c062720 0x7f5a0c072b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a0a800640 1 -- 172.21.15.98:0/1547253186 >> v1:172.21.15.98:6789/0 conn(0x7f5a0c154a80 legacy=0x7f5a0c0c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a0a800640 1 -- 172.21.15.98:0/1547253186 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0c071d70 con 0x7f5a0c15e6b0 2024-05-27T00:15:30.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.141+0000 7f5a09400640 1 -- 172.21.15.98:0/1547253186 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2095228529 0 0) 0x7f5a0c0700d0 con 0x7f5a0c154a80 2024-05-27T00:15:30.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a0a800640 1 --2- 172.21.15.98:0/1547253186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c15e6b0 0x7f5a0c06ed10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0c3e40 tx=0x7f5a0401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630339be457c98ba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a09400640 1 -- 172.21.15.98:0/1547253186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a040b2070 con 0x7f5a0c15e6b0 2024-05-27T00:15:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a09400640 1 -- 172.21.15.98:0/1547253186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a040a5850 con 0x7f5a0c15e6b0 2024-05-27T00:15:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a09400640 1 -- 172.21.15.98:0/1547253186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a040a5c20 con 0x7f5a0c15e6b0 2024-05-27T00:15:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 -- 172.21.15.98:0/1547253186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c15e6b0 msgr2=0x7f5a0c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 --2- 172.21.15.98:0/1547253186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c15e6b0 0x7f5a0c06ed10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0c3e40 tx=0x7f5a0401c8a0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 -- 172.21.15.98:0/1547253186 shutdown_connections 2024-05-27T00:15:30.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 --2- 172.21.15.98:0/1547253186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c15e6b0 0x7f5a0c06ed10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 --2- 172.21.15.98:0/1547253186 >> v2:172.21.15.204:3300/0 conn(0x7f5a0c062720 0x7f5a0c072b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 -- 172.21.15.98:0/1547253186 >> 172.21.15.98:0/1547253186 conn(0x7f5a0c0cc560 msgr2=0x7f5a0c0cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:30.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 -- 172.21.15.98:0/1547253186 shutdown_connections 2024-05-27T00:15:30.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 -- 172.21.15.98:0/1547253186 wait complete. 2024-05-27T00:15:30.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 Processor -- start 2024-05-27T00:15:30.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 -- start start 2024-05-27T00:15:30.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c062720 0x7f5a0c14a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.145+0000 7f5a10c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a0c154a80 0x7f5a0c1458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a10c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a0c15e6b0 0x7f5a0c15ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a10c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c167210 con 0x7f5a0c062720 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a10c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c167090 con 0x7f5a0c15e6b0 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a10c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5a0c167390 con 0x7f5a0c154a80 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a09e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c062720 0x7f5a0c14a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a09e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c062720 0x7f5a0c14a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53496/0 (socket says 172.21.15.98:53496) 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a09e00640 1 -- 172.21.15.98:0/2795921679 learned_addr learned my addr 172.21.15.98:0/2795921679 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a0a800640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a0c15e6b0 0x7f5a0c15ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a01400640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a0c154a80 0x7f5a0c1458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a09e00640 1 -- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a0c154a80 msgr2=0x7f5a0c1458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a09e00640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a0c154a80 0x7f5a0c1458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a09e00640 1 -- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a0c15e6b0 msgr2=0x7f5a0c15ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a09e00640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a0c15e6b0 0x7f5a0c15ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a09e00640 1 -- 172.21.15.98:0/2795921679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0c15dde0 con 0x7f5a0c062720 2024-05-27T00:15:30.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a0a800640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a0c15e6b0 0x7f5a0c15ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:30.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a09e00640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c062720 0x7f5a0c14a030 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c520 tx=0x7f59fc00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:30.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a02a00640 1 -- 172.21.15.98:0/2795921679 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc018070 con 0x7f5a0c062720 2024-05-27T00:15:30.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a02a00640 1 -- 172.21.15.98:0/2795921679 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59fc0074e0 con 0x7f5a0c062720 2024-05-27T00:15:30.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a02a00640 1 -- 172.21.15.98:0/2795921679 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0049f0 con 0x7f5a0c062720 2024-05-27T00:15:30.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a10c00640 1 -- 172.21.15.98:0/2795921679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0c15db00 con 0x7f5a0c062720 2024-05-27T00:15:30.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a10c00640 1 -- 172.21.15.98:0/2795921679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0c1f3860 con 0x7f5a0c062720 2024-05-27T00:15:30.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a10c00640 1 -- 172.21.15.98:0/2795921679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0c075cb0 con 0x7f5a0c062720 2024-05-27T00:15:30.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.149+0000 7f5a02a00640 1 -- 172.21.15.98:0/2795921679 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f59fc01b070 con 0x7f5a0c062720 2024-05-27T00:15:30.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.153+0000 7f5a02a00640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59e80678c0 0x7f59e8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.153+0000 7f5a02a00640 1 -- 172.21.15.98:0/2795921679 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f59fc0850d0 con 0x7f5a0c062720 2024-05-27T00:15:30.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.153+0000 7f5a01400640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59e80678c0 0x7f59e8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.153+0000 7f5a01400640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59e80678c0 0x7f59e8069d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c146350 tx=0x7f59f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:30.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.157+0000 7f5a02a00640 1 -- 172.21.15.98:0/2795921679 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f59fc055c10 con 0x7f5a0c062720 2024-05-27T00:15:30.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.313+0000 7f5a10c00640 1 -- 172.21.15.98:0/2795921679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a0c071670 con 0x7f5a0c062720 2024-05-27T00:15:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.313+0000 7f5a02a00640 1 -- 172.21.15.98:0/2795921679 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16581 (secure 0 0 0) 0x7f59fc055510 con 0x7f5a0c062720 2024-05-27T00:15:30.317 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:30.317 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:26.579229+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:30.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 -- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59e80678c0 msgr2=0x7f59e8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59e80678c0 0x7f59e8069d80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c146350 tx=0x7f59f4008040 comp rx=0 tx=0).stop 2024-05-27T00:15:30.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 -- 172.21.15.98:0/2795921679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c062720 msgr2=0x7f5a0c14a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c062720 0x7f5a0c14a030 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c520 tx=0x7f59fc00c9e0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 -- 172.21.15.98:0/2795921679 shutdown_connections 2024-05-27T00:15:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59e80678c0 0x7f59e8069d80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a0c15e6b0 0x7f5a0c15ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a0c154a80 0x7f5a0c1458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 --2- 172.21.15.98:0/2795921679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a0c062720 0x7f5a0c14a030 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 -- 172.21.15.98:0/2795921679 >> 172.21.15.98:0/2795921679 conn(0x7f5a0c0cc560 msgr2=0x7f5a0c0cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 -- 172.21.15.98:0/2795921679 shutdown_connections 2024-05-27T00:15:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.317+0000 7f5a10c00640 1 -- 172.21.15.98:0/2795921679 wait complete. 2024-05-27T00:15:30.338 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-27T00:15:30.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:15:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcf800640 1 Processor -- start 2024-05-27T00:15:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcf800640 1 -- start start 2024-05-27T00:15:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcf800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc80c79f0 0x7f4dc8157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcf800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc80c17d0 con 0x7f4dc80731f0 2024-05-27T00:15:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcf800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc8070e10 con 0x7f4dc80735d0 2024-05-27T00:15:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcf800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc8071990 con 0x7f4dc80c79f0 2024-05-27T00:15:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcce00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4dc80731f0 0x7f4dc80c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43276/0 (socket says 172.21.15.98:43276) 2024-05-27T00:15:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcce00640 1 -- 172.21.15.98:0/719992811 learned_addr learned my addr 172.21.15.98:0/719992811 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcd800640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc80c79f0 0x7f4dc8157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1429632003 0 0) 0x7f4dc80c17d0 con 0x7f4dc80731f0 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dac003530 con 0x7f4dc80731f0 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 161195932 0 0) 0x7f4dac003530 con 0x7f4dc80731f0 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc80c79f0 msgr2=0x7f4dc8157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc80c79f0 0x7f4dc8157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 >> v1:172.21.15.204:6789/0 conn(0x7f4dc80735d0 legacy=0x7f4dc8151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc80710f0 con 0x7f4dc80731f0 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f4db8001a00 con 0x7f4dc80731f0 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 >> v1:172.21.15.98:6789/0 conn(0x7f4dc80731f0 legacy=0x7f4dc80c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac0039f0 0x7f4dac003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dac004810 0x7f4dac004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc80c79f0 0x7f4dac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc8071990 con 0x7f4dac0039f0 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac00bb10 con 0x7f4dac004810 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4dac00bd10 con 0x7f4dc80c79f0 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcce00640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac0039f0 0x7f4dac003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dc7e00640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dac004810 0x7f4dac004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.473+0000 7f4dcd800640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc80c79f0 0x7f4dac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcce00640 1 -- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc80c79f0 msgr2=0x7f4dac0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcce00640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc80c79f0 0x7f4dac0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcce00640 1 -- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dac004810 msgr2=0x7f4dac004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcce00640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dac004810 0x7f4dac004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcce00640 1 -- 172.21.15.98:0/719992811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dac003530 con 0x7f4dac0039f0 2024-05-27T00:15:30.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dc7e00640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dac004810 0x7f4dac004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:30.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcce00640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac0039f0 0x7f4dac003e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4db800a830 tx=0x7f4db801faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef7065b7bab699c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:30.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db800a480 con 0x7f4dac0039f0 2024-05-27T00:15:30.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db80aa6d0 con 0x7f4dac0039f0 2024-05-27T00:15:30.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dc7400640 1 -- 172.21.15.98:0/719992811 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db80aaae0 con 0x7f4dac0039f0 2024-05-27T00:15:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcf800640 1 -- 172.21.15.98:0/719992811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac0039f0 msgr2=0x7f4dac003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcf800640 1 --2- 172.21.15.98:0/719992811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dac0039f0 0x7f4dac003e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4db800a830 tx=0x7f4db801faa0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcd800640 1 -- 172.21.15.98:0/719992811 reap_dead start 2024-05-27T00:15:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcf800640 1 -- 172.21.15.98:0/719992811 shutdown_connections 2024-05-27T00:15:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.477+0000 7f4dcf800640 1 -- 172.21.15.98:0/719992811 >> 172.21.15.98:0/719992811 conn(0x7f4dc8067530 msgr2=0x7f4dc806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 -- 172.21.15.98:0/719992811 shutdown_connections 2024-05-27T00:15:30.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 -- 172.21.15.98:0/719992811 wait complete. 2024-05-27T00:15:30.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 Processor -- start 2024-05-27T00:15:30.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 -- start start 2024-05-27T00:15:30.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dac0039f0 0x7f4dc8155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc8159b10 0x7f4dc8159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dc81f0aa0 0x7f4dc8153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc80c8400 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc80716b0 con 0x7f4dac0039f0 2024-05-27T00:15:30.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc8151b40 con 0x7f4dc8159b10 2024-05-27T00:15:30.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcd800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dc81f0aa0 0x7f4dc8153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dac0039f0 0x7f4dc8155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcd800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dc81f0aa0 0x7f4dc8153f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53524/0 (socket says 172.21.15.98:53524) 2024-05-27T00:15:30.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcd800640 1 -- 172.21.15.98:0/2033920412 learned_addr learned my addr 172.21.15.98:0/2033920412 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:30.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dc7e00640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc8159b10 0x7f4dc8159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcd800640 1 -- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc8159b10 msgr2=0x7f4dc8159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcd800640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc8159b10 0x7f4dc8159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcd800640 1 -- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dac0039f0 msgr2=0x7f4dc8155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcd800640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dac0039f0 0x7f4dc8155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcd800640 1 -- 172.21.15.98:0/2033920412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc81587f0 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dc7e00640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc8159b10 0x7f4dc8159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:30.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcd800640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dc81f0aa0 0x7f4dc8153f00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0011490 tx=0x7f4dc0011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:30.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dc4c00640 1 -- 172.21.15.98:0/2033920412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0003620 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dc4c00640 1 -- 172.21.15.98:0/2033920412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc001b070 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 -- 172.21.15.98:0/2033920412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc8152060 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dc4c00640 1 -- 172.21.15.98:0/2033920412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0003a00 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.481+0000 7f4dcf800640 1 -- 172.21.15.98:0/2033920412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc8158a90 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.485+0000 7f4dcf800640 1 -- 172.21.15.98:0/2033920412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dc8075cb0 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.485+0000 7f4dc4c00640 1 -- 172.21.15.98:0/2033920412 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4dc0003ba0 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.485+0000 7f4dc4c00640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4da80678c0 0x7f4da8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.485+0000 7f4dc4c00640 1 -- 172.21.15.98:0/2033920412 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f4dc00859a0 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.485+0000 7f4dcce00640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4da80678c0 0x7f4da8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.485+0000 7f4dcce00640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4da80678c0 0x7f4da8069d80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4db80b1000 tx=0x7f4db80ab000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.489+0000 7f4dc4c00640 1 -- 172.21.15.98:0/2033920412 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4dc0024020 con 0x7f4dc81f0aa0 2024-05-27T00:15:30.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.653+0000 7f4dcf800640 1 -- 172.21.15.98:0/2033920412 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4dc8158510 con 0x7f4da80678c0 2024-05-27T00:15:30.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.673+0000 7f4dc4c00640 1 -- 172.21.15.98:0/2033920412 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251787 (secure 0 0 0) 0x7f4db81017a0 con 0x7f4da80678c0 2024-05-27T00:15:30.679 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:30.680 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:15:30.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 -- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4da80678c0 msgr2=0x7f4da8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4da80678c0 0x7f4da8069d80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4db80b1000 tx=0x7f4db80ab000 comp rx=0 tx=0).stop 2024-05-27T00:15:30.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 -- 172.21.15.98:0/2033920412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dc81f0aa0 msgr2=0x7f4dc8153f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dc81f0aa0 0x7f4dc8153f00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0011490 tx=0x7f4dc0011950 comp rx=0 tx=0).stop 2024-05-27T00:15:30.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 -- 172.21.15.98:0/2033920412 shutdown_connections 2024-05-27T00:15:30.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4dc81f0aa0 0x7f4dc8153f00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4dc8159b10 0x7f4dc8159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4da80678c0 0x7f4da8069d80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 --2- 172.21.15.98:0/2033920412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4dac0039f0 0x7f4dc8155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 -- 172.21.15.98:0/2033920412 >> 172.21.15.98:0/2033920412 conn(0x7f4dc8067530 msgr2=0x7f4dc80c6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:30.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 -- 172.21.15.98:0/2033920412 shutdown_connections 2024-05-27T00:15:30.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.677+0000 7f4dcf800640 1 -- 172.21.15.98:0/2033920412 wait complete. 2024-05-27T00:15:30.699 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-05-27T00:15:29.133955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219004,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755716,"statfs":{"total":773094113280,"available":772869853184,"internally_reserved":0,"allocated":2891776,"data_stored":1761600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.563533"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644738+0000","last_change":"2024-05-27T00:15:25.640698+0000","last_active":"2024-05-27T00:15:25.644738+0000","last_peered":"2024-05-27T00:15:25.644738+0000","last_clean":"2024-05-27T00:15:25.644738+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:15:25.644738+0000","last_undegraded":"2024-05-27T00:15:25.644738+0000","last_fullsized":"2024-05-27T00:15:25.644738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.600259+0000","last_change":"2024-05-27T00:15:25.598672+0000","last_active":"2024-05-27T00:15:25.600259+0000","last_peered":"2024-05-27T00:15:25.600259+0000","last_clean":"2024-05-27T00:15:25.600259+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:15:25.600259+0000","last_undegraded":"2024-05-27T00:15:25.600259+0000","last_fullsized":"2024-05-27T00:15:25.600259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.594509+0000","last_change":"2024-05-27T00:15:25.593316+0000","last_active":"2024-05-27T00:15:25.594509+0000","last_peered":"2024-05-27T00:15:25.594509+0000","last_clean":"2024-05-27T00:15:25.594509+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:15:25.594509+0000","last_undegraded":"2024-05-27T00:15:25.594509+0000","last_fullsized":"2024-05-27T00:15:25.594509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:27.674418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636581+0000","last_change":"2024-05-27T00:15:25.620237+0000","last_active":"2024-05-27T00:15:25.636581+0000","last_peered":"2024-05-27T00:15:25.636581+0000","last_clean":"2024-05-27T00:15:25.636581+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:15:25.636581+0000","last_undegraded":"2024-05-27T00:15:25.636581+0000","last_fullsized":"2024-05-27T00:15:25.636581+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:20.567777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644713+0000","last_change":"2024-05-27T00:15:25.640146+0000","last_active":"2024-05-27T00:15:25.644713+0000","last_peered":"2024-05-27T00:15:25.644713+0000","last_clean":"2024-05-27T00:15:25.644713+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:15:25.644713+0000","last_undegraded":"2024-05-27T00:15:25.644713+0000","last_fullsized":"2024-05-27T00:15:25.644713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636638+0000","last_change":"2024-05-27T00:15:25.627128+0000","last_active":"2024-05-27T00:15:25.636638+0000","last_peered":"2024-05-27T00:15:25.636638+0000","last_clean":"2024-05-27T00:15:25.636638+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:15:25.636638+0000","last_undegraded":"2024-05-27T00:15:25.636638+0000","last_fullsized":"2024-05-27T00:15:25.636638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.643006+0000","last_change":"2024-05-27T00:15:25.630446+0000","last_active":"2024-05-27T00:15:25.643006+0000","last_peered":"2024-05-27T00:15:25.643006+0000","last_clean":"2024-05-27T00:15:25.643006+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:15:25.643006+0000","last_undegraded":"2024-05-27T00:15:25.643006+0000","last_fullsized":"2024-05-27T00:15:25.643006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.613273+0000","last_change":"2024-05-27T00:15:25.612157+0000","last_active":"2024-05-27T00:15:25.613273+0000","last_peered":"2024-05-27T00:15:25.613273+0000","last_clean":"2024-05-27T00:15:25.613273+0000","last_became_active":"2024-05-27T00:15:25.611884+0000","last_became_peered":"2024-05-27T00:15:25.611884+0000","last_unstale":"2024-05-27T00:15:25.613273+0000","last_undegraded":"2024-05-27T00:15:25.613273+0000","last_fullsized":"2024-05-27T00:15:25.613273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:57.827118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.646283+0000","last_change":"2024-05-27T00:15:25.643312+0000","last_active":"2024-05-27T00:15:25.646283+0000","last_peered":"2024-05-27T00:15:25.646283+0000","last_clean":"2024-05-27T00:15:25.646283+0000","last_became_active":"2024-05-27T00:15:25.642663+0000","last_became_peered":"2024-05-27T00:15:25.642663+0000","last_unstale":"2024-05-27T00:15:25.646283+0000","last_undegraded":"2024-05-27T00:15:25.646283+0000","last_fullsized":"2024-05-27T00:15:25.646283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.637196+0000","last_change":"2024-05-27T00:15:25.629892+0000","last_active":"2024-05-27T00:15:25.637196+0000","last_peered":"2024-05-27T00:15:25.637196+0000","last_clean":"2024-05-27T00:15:25.637196+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:15:25.637196+0000","last_undegraded":"2024-05-27T00:15:25.637196+0000","last_fullsized":"2024-05-27T00:15:25.637196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:34:15.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.621642+0000","last_change":"2024-05-27T00:15:25.614887+0000","last_active":"2024-05-27T00:15:25.621642+0000","last_peered":"2024-05-27T00:15:25.621642+0000","last_clean":"2024-05-27T00:15:25.621642+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:15:25.621642+0000","last_undegraded":"2024-05-27T00:15:25.621642+0000","last_fullsized":"2024-05-27T00:15:25.621642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:18.151532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.630397+0000","last_change":"2024-05-27T00:15:25.623864+0000","last_active":"2024-05-27T00:15:25.630397+0000","last_peered":"2024-05-27T00:15:25.630397+0000","last_clean":"2024-05-27T00:15:25.630397+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:15:25.630397+0000","last_undegraded":"2024-05-27T00:15:25.630397+0000","last_fullsized":"2024-05-27T00:15:25.630397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:27.674418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.616857+0000","last_change":"2024-05-27T00:15:25.615495+0000","last_active":"2024-05-27T00:15:25.616857+0000","last_peered":"2024-05-27T00:15:25.616857+0000","last_clean":"2024-05-27T00:15:25.616857+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:15:25.616857+0000","last_undegraded":"2024-05-27T00:15:25.616857+0000","last_fullsized":"2024-05-27T00:15:25.616857+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:07:03.281228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.596028+0000","last_change":"2024-05-27T00:15:25.594508+0000","last_active":"2024-05-27T00:15:25.596028+0000","last_peered":"2024-05-27T00:15:25.596028+0000","last_clean":"2024-05-27T00:15:25.596028+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:15:25.596028+0000","last_undegraded":"2024-05-27T00:15:25.596028+0000","last_fullsized":"2024-05-27T00:15:25.596028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:53.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612962+0000","last_change":"2024-05-27T00:15:25.611818+0000","last_active":"2024-05-27T00:15:25.612962+0000","last_peered":"2024-05-27T00:15:25.612962+0000","last_clean":"2024-05-27T00:15:25.612962+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:15:25.612962+0000","last_undegraded":"2024-05-27T00:15:25.612962+0000","last_fullsized":"2024-05-27T00:15:25.612962+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.634813+0000","last_change":"2024-05-27T00:15:25.611365+0000","last_active":"2024-05-27T00:15:25.634813+0000","last_peered":"2024-05-27T00:15:25.634813+0000","last_clean":"2024-05-27T00:15:25.634813+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:15:25.634813+0000","last_undegraded":"2024-05-27T00:15:25.634813+0000","last_fullsized":"2024-05-27T00:15:25.634813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.641449+0000","last_change":"2024-05-27T00:15:25.632120+0000","last_active":"2024-05-27T00:15:25.641449+0000","last_peered":"2024-05-27T00:15:25.641449+0000","last_clean":"2024-05-27T00:15:25.641449+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:15:25.641449+0000","last_undegraded":"2024-05-27T00:15:25.641449+0000","last_fullsized":"2024-05-27T00:15:25.641449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.617354+0000","last_change":"2024-05-27T00:15:25.615868+0000","last_active":"2024-05-27T00:15:25.617354+0000","last_peered":"2024-05-27T00:15:25.617354+0000","last_clean":"2024-05-27T00:15:25.617354+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:15:25.617354+0000","last_undegraded":"2024-05-27T00:15:25.617354+0000","last_fullsized":"2024-05-27T00:15:25.617354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:14.969827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.642872+0000","last_change":"2024-05-27T00:15:25.629101+0000","last_active":"2024-05-27T00:15:25.642872+0000","last_peered":"2024-05-27T00:15:25.642872+0000","last_clean":"2024-05-27T00:15:25.642872+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:15:25.642872+0000","last_undegraded":"2024-05-27T00:15:25.642872+0000","last_fullsized":"2024-05-27T00:15:25.642872+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.629018+0000","last_change":"2024-05-27T00:15:25.619351+0000","last_active":"2024-05-27T00:15:25.629018+0000","last_peered":"2024-05-27T00:15:25.629018+0000","last_clean":"2024-05-27T00:15:25.629018+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:15:25.629018+0000","last_undegraded":"2024-05-27T00:15:25.629018+0000","last_fullsized":"2024-05-27T00:15:25.629018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:19.321535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.617528+0000","last_change":"2024-05-27T00:15:25.616436+0000","last_active":"2024-05-27T00:15:25.617528+0000","last_peered":"2024-05-27T00:15:25.617528+0000","last_clean":"2024-05-27T00:15:25.617528+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:15:25.617528+0000","last_undegraded":"2024-05-27T00:15:25.617528+0000","last_fullsized":"2024-05-27T00:15:25.617528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:20.567777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.641275+0000","last_change":"2024-05-27T00:15:25.629373+0000","last_active":"2024-05-27T00:15:25.641275+0000","last_peered":"2024-05-27T00:15:25.641275+0000","last_clean":"2024-05-27T00:15:25.641275+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:15:25.641275+0000","last_undegraded":"2024-05-27T00:15:25.641275+0000","last_fullsized":"2024-05-27T00:15:25.641275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612525+0000","last_change":"2024-05-27T00:15:25.610707+0000","last_active":"2024-05-27T00:15:25.612525+0000","last_peered":"2024-05-27T00:15:25.612525+0000","last_clean":"2024-05-27T00:15:25.612525+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:15:25.612525+0000","last_undegraded":"2024-05-27T00:15:25.612525+0000","last_fullsized":"2024-05-27T00:15:25.612525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:17:16.223164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.635989+0000","last_change":"2024-05-27T00:15:25.613013+0000","last_active":"2024-05-27T00:15:25.635989+0000","last_peered":"2024-05-27T00:15:25.635989+0000","last_clean":"2024-05-27T00:15:25.635989+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:15:25.635989+0000","last_undegraded":"2024-05-27T00:15:25.635989+0000","last_fullsized":"2024-05-27T00:15:25.635989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612412+0000","last_change":"2024-05-27T00:15:25.610070+0000","last_active":"2024-05-27T00:15:25.612412+0000","last_peered":"2024-05-27T00:15:25.612412+0000","last_clean":"2024-05-27T00:15:25.612412+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:15:25.612412+0000","last_undegraded":"2024-05-27T00:15:25.612412+0000","last_fullsized":"2024-05-27T00:15:25.612412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:50:17.488247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.637249+0000","last_change":"2024-05-27T00:15:25.633470+0000","last_active":"2024-05-27T00:15:25.637249+0000","last_peered":"2024-05-27T00:15:25.637249+0000","last_clean":"2024-05-27T00:15:25.637249+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:15:25.637249+0000","last_undegraded":"2024-05-27T00:15:25.637249+0000","last_fullsized":"2024-05-27T00:15:25.637249+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.598794+0000","last_change":"2024-05-27T00:15:25.597615+0000","last_active":"2024-05-27T00:15:25.598794+0000","last_peered":"2024-05-27T00:15:25.598794+0000","last_clean":"2024-05-27T00:15:25.598794+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:15:25.598794+0000","last_undegraded":"2024-05-27T00:15:25.598794+0000","last_fullsized":"2024-05-27T00:15:25.598794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612682+0000","last_change":"2024-05-27T00:15:25.610933+0000","last_active":"2024-05-27T00:15:25.612682+0000","last_peered":"2024-05-27T00:15:25.612682+0000","last_clean":"2024-05-27T00:15:25.612682+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:15:25.612682+0000","last_undegraded":"2024-05-27T00:15:25.612682+0000","last_fullsized":"2024-05-27T00:15:25.612682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:51:00.406879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.600213+0000","last_change":"2024-05-27T00:15:25.598227+0000","last_active":"2024-05-27T00:15:25.600213+0000","last_peered":"2024-05-27T00:15:25.600213+0000","last_clean":"2024-05-27T00:15:25.600213+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:15:25.600213+0000","last_undegraded":"2024-05-27T00:15:25.600213+0000","last_fullsized":"2024-05-27T00:15:25.600213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:14.969827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636424+0000","last_change":"2024-05-27T00:15:25.620621+0000","last_active":"2024-05-27T00:15:25.636424+0000","last_peered":"2024-05-27T00:15:25.636424+0000","last_clean":"2024-05-27T00:15:25.636424+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:15:25.636424+0000","last_undegraded":"2024-05-27T00:15:25.636424+0000","last_fullsized":"2024-05-27T00:15:25.636424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:34:15.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.606866+0000","last_change":"2024-05-27T00:15:25.605839+0000","last_active":"2024-05-27T00:15:25.606866+0000","last_peered":"2024-05-27T00:15:25.606866+0000","last_clean":"2024-05-27T00:15:25.606866+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:15:25.606866+0000","last_undegraded":"2024-05-27T00:15:25.606866+0000","last_fullsized":"2024-05-27T00:15:25.606866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.624139+0000","last_change":"2024-05-27T00:15:25.607188+0000","last_active":"2024-05-27T00:15:25.624139+0000","last_peered":"2024-05-27T00:15:25.624139+0000","last_clean":"2024-05-27T00:15:25.624139+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:15:25.624139+0000","last_undegraded":"2024-05-27T00:15:25.624139+0000","last_fullsized":"2024-05-27T00:15:25.624139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.629902+0000","last_change":"2024-05-27T00:15:25.615579+0000","last_active":"2024-05-27T00:15:25.629902+0000","last_peered":"2024-05-27T00:15:25.629902+0000","last_clean":"2024-05-27T00:15:25.629902+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:15:25.629902+0000","last_undegraded":"2024-05-27T00:15:25.629902+0000","last_fullsized":"2024-05-27T00:15:25.629902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:56.623917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.588223+0000","last_change":"2024-05-27T00:15:24.611108+0000","last_active":"2024-05-27T00:15:25.588223+0000","last_peered":"2024-05-27T00:15:25.588223+0000","last_clean":"2024-05-27T00:15:25.588223+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:15:25.588223+0000","last_undegraded":"2024-05-27T00:15:25.588223+0000","last_fullsized":"2024-05-27T00:15:25.588223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:11.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.617450+0000","last_change":"2024-05-27T00:15:25.614251+0000","last_active":"2024-05-27T00:15:25.617450+0000","last_peered":"2024-05-27T00:15:25.617450+0000","last_clean":"2024-05-27T00:15:25.617450+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:15:25.617450+0000","last_undegraded":"2024-05-27T00:15:25.617450+0000","last_fullsized":"2024-05-27T00:15:25.617450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:19.321535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.604413+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:15:25.604413+0000","last_peered":"2024-05-27T00:15:25.604413+0000","last_clean":"2024-05-27T00:15:25.604413+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:15:25.604413+0000","last_undegraded":"2024-05-27T00:15:25.604413+0000","last_fullsized":"2024-05-27T00:15:25.604413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.586705+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:15:25.586705+0000","last_peered":"2024-05-27T00:15:25.586705+0000","last_clean":"2024-05-27T00:15:25.586705+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:15:25.586705+0000","last_undegraded":"2024-05-27T00:15:25.586705+0000","last_fullsized":"2024-05-27T00:15:25.586705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:46:08.645927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.598107+0000","last_change":"2024-05-27T00:15:25.596844+0000","last_active":"2024-05-27T00:15:25.598107+0000","last_peered":"2024-05-27T00:15:25.598107+0000","last_clean":"2024-05-27T00:15:25.598107+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:15:25.598107+0000","last_undegraded":"2024-05-27T00:15:25.598107+0000","last_fullsized":"2024-05-27T00:15:25.598107+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:38:49.590477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.595284+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:15:25.595284+0000","last_peered":"2024-05-27T00:15:25.595284+0000","last_clean":"2024-05-27T00:15:25.595284+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:15:25.595284+0000","last_undegraded":"2024-05-27T00:15:25.595284+0000","last_fullsized":"2024-05-27T00:15:25.595284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.594566+0000","last_change":"2024-05-27T00:15:25.593149+0000","last_active":"2024-05-27T00:15:25.594566+0000","last_peered":"2024-05-27T00:15:25.594566+0000","last_clean":"2024-05-27T00:15:25.594566+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:15:25.594566+0000","last_undegraded":"2024-05-27T00:15:25.594566+0000","last_fullsized":"2024-05-27T00:15:25.594566+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:34:15.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.605830+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:15:25.605830+0000","last_peered":"2024-05-27T00:15:25.605830+0000","last_clean":"2024-05-27T00:15:25.605830+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:15:25.605830+0000","last_undegraded":"2024-05-27T00:15:25.605830+0000","last_fullsized":"2024-05-27T00:15:25.605830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636125+0000","last_change":"2024-05-27T00:15:25.618650+0000","last_active":"2024-05-27T00:15:25.636125+0000","last_peered":"2024-05-27T00:15:25.636125+0000","last_clean":"2024-05-27T00:15:25.636125+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:15:25.636125+0000","last_undegraded":"2024-05-27T00:15:25.636125+0000","last_fullsized":"2024-05-27T00:15:25.636125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:18.151532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.605486+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:15:25.605486+0000","last_peered":"2024-05-27T00:15:25.605486+0000","last_clean":"2024-05-27T00:15:25.605486+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:15:25.605486+0000","last_undegraded":"2024-05-27T00:15:25.605486+0000","last_fullsized":"2024-05-27T00:15:25.605486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.616710+0000","last_change":"2024-05-27T00:15:25.613142+0000","last_active":"2024-05-27T00:15:25.616710+0000","last_peered":"2024-05-27T00:15:25.616710+0000","last_clean":"2024-05-27T00:15:25.616710+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:15:25.616710+0000","last_undegraded":"2024-05-27T00:15:25.616710+0000","last_fullsized":"2024-05-27T00:15:25.616710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:18.151532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.582658+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:15:25.582658+0000","last_peered":"2024-05-27T00:15:25.582658+0000","last_clean":"2024-05-27T00:15:25.582658+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:15:25.582658+0000","last_undegraded":"2024-05-27T00:15:25.582658+0000","last_fullsized":"2024-05-27T00:15:25.582658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.630339+0000","last_change":"2024-05-27T00:15:25.622383+0000","last_active":"2024-05-27T00:15:25.630339+0000","last_peered":"2024-05-27T00:15:25.630339+0000","last_clean":"2024-05-27T00:15:25.630339+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:15:25.630339+0000","last_undegraded":"2024-05-27T00:15:25.630339+0000","last_fullsized":"2024-05-27T00:15:25.630339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:24.430219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.605403+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:15:25.605403+0000","last_peered":"2024-05-27T00:15:25.605403+0000","last_clean":"2024-05-27T00:15:25.605403+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:15:25.605403+0000","last_undegraded":"2024-05-27T00:15:25.605403+0000","last_fullsized":"2024-05-27T00:15:25.605403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.642810+0000","last_change":"2024-05-27T00:15:25.628646+0000","last_active":"2024-05-27T00:15:25.642810+0000","last_peered":"2024-05-27T00:15:25.642810+0000","last_clean":"2024-05-27T00:15:25.642810+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:15:25.642810+0000","last_undegraded":"2024-05-27T00:15:25.642810+0000","last_fullsized":"2024-05-27T00:15:25.642810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:34:15.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.590531+0000","last_change":"2024-05-27T00:15:24.605986+0000","last_active":"2024-05-27T00:15:25.590531+0000","last_peered":"2024-05-27T00:15:25.590531+0000","last_clean":"2024-05-27T00:15:25.590531+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:15:25.590531+0000","last_undegraded":"2024-05-27T00:15:25.590531+0000","last_fullsized":"2024-05-27T00:15:25.590531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.625031+0000","last_change":"2024-05-27T00:15:25.624136+0000","last_active":"2024-05-27T00:15:25.625031+0000","last_peered":"2024-05-27T00:15:25.625031+0000","last_clean":"2024-05-27T00:15:25.625031+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:15:25.625031+0000","last_undegraded":"2024-05-27T00:15:25.625031+0000","last_fullsized":"2024-05-27T00:15:25.625031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:34:15.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.613823+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:15:25.613823+0000","last_peered":"2024-05-27T00:15:25.613823+0000","last_clean":"2024-05-27T00:15:25.613823+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:15:25.613823+0000","last_undegraded":"2024-05-27T00:15:25.613823+0000","last_fullsized":"2024-05-27T00:15:25.613823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644760+0000","last_change":"2024-05-27T00:15:25.641965+0000","last_active":"2024-05-27T00:15:25.644760+0000","last_peered":"2024-05-27T00:15:25.644760+0000","last_clean":"2024-05-27T00:15:25.644760+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:15:25.644760+0000","last_undegraded":"2024-05-27T00:15:25.644760+0000","last_fullsized":"2024-05-27T00:15:25.644760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.603619+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:15:25.603619+0000","last_peered":"2024-05-27T00:15:25.603619+0000","last_clean":"2024-05-27T00:15:25.603619+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:15:25.603619+0000","last_undegraded":"2024-05-27T00:15:25.603619+0000","last_fullsized":"2024-05-27T00:15:25.603619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.624196+0000","last_change":"2024-05-27T00:15:25.608549+0000","last_active":"2024-05-27T00:15:25.624196+0000","last_peered":"2024-05-27T00:15:25.624196+0000","last_clean":"2024-05-27T00:15:25.624196+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:15:25.624196+0000","last_undegraded":"2024-05-27T00:15:25.624196+0000","last_fullsized":"2024-05-27T00:15:25.624196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.588573+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:15:25.588573+0000","last_peered":"2024-05-27T00:15:25.588573+0000","last_clean":"2024-05-27T00:15:25.588573+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:15:25.588573+0000","last_undegraded":"2024-05-27T00:15:25.588573+0000","last_fullsized":"2024-05-27T00:15:25.588573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644452+0000","last_change":"2024-05-27T00:15:25.638558+0000","last_active":"2024-05-27T00:15:25.644452+0000","last_peered":"2024-05-27T00:15:25.644452+0000","last_clean":"2024-05-27T00:15:25.644452+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:15:25.644452+0000","last_undegraded":"2024-05-27T00:15:25.644452+0000","last_fullsized":"2024-05-27T00:15:25.644452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.590059+0000","last_change":"2024-05-27T00:15:24.601924+0000","last_active":"2024-05-27T00:15:25.590059+0000","last_peered":"2024-05-27T00:15:25.590059+0000","last_clean":"2024-05-27T00:15:25.590059+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:15:25.590059+0000","last_undegraded":"2024-05-27T00:15:25.590059+0000","last_fullsized":"2024-05-27T00:15:25.590059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:32.185839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644521+0000","last_change":"2024-05-27T00:15:25.639140+0000","last_active":"2024-05-27T00:15:25.644521+0000","last_peered":"2024-05-27T00:15:25.644521+0000","last_clean":"2024-05-27T00:15:25.644521+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:15:25.644521+0000","last_undegraded":"2024-05-27T00:15:25.644521+0000","last_fullsized":"2024-05-27T00:15:25.644521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.588234+0000","last_change":"2024-05-27T00:15:24.600878+0000","last_active":"2024-05-27T00:15:25.588234+0000","last_peered":"2024-05-27T00:15:25.588234+0000","last_clean":"2024-05-27T00:15:25.588234+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:15:25.588234+0000","last_undegraded":"2024-05-27T00:15:25.588234+0000","last_fullsized":"2024-05-27T00:15:25.588234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.648649+0000","last_change":"2024-05-27T00:15:25.644249+0000","last_active":"2024-05-27T00:15:25.648649+0000","last_peered":"2024-05-27T00:15:25.648649+0000","last_clean":"2024-05-27T00:15:25.648649+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:15:25.648649+0000","last_undegraded":"2024-05-27T00:15:25.648649+0000","last_fullsized":"2024-05-27T00:15:25.648649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.582542+0000","last_change":"2024-05-27T00:15:24.603452+0000","last_active":"2024-05-27T00:15:25.582542+0000","last_peered":"2024-05-27T00:15:25.582542+0000","last_clean":"2024-05-27T00:15:25.582542+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:15:25.582542+0000","last_undegraded":"2024-05-27T00:15:25.582542+0000","last_fullsized":"2024-05-27T00:15:25.582542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636229+0000","last_change":"2024-05-27T00:15:25.619341+0000","last_active":"2024-05-27T00:15:25.636229+0000","last_peered":"2024-05-27T00:15:25.636229+0000","last_clean":"2024-05-27T00:15:25.636229+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:15:25.636229+0000","last_undegraded":"2024-05-27T00:15:25.636229+0000","last_fullsized":"2024-05-27T00:15:25.636229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:19.321535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.588785+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:15:25.588785+0000","last_peered":"2024-05-27T00:15:25.588785+0000","last_clean":"2024-05-27T00:15:25.588785+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:15:25.588785+0000","last_undegraded":"2024-05-27T00:15:25.588785+0000","last_fullsized":"2024-05-27T00:15:25.588785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644355+0000","last_change":"2024-05-27T00:15:25.639462+0000","last_active":"2024-05-27T00:15:25.644355+0000","last_peered":"2024-05-27T00:15:25.644355+0000","last_clean":"2024-05-27T00:15:25.644355+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:15:25.644355+0000","last_undegraded":"2024-05-27T00:15:25.644355+0000","last_fullsized":"2024-05-27T00:15:25.644355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":168,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.610911+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:15:25.610911+0000","last_peered":"2024-05-27T00:15:25.610911+0000","last_clean":"2024-05-27T00:15:25.610911+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:15:25.610911+0000","last_undegraded":"2024-05-27T00:15:25.610911+0000","last_fullsized":"2024-05-27T00:15:25.610911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.596096+0000","last_change":"2024-05-27T00:15:24.603781+0000","last_active":"2024-05-27T00:15:25.596096+0000","last_peered":"2024-05-27T00:15:25.596096+0000","last_clean":"2024-05-27T00:15:25.596096+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:15:25.596096+0000","last_undegraded":"2024-05-27T00:15:25.596096+0000","last_fullsized":"2024-05-27T00:15:25.596096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.632804+0000","last_change":"2024-05-27T00:15:25.631978+0000","last_active":"2024-05-27T00:15:25.632804+0000","last_peered":"2024-05-27T00:15:25.632804+0000","last_clean":"2024-05-27T00:15:25.632804+0000","last_became_active":"2024-05-27T00:15:25.631697+0000","last_became_peered":"2024-05-27T00:15:25.631697+0000","last_unstale":"2024-05-27T00:15:25.632804+0000","last_undegraded":"2024-05-27T00:15:25.632804+0000","last_fullsized":"2024-05-27T00:15:25.632804+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:20:40.605562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.610777+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:15:25.610777+0000","last_peered":"2024-05-27T00:15:25.610777+0000","last_clean":"2024-05-27T00:15:25.610777+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:15:25.610777+0000","last_undegraded":"2024-05-27T00:15:25.610777+0000","last_fullsized":"2024-05-27T00:15:25.610777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612161+0000","last_change":"2024-05-27T00:15:25.610627+0000","last_active":"2024-05-27T00:15:25.612161+0000","last_peered":"2024-05-27T00:15:25.612161+0000","last_clean":"2024-05-27T00:15:25.612161+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:15:25.612161+0000","last_undegraded":"2024-05-27T00:15:25.612161+0000","last_fullsized":"2024-05-27T00:15:25.612161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.588094+0000","last_change":"2024-05-27T00:15:24.598182+0000","last_active":"2024-05-27T00:15:25.588094+0000","last_peered":"2024-05-27T00:15:25.588094+0000","last_clean":"2024-05-27T00:15:25.588094+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:15:25.588094+0000","last_undegraded":"2024-05-27T00:15:25.588094+0000","last_fullsized":"2024-05-27T00:15:25.588094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.594401+0000","last_change":"2024-05-27T00:15:25.591449+0000","last_active":"2024-05-27T00:15:25.594401+0000","last_peered":"2024-05-27T00:15:25.594401+0000","last_clean":"2024-05-27T00:15:25.594401+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:15:25.594401+0000","last_undegraded":"2024-05-27T00:15:25.594401+0000","last_fullsized":"2024-05-27T00:15:25.594401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:20.567777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.611803+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:15:25.611803+0000","last_peered":"2024-05-27T00:15:25.611803+0000","last_clean":"2024-05-27T00:15:25.611803+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:15:25.611803+0000","last_undegraded":"2024-05-27T00:15:25.611803+0000","last_fullsized":"2024-05-27T00:15:25.611803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.630803+0000","last_change":"2024-05-27T00:15:25.614073+0000","last_active":"2024-05-27T00:15:25.630803+0000","last_peered":"2024-05-27T00:15:25.630803+0000","last_clean":"2024-05-27T00:15:25.630803+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:15:25.630803+0000","last_undegraded":"2024-05-27T00:15:25.630803+0000","last_fullsized":"2024-05-27T00:15:25.630803+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:17:28.989259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.581600+0000","last_change":"2024-05-27T00:15:24.603518+0000","last_active":"2024-05-27T00:15:25.581600+0000","last_peered":"2024-05-27T00:15:25.581600+0000","last_clean":"2024-05-27T00:15:25.581600+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:15:25.581600+0000","last_undegraded":"2024-05-27T00:15:25.581600+0000","last_fullsized":"2024-05-27T00:15:25.581600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636656+0000","last_change":"2024-05-27T00:15:25.621519+0000","last_active":"2024-05-27T00:15:25.636656+0000","last_peered":"2024-05-27T00:15:25.636656+0000","last_clean":"2024-05-27T00:15:25.636656+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:15:25.636656+0000","last_undegraded":"2024-05-27T00:15:25.636656+0000","last_fullsized":"2024-05-27T00:15:25.636656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.617116+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:15:25.617116+0000","last_peered":"2024-05-27T00:15:25.617116+0000","last_clean":"2024-05-27T00:15:25.617116+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:15:25.617116+0000","last_undegraded":"2024-05-27T00:15:25.617116+0000","last_fullsized":"2024-05-27T00:15:25.617116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.597110+0000","last_change":"2024-05-27T00:15:25.593180+0000","last_active":"2024-05-27T00:15:25.597110+0000","last_peered":"2024-05-27T00:15:25.597110+0000","last_clean":"2024-05-27T00:15:25.597110+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:15:25.597110+0000","last_undegraded":"2024-05-27T00:15:25.597110+0000","last_fullsized":"2024-05-27T00:15:25.597110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.582108+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:15:25.582108+0000","last_peered":"2024-05-27T00:15:25.582108+0000","last_clean":"2024-05-27T00:15:25.582108+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:15:25.582108+0000","last_undegraded":"2024-05-27T00:15:25.582108+0000","last_fullsized":"2024-05-27T00:15:25.582108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.632794+0000","last_change":"2024-05-27T00:15:25.631919+0000","last_active":"2024-05-27T00:15:25.632794+0000","last_peered":"2024-05-27T00:15:25.632794+0000","last_clean":"2024-05-27T00:15:25.632794+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:15:25.632794+0000","last_undegraded":"2024-05-27T00:15:25.632794+0000","last_fullsized":"2024-05-27T00:15:25.632794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:14:01.167988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.606237+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:15:25.606237+0000","last_peered":"2024-05-27T00:15:25.606237+0000","last_clean":"2024-05-27T00:15:25.606237+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:15:25.606237+0000","last_undegraded":"2024-05-27T00:15:25.606237+0000","last_fullsized":"2024-05-27T00:15:25.606237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.597970+0000","last_change":"2024-05-27T00:15:25.596792+0000","last_active":"2024-05-27T00:15:25.597970+0000","last_peered":"2024-05-27T00:15:25.597970+0000","last_clean":"2024-05-27T00:15:25.597970+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:15:25.597970+0000","last_undegraded":"2024-05-27T00:15:25.597970+0000","last_fullsized":"2024-05-27T00:15:25.597970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.610393+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:15:25.610393+0000","last_peered":"2024-05-27T00:15:25.610393+0000","last_clean":"2024-05-27T00:15:25.610393+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:15:25.610393+0000","last_undegraded":"2024-05-27T00:15:25.610393+0000","last_fullsized":"2024-05-27T00:15:25.610393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.616145+0000","last_change":"2024-05-27T00:15:25.610697+0000","last_active":"2024-05-27T00:15:25.616145+0000","last_peered":"2024-05-27T00:15:25.616145+0000","last_clean":"2024-05-27T00:15:25.616145+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:15:25.616145+0000","last_undegraded":"2024-05-27T00:15:25.616145+0000","last_fullsized":"2024-05-27T00:15:25.616145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:40:21.445109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.582789+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:15:25.582789+0000","last_peered":"2024-05-27T00:15:25.582789+0000","last_clean":"2024-05-27T00:15:25.582789+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:15:25.582789+0000","last_undegraded":"2024-05-27T00:15:25.582789+0000","last_fullsized":"2024-05-27T00:15:25.582789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.599907+0000","last_change":"2024-05-27T00:15:25.597164+0000","last_active":"2024-05-27T00:15:25.599907+0000","last_peered":"2024-05-27T00:15:25.599907+0000","last_clean":"2024-05-27T00:15:25.599907+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:15:25.599907+0000","last_undegraded":"2024-05-27T00:15:25.599907+0000","last_fullsized":"2024-05-27T00:15:25.599907+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:06:14.025980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.581442+0000","last_change":"2024-05-27T00:15:24.598238+0000","last_active":"2024-05-27T00:15:25.581442+0000","last_peered":"2024-05-27T00:15:25.581442+0000","last_clean":"2024-05-27T00:15:25.581442+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:15:25.581442+0000","last_undegraded":"2024-05-27T00:15:25.581442+0000","last_fullsized":"2024-05-27T00:15:25.581442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.601936+0000","last_change":"2024-05-27T00:15:25.601140+0000","last_active":"2024-05-27T00:15:25.601936+0000","last_peered":"2024-05-27T00:15:25.601936+0000","last_clean":"2024-05-27T00:15:25.601936+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:15:25.601936+0000","last_undegraded":"2024-05-27T00:15:25.601936+0000","last_fullsized":"2024-05-27T00:15:25.601936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:42:49.298822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.619554+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:15:25.619554+0000","last_peered":"2024-05-27T00:15:25.619554+0000","last_clean":"2024-05-27T00:15:25.619554+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:15:25.619554+0000","last_undegraded":"2024-05-27T00:15:25.619554+0000","last_fullsized":"2024-05-27T00:15:25.619554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.627220+0000","last_change":"2024-05-27T00:15:25.619694+0000","last_active":"2024-05-27T00:15:25.627220+0000","last_peered":"2024-05-27T00:15:25.627220+0000","last_clean":"2024-05-27T00:15:25.627220+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:15:25.627220+0000","last_undegraded":"2024-05-27T00:15:25.627220+0000","last_fullsized":"2024-05-27T00:15:25.627220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:20.567777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636921+0000","last_change":"2024-05-27T00:15:25.627578+0000","last_active":"2024-05-27T00:15:25.636921+0000","last_peered":"2024-05-27T00:15:25.636921+0000","last_clean":"2024-05-27T00:15:25.636921+0000","last_became_active":"2024-05-27T00:15:25.622421+0000","last_became_peered":"2024-05-27T00:15:25.622421+0000","last_unstale":"2024-05-27T00:15:25.636921+0000","last_undegraded":"2024-05-27T00:15:25.636921+0000","last_fullsized":"2024-05-27T00:15:25.636921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.610306+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:15:25.610306+0000","last_peered":"2024-05-27T00:15:25.610306+0000","last_clean":"2024-05-27T00:15:25.610306+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:15:25.610306+0000","last_undegraded":"2024-05-27T00:15:25.610306+0000","last_fullsized":"2024-05-27T00:15:25.610306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636662+0000","last_change":"2024-05-27T00:15:25.623207+0000","last_active":"2024-05-27T00:15:25.636662+0000","last_peered":"2024-05-27T00:15:25.636662+0000","last_clean":"2024-05-27T00:15:25.636662+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:15:25.636662+0000","last_undegraded":"2024-05-27T00:15:25.636662+0000","last_fullsized":"2024-05-27T00:15:25.636662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612125+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:15:25.612125+0000","last_peered":"2024-05-27T00:15:25.612125+0000","last_clean":"2024-05-27T00:15:25.612125+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:15:25.612125+0000","last_undegraded":"2024-05-27T00:15:25.612125+0000","last_fullsized":"2024-05-27T00:15:25.612125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.616696+0000","last_change":"2024-05-27T00:15:25.613018+0000","last_active":"2024-05-27T00:15:25.616696+0000","last_peered":"2024-05-27T00:15:25.616696+0000","last_clean":"2024-05-27T00:15:25.616696+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:15:25.616696+0000","last_undegraded":"2024-05-27T00:15:25.616696+0000","last_fullsized":"2024-05-27T00:15:25.616696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.587240+0000","last_change":"2024-05-27T00:15:24.600571+0000","last_active":"2024-05-27T00:15:25.587240+0000","last_peered":"2024-05-27T00:15:25.587240+0000","last_clean":"2024-05-27T00:15:25.587240+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:15:25.587240+0000","last_undegraded":"2024-05-27T00:15:25.587240+0000","last_fullsized":"2024-05-27T00:15:25.587240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:44:21.645223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.616804+0000","last_change":"2024-05-27T00:15:25.615649+0000","last_active":"2024-05-27T00:15:25.616804+0000","last_peered":"2024-05-27T00:15:25.616804+0000","last_clean":"2024-05-27T00:15:25.616804+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:15:25.616804+0000","last_undegraded":"2024-05-27T00:15:25.616804+0000","last_fullsized":"2024-05-27T00:15:25.616804+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:00:16.995327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.590461+0000","last_change":"2024-05-27T00:15:24.615066+0000","last_active":"2024-05-27T00:15:25.590461+0000","last_peered":"2024-05-27T00:15:25.590461+0000","last_clean":"2024-05-27T00:15:25.590461+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:15:25.590461+0000","last_undegraded":"2024-05-27T00:15:25.590461+0000","last_fullsized":"2024-05-27T00:15:25.590461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":806912,"data_stored":662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:15:30.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:15:30.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.877+0000 7fd15b600640 1 Processor -- start 2024-05-27T00:15:30.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.877+0000 7fd15b600640 1 -- start start 2024-05-27T00:15:30.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.877+0000 7fd15b600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1540cc8f0 0x7fd154070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.877+0000 7fd15b600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd1540c8670 0x7fd1540c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.877+0000 7fd15b600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1540c1000 con 0x7fd1541511d0 2024-05-27T00:15:30.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.877+0000 7fd15b600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd1540b1dd0 con 0x7fd1540c8670 2024-05-27T00:15:30.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.877+0000 7fd15b600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd1540c1a20 con 0x7fd1540cc8f0 2024-05-27T00:15:30.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd158c00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd1541511d0 0x7fd1540c1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43290/0 (socket says 172.21.15.98:43290) 2024-05-27T00:15:30.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd158c00640 1 -- 172.21.15.98:0/3531366436 learned_addr learned my addr 172.21.15.98:0/3531366436 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:30.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd159600640 1 --2- 172.21.15.98:0/3531366436 >> v2:172.21.15.204:3300/0 conn(0x7fd1540c8670 0x7fd1540c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd14be00640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1540cc8f0 0x7fd154070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526897259 0 0) 0x7fd1540c1000 con 0x7fd1541511d0 2024-05-27T00:15:30.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd138003530 con 0x7fd1541511d0 2024-05-27T00:15:30.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1566187783 0 0) 0x7fd138003530 con 0x7fd1541511d0 2024-05-27T00:15:30.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1540cc8f0 msgr2=0x7fd154070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:30.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1540cc8f0 0x7fd154070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 >> v2:172.21.15.204:3300/0 conn(0x7fd1540c8670 msgr2=0x7fd1540c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 --2- 172.21.15.98:0/3531366436 >> v2:172.21.15.204:3300/0 conn(0x7fd1540c8670 0x7fd1540c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1541515b0 con 0x7fd1541511d0 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fd144001a00 con 0x7fd1541511d0 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 >> v1:172.21.15.98:6789/0 conn(0x7fd1541511d0 legacy=0x7fd1540c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1540cc8f0 0x7fd1380039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd138003f90 0x7fd138004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 0x7fd1380051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1540b1dd0 con 0x7fd138004d50 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1540c1a20 con 0x7fd138003f90 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd13800ba80 con 0x7fd1540cc8f0 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd159600640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 0x7fd1380051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd14be00640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd138003f90 0x7fd138004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd158c00640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1540cc8f0 0x7fd1380039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd159600640 1 -- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1540cc8f0 msgr2=0x7fd1380039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:30.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd159600640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1540cc8f0 0x7fd1380039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd159600640 1 -- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd138003f90 msgr2=0x7fd138004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:30.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd159600640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd138003f90 0x7fd138004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.881+0000 7fd159600640 1 -- 172.21.15.98:0/3531366436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd138003530 con 0x7fd138004d50 2024-05-27T00:15:30.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd159600640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 0x7fd1380051d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009ae0 tx=0x7fd14c01cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37002690fa028c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0b2070 con 0x7fd138004d50 2024-05-27T00:15:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd14c0a5590 con 0x7fd138004d50 2024-05-27T00:15:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd153e00640 1 -- 172.21.15.98:0/3531366436 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0096b0 con 0x7fd138004d50 2024-05-27T00:15:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd15b600640 1 -- 172.21.15.98:0/3531366436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 msgr2=0x7fd1380051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd15b600640 1 --2- 172.21.15.98:0/3531366436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 0x7fd1380051d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009ae0 tx=0x7fd14c01cab0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd159600640 1 -- 172.21.15.98:0/3531366436 reap_dead start 2024-05-27T00:15:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd15b600640 1 -- 172.21.15.98:0/3531366436 shutdown_connections 2024-05-27T00:15:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd15b600640 1 -- 172.21.15.98:0/3531366436 >> 172.21.15.98:0/3531366436 conn(0x7fd154067530 msgr2=0x7fd15406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd15b600640 1 -- 172.21.15.98:0/3531366436 shutdown_connections 2024-05-27T00:15:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd15b600640 1 -- 172.21.15.98:0/3531366436 wait complete. 2024-05-27T00:15:30.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd15b600640 1 Processor -- start 2024-05-27T00:15:30.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.885+0000 7fd15b600640 1 -- start start 2024-05-27T00:15:30.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd15b600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 0x7fd154152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd15b600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1541f33c0 0x7fd1540c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd15b600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd154145630 0x7fd154145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd15b600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1540c80e0 con 0x7fd138004d50 2024-05-27T00:15:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd15b600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1540c1da0 con 0x7fd154145630 2024-05-27T00:15:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd15b600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd1540b1a60 con 0x7fd1541f33c0 2024-05-27T00:15:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd158c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 0x7fd154152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd158c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 0x7fd154152cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53540/0 (socket says 172.21.15.98:53540) 2024-05-27T00:15:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd158c00640 1 -- 172.21.15.98:0/324904704 learned_addr learned my addr 172.21.15.98:0/324904704 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd14be00640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1541f33c0 0x7fd1540c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd159600640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd154145630 0x7fd154145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd158c00640 1 -- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1541f33c0 msgr2=0x7fd1540c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd158c00640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1541f33c0 0x7fd1540c23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd158c00640 1 -- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd154145630 msgr2=0x7fd154145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd158c00640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd154145630 0x7fd154145ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd158c00640 1 -- 172.21.15.98:0/324904704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd154145ff0 con 0x7fd138004d50 2024-05-27T00:15:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd14be00640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1541f33c0 0x7fd1540c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd158c00640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 0x7fd154152cd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd144007ae0 tx=0x7fd1440115c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:30.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd151600640 1 -- 172.21.15.98:0/324904704 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14401c070 con 0x7fd138004d50 2024-05-27T00:15:30.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd151600640 1 -- 172.21.15.98:0/324904704 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd144004d70 con 0x7fd138004d50 2024-05-27T00:15:30.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd151600640 1 -- 172.21.15.98:0/324904704 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144010040 con 0x7fd138004d50 2024-05-27T00:15:30.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd15b600640 1 -- 172.21.15.98:0/324904704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1541f0040 con 0x7fd138004d50 2024-05-27T00:15:30.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd15b600640 1 -- 172.21.15.98:0/324904704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1541f02e0 con 0x7fd138004d50 2024-05-27T00:15:30.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd15b600640 1 -- 172.21.15.98:0/324904704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd154075cb0 con 0x7fd138004d50 2024-05-27T00:15:30.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.889+0000 7fd151600640 1 -- 172.21.15.98:0/324904704 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd14401f070 con 0x7fd138004d50 2024-05-27T00:15:30.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.893+0000 7fd151600640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd1300678c0 0x7fd130069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:30.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.893+0000 7fd151600640 1 -- 172.21.15.98:0/324904704 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd144088fd0 con 0x7fd138004d50 2024-05-27T00:15:30.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.893+0000 7fd14be00640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd1300678c0 0x7fd130069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.893+0000 7fd14be00640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd1300678c0 0x7fd130069d80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd154163380 tx=0x7fd13c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:30.897+0000 7fd151600640 1 -- 172.21.15.98:0/324904704 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd1440594b0 con 0x7fd138004d50 2024-05-27T00:15:31.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.045+0000 7fd15b600640 1 -- 172.21.15.98:0/324904704 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd1540713c0 con 0x7fd1300678c0 2024-05-27T00:15:31.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.065+0000 7fd151600640 1 -- 172.21.15.98:0/324904704 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251787 (secure 0 0 0) 0x7fd1540713c0 con 0x7fd1300678c0 2024-05-27T00:15:31.070 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:31.071 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:15:31.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 -- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd1300678c0 msgr2=0x7fd130069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd1300678c0 0x7fd130069d80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd154163380 tx=0x7fd13c008040 comp rx=0 tx=0).stop 2024-05-27T00:15:31.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 -- 172.21.15.98:0/324904704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 msgr2=0x7fd154152cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 0x7fd154152cd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd144007ae0 tx=0x7fd1440115c0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 -- 172.21.15.98:0/324904704 shutdown_connections 2024-05-27T00:15:31.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd1300678c0 0x7fd130069d80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd154145630 0x7fd154145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1541f33c0 0x7fd1540c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 --2- 172.21.15.98:0/324904704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd138004d50 0x7fd154152cd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 -- 172.21.15.98:0/324904704 >> 172.21.15.98:0/324904704 conn(0x7fd154067530 msgr2=0x7fd1540cc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:31.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 -- 172.21.15.98:0/324904704 shutdown_connections 2024-05-27T00:15:31.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.069+0000 7fd15b600640 1 -- 172.21.15.98:0/324904704 wait complete. 2024-05-27T00:15:31.090 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-05-27T00:15:29.133955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219004,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755716,"statfs":{"total":773094113280,"available":772869853184,"internally_reserved":0,"allocated":2891776,"data_stored":1761600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.563533"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644738+0000","last_change":"2024-05-27T00:15:25.640698+0000","last_active":"2024-05-27T00:15:25.644738+0000","last_peered":"2024-05-27T00:15:25.644738+0000","last_clean":"2024-05-27T00:15:25.644738+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:15:25.644738+0000","last_undegraded":"2024-05-27T00:15:25.644738+0000","last_fullsized":"2024-05-27T00:15:25.644738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.600259+0000","last_change":"2024-05-27T00:15:25.598672+0000","last_active":"2024-05-27T00:15:25.600259+0000","last_peered":"2024-05-27T00:15:25.600259+0000","last_clean":"2024-05-27T00:15:25.600259+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:15:25.600259+0000","last_undegraded":"2024-05-27T00:15:25.600259+0000","last_fullsized":"2024-05-27T00:15:25.600259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.594509+0000","last_change":"2024-05-27T00:15:25.593316+0000","last_active":"2024-05-27T00:15:25.594509+0000","last_peered":"2024-05-27T00:15:25.594509+0000","last_clean":"2024-05-27T00:15:25.594509+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:15:25.594509+0000","last_undegraded":"2024-05-27T00:15:25.594509+0000","last_fullsized":"2024-05-27T00:15:25.594509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:27.674418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636581+0000","last_change":"2024-05-27T00:15:25.620237+0000","last_active":"2024-05-27T00:15:25.636581+0000","last_peered":"2024-05-27T00:15:25.636581+0000","last_clean":"2024-05-27T00:15:25.636581+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:15:25.636581+0000","last_undegraded":"2024-05-27T00:15:25.636581+0000","last_fullsized":"2024-05-27T00:15:25.636581+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:20.567777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644713+0000","last_change":"2024-05-27T00:15:25.640146+0000","last_active":"2024-05-27T00:15:25.644713+0000","last_peered":"2024-05-27T00:15:25.644713+0000","last_clean":"2024-05-27T00:15:25.644713+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:15:25.644713+0000","last_undegraded":"2024-05-27T00:15:25.644713+0000","last_fullsized":"2024-05-27T00:15:25.644713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636638+0000","last_change":"2024-05-27T00:15:25.627128+0000","last_active":"2024-05-27T00:15:25.636638+0000","last_peered":"2024-05-27T00:15:25.636638+0000","last_clean":"2024-05-27T00:15:25.636638+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:15:25.636638+0000","last_undegraded":"2024-05-27T00:15:25.636638+0000","last_fullsized":"2024-05-27T00:15:25.636638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.643006+0000","last_change":"2024-05-27T00:15:25.630446+0000","last_active":"2024-05-27T00:15:25.643006+0000","last_peered":"2024-05-27T00:15:25.643006+0000","last_clean":"2024-05-27T00:15:25.643006+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:15:25.643006+0000","last_undegraded":"2024-05-27T00:15:25.643006+0000","last_fullsized":"2024-05-27T00:15:25.643006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.613273+0000","last_change":"2024-05-27T00:15:25.612157+0000","last_active":"2024-05-27T00:15:25.613273+0000","last_peered":"2024-05-27T00:15:25.613273+0000","last_clean":"2024-05-27T00:15:25.613273+0000","last_became_active":"2024-05-27T00:15:25.611884+0000","last_became_peered":"2024-05-27T00:15:25.611884+0000","last_unstale":"2024-05-27T00:15:25.613273+0000","last_undegraded":"2024-05-27T00:15:25.613273+0000","last_fullsized":"2024-05-27T00:15:25.613273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:57.827118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.646283+0000","last_change":"2024-05-27T00:15:25.643312+0000","last_active":"2024-05-27T00:15:25.646283+0000","last_peered":"2024-05-27T00:15:25.646283+0000","last_clean":"2024-05-27T00:15:25.646283+0000","last_became_active":"2024-05-27T00:15:25.642663+0000","last_became_peered":"2024-05-27T00:15:25.642663+0000","last_unstale":"2024-05-27T00:15:25.646283+0000","last_undegraded":"2024-05-27T00:15:25.646283+0000","last_fullsized":"2024-05-27T00:15:25.646283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.637196+0000","last_change":"2024-05-27T00:15:25.629892+0000","last_active":"2024-05-27T00:15:25.637196+0000","last_peered":"2024-05-27T00:15:25.637196+0000","last_clean":"2024-05-27T00:15:25.637196+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:15:25.637196+0000","last_undegraded":"2024-05-27T00:15:25.637196+0000","last_fullsized":"2024-05-27T00:15:25.637196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:34:15.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.621642+0000","last_change":"2024-05-27T00:15:25.614887+0000","last_active":"2024-05-27T00:15:25.621642+0000","last_peered":"2024-05-27T00:15:25.621642+0000","last_clean":"2024-05-27T00:15:25.621642+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:15:25.621642+0000","last_undegraded":"2024-05-27T00:15:25.621642+0000","last_fullsized":"2024-05-27T00:15:25.621642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:18.151532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.630397+0000","last_change":"2024-05-27T00:15:25.623864+0000","last_active":"2024-05-27T00:15:25.630397+0000","last_peered":"2024-05-27T00:15:25.630397+0000","last_clean":"2024-05-27T00:15:25.630397+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:15:25.630397+0000","last_undegraded":"2024-05-27T00:15:25.630397+0000","last_fullsized":"2024-05-27T00:15:25.630397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:27.674418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.616857+0000","last_change":"2024-05-27T00:15:25.615495+0000","last_active":"2024-05-27T00:15:25.616857+0000","last_peered":"2024-05-27T00:15:25.616857+0000","last_clean":"2024-05-27T00:15:25.616857+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:15:25.616857+0000","last_undegraded":"2024-05-27T00:15:25.616857+0000","last_fullsized":"2024-05-27T00:15:25.616857+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:07:03.281228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.596028+0000","last_change":"2024-05-27T00:15:25.594508+0000","last_active":"2024-05-27T00:15:25.596028+0000","last_peered":"2024-05-27T00:15:25.596028+0000","last_clean":"2024-05-27T00:15:25.596028+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:15:25.596028+0000","last_undegraded":"2024-05-27T00:15:25.596028+0000","last_fullsized":"2024-05-27T00:15:25.596028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:53.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612962+0000","last_change":"2024-05-27T00:15:25.611818+0000","last_active":"2024-05-27T00:15:25.612962+0000","last_peered":"2024-05-27T00:15:25.612962+0000","last_clean":"2024-05-27T00:15:25.612962+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:15:25.612962+0000","last_undegraded":"2024-05-27T00:15:25.612962+0000","last_fullsized":"2024-05-27T00:15:25.612962+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.634813+0000","last_change":"2024-05-27T00:15:25.611365+0000","last_active":"2024-05-27T00:15:25.634813+0000","last_peered":"2024-05-27T00:15:25.634813+0000","last_clean":"2024-05-27T00:15:25.634813+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:15:25.634813+0000","last_undegraded":"2024-05-27T00:15:25.634813+0000","last_fullsized":"2024-05-27T00:15:25.634813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.641449+0000","last_change":"2024-05-27T00:15:25.632120+0000","last_active":"2024-05-27T00:15:25.641449+0000","last_peered":"2024-05-27T00:15:25.641449+0000","last_clean":"2024-05-27T00:15:25.641449+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:15:25.641449+0000","last_undegraded":"2024-05-27T00:15:25.641449+0000","last_fullsized":"2024-05-27T00:15:25.641449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.617354+0000","last_change":"2024-05-27T00:15:25.615868+0000","last_active":"2024-05-27T00:15:25.617354+0000","last_peered":"2024-05-27T00:15:25.617354+0000","last_clean":"2024-05-27T00:15:25.617354+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:15:25.617354+0000","last_undegraded":"2024-05-27T00:15:25.617354+0000","last_fullsized":"2024-05-27T00:15:25.617354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:14.969827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.642872+0000","last_change":"2024-05-27T00:15:25.629101+0000","last_active":"2024-05-27T00:15:25.642872+0000","last_peered":"2024-05-27T00:15:25.642872+0000","last_clean":"2024-05-27T00:15:25.642872+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:15:25.642872+0000","last_undegraded":"2024-05-27T00:15:25.642872+0000","last_fullsized":"2024-05-27T00:15:25.642872+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.629018+0000","last_change":"2024-05-27T00:15:25.619351+0000","last_active":"2024-05-27T00:15:25.629018+0000","last_peered":"2024-05-27T00:15:25.629018+0000","last_clean":"2024-05-27T00:15:25.629018+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:15:25.629018+0000","last_undegraded":"2024-05-27T00:15:25.629018+0000","last_fullsized":"2024-05-27T00:15:25.629018+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:19.321535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.617528+0000","last_change":"2024-05-27T00:15:25.616436+0000","last_active":"2024-05-27T00:15:25.617528+0000","last_peered":"2024-05-27T00:15:25.617528+0000","last_clean":"2024-05-27T00:15:25.617528+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:15:25.617528+0000","last_undegraded":"2024-05-27T00:15:25.617528+0000","last_fullsized":"2024-05-27T00:15:25.617528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:20.567777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.641275+0000","last_change":"2024-05-27T00:15:25.629373+0000","last_active":"2024-05-27T00:15:25.641275+0000","last_peered":"2024-05-27T00:15:25.641275+0000","last_clean":"2024-05-27T00:15:25.641275+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:15:25.641275+0000","last_undegraded":"2024-05-27T00:15:25.641275+0000","last_fullsized":"2024-05-27T00:15:25.641275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612525+0000","last_change":"2024-05-27T00:15:25.610707+0000","last_active":"2024-05-27T00:15:25.612525+0000","last_peered":"2024-05-27T00:15:25.612525+0000","last_clean":"2024-05-27T00:15:25.612525+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:15:25.612525+0000","last_undegraded":"2024-05-27T00:15:25.612525+0000","last_fullsized":"2024-05-27T00:15:25.612525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:17:16.223164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.635989+0000","last_change":"2024-05-27T00:15:25.613013+0000","last_active":"2024-05-27T00:15:25.635989+0000","last_peered":"2024-05-27T00:15:25.635989+0000","last_clean":"2024-05-27T00:15:25.635989+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:15:25.635989+0000","last_undegraded":"2024-05-27T00:15:25.635989+0000","last_fullsized":"2024-05-27T00:15:25.635989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612412+0000","last_change":"2024-05-27T00:15:25.610070+0000","last_active":"2024-05-27T00:15:25.612412+0000","last_peered":"2024-05-27T00:15:25.612412+0000","last_clean":"2024-05-27T00:15:25.612412+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:15:25.612412+0000","last_undegraded":"2024-05-27T00:15:25.612412+0000","last_fullsized":"2024-05-27T00:15:25.612412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:50:17.488247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.637249+0000","last_change":"2024-05-27T00:15:25.633470+0000","last_active":"2024-05-27T00:15:25.637249+0000","last_peered":"2024-05-27T00:15:25.637249+0000","last_clean":"2024-05-27T00:15:25.637249+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:15:25.637249+0000","last_undegraded":"2024-05-27T00:15:25.637249+0000","last_fullsized":"2024-05-27T00:15:25.637249+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.598794+0000","last_change":"2024-05-27T00:15:25.597615+0000","last_active":"2024-05-27T00:15:25.598794+0000","last_peered":"2024-05-27T00:15:25.598794+0000","last_clean":"2024-05-27T00:15:25.598794+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:15:25.598794+0000","last_undegraded":"2024-05-27T00:15:25.598794+0000","last_fullsized":"2024-05-27T00:15:25.598794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612682+0000","last_change":"2024-05-27T00:15:25.610933+0000","last_active":"2024-05-27T00:15:25.612682+0000","last_peered":"2024-05-27T00:15:25.612682+0000","last_clean":"2024-05-27T00:15:25.612682+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:15:25.612682+0000","last_undegraded":"2024-05-27T00:15:25.612682+0000","last_fullsized":"2024-05-27T00:15:25.612682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:51:00.406879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.600213+0000","last_change":"2024-05-27T00:15:25.598227+0000","last_active":"2024-05-27T00:15:25.600213+0000","last_peered":"2024-05-27T00:15:25.600213+0000","last_clean":"2024-05-27T00:15:25.600213+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:15:25.600213+0000","last_undegraded":"2024-05-27T00:15:25.600213+0000","last_fullsized":"2024-05-27T00:15:25.600213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:14.969827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636424+0000","last_change":"2024-05-27T00:15:25.620621+0000","last_active":"2024-05-27T00:15:25.636424+0000","last_peered":"2024-05-27T00:15:25.636424+0000","last_clean":"2024-05-27T00:15:25.636424+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:15:25.636424+0000","last_undegraded":"2024-05-27T00:15:25.636424+0000","last_fullsized":"2024-05-27T00:15:25.636424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:34:15.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.606866+0000","last_change":"2024-05-27T00:15:25.605839+0000","last_active":"2024-05-27T00:15:25.606866+0000","last_peered":"2024-05-27T00:15:25.606866+0000","last_clean":"2024-05-27T00:15:25.606866+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:15:25.606866+0000","last_undegraded":"2024-05-27T00:15:25.606866+0000","last_fullsized":"2024-05-27T00:15:25.606866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.624139+0000","last_change":"2024-05-27T00:15:25.607188+0000","last_active":"2024-05-27T00:15:25.624139+0000","last_peered":"2024-05-27T00:15:25.624139+0000","last_clean":"2024-05-27T00:15:25.624139+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:15:25.624139+0000","last_undegraded":"2024-05-27T00:15:25.624139+0000","last_fullsized":"2024-05-27T00:15:25.624139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.629902+0000","last_change":"2024-05-27T00:15:25.615579+0000","last_active":"2024-05-27T00:15:25.629902+0000","last_peered":"2024-05-27T00:15:25.629902+0000","last_clean":"2024-05-27T00:15:25.629902+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:15:25.629902+0000","last_undegraded":"2024-05-27T00:15:25.629902+0000","last_fullsized":"2024-05-27T00:15:25.629902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:56.623917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.588223+0000","last_change":"2024-05-27T00:15:24.611108+0000","last_active":"2024-05-27T00:15:25.588223+0000","last_peered":"2024-05-27T00:15:25.588223+0000","last_clean":"2024-05-27T00:15:25.588223+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:15:25.588223+0000","last_undegraded":"2024-05-27T00:15:25.588223+0000","last_fullsized":"2024-05-27T00:15:25.588223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:11.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.617450+0000","last_change":"2024-05-27T00:15:25.614251+0000","last_active":"2024-05-27T00:15:25.617450+0000","last_peered":"2024-05-27T00:15:25.617450+0000","last_clean":"2024-05-27T00:15:25.617450+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:15:25.617450+0000","last_undegraded":"2024-05-27T00:15:25.617450+0000","last_fullsized":"2024-05-27T00:15:25.617450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:19.321535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.604413+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:15:25.604413+0000","last_peered":"2024-05-27T00:15:25.604413+0000","last_clean":"2024-05-27T00:15:25.604413+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:15:25.604413+0000","last_undegraded":"2024-05-27T00:15:25.604413+0000","last_fullsized":"2024-05-27T00:15:25.604413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.586705+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:15:25.586705+0000","last_peered":"2024-05-27T00:15:25.586705+0000","last_clean":"2024-05-27T00:15:25.586705+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:15:25.586705+0000","last_undegraded":"2024-05-27T00:15:25.586705+0000","last_fullsized":"2024-05-27T00:15:25.586705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:46:08.645927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.598107+0000","last_change":"2024-05-27T00:15:25.596844+0000","last_active":"2024-05-27T00:15:25.598107+0000","last_peered":"2024-05-27T00:15:25.598107+0000","last_clean":"2024-05-27T00:15:25.598107+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:15:25.598107+0000","last_undegraded":"2024-05-27T00:15:25.598107+0000","last_fullsized":"2024-05-27T00:15:25.598107+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:38:49.590477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.595284+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:15:25.595284+0000","last_peered":"2024-05-27T00:15:25.595284+0000","last_clean":"2024-05-27T00:15:25.595284+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:15:25.595284+0000","last_undegraded":"2024-05-27T00:15:25.595284+0000","last_fullsized":"2024-05-27T00:15:25.595284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.594566+0000","last_change":"2024-05-27T00:15:25.593149+0000","last_active":"2024-05-27T00:15:25.594566+0000","last_peered":"2024-05-27T00:15:25.594566+0000","last_clean":"2024-05-27T00:15:25.594566+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:15:25.594566+0000","last_undegraded":"2024-05-27T00:15:25.594566+0000","last_fullsized":"2024-05-27T00:15:25.594566+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:34:15.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.605830+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:15:25.605830+0000","last_peered":"2024-05-27T00:15:25.605830+0000","last_clean":"2024-05-27T00:15:25.605830+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:15:25.605830+0000","last_undegraded":"2024-05-27T00:15:25.605830+0000","last_fullsized":"2024-05-27T00:15:25.605830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636125+0000","last_change":"2024-05-27T00:15:25.618650+0000","last_active":"2024-05-27T00:15:25.636125+0000","last_peered":"2024-05-27T00:15:25.636125+0000","last_clean":"2024-05-27T00:15:25.636125+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:15:25.636125+0000","last_undegraded":"2024-05-27T00:15:25.636125+0000","last_fullsized":"2024-05-27T00:15:25.636125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:18.151532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.605486+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:15:25.605486+0000","last_peered":"2024-05-27T00:15:25.605486+0000","last_clean":"2024-05-27T00:15:25.605486+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:15:25.605486+0000","last_undegraded":"2024-05-27T00:15:25.605486+0000","last_fullsized":"2024-05-27T00:15:25.605486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.616710+0000","last_change":"2024-05-27T00:15:25.613142+0000","last_active":"2024-05-27T00:15:25.616710+0000","last_peered":"2024-05-27T00:15:25.616710+0000","last_clean":"2024-05-27T00:15:25.616710+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:15:25.616710+0000","last_undegraded":"2024-05-27T00:15:25.616710+0000","last_fullsized":"2024-05-27T00:15:25.616710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:18.151532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.582658+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:15:25.582658+0000","last_peered":"2024-05-27T00:15:25.582658+0000","last_clean":"2024-05-27T00:15:25.582658+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:15:25.582658+0000","last_undegraded":"2024-05-27T00:15:25.582658+0000","last_fullsized":"2024-05-27T00:15:25.582658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.630339+0000","last_change":"2024-05-27T00:15:25.622383+0000","last_active":"2024-05-27T00:15:25.630339+0000","last_peered":"2024-05-27T00:15:25.630339+0000","last_clean":"2024-05-27T00:15:25.630339+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:15:25.630339+0000","last_undegraded":"2024-05-27T00:15:25.630339+0000","last_fullsized":"2024-05-27T00:15:25.630339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:24.430219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.605403+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:15:25.605403+0000","last_peered":"2024-05-27T00:15:25.605403+0000","last_clean":"2024-05-27T00:15:25.605403+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:15:25.605403+0000","last_undegraded":"2024-05-27T00:15:25.605403+0000","last_fullsized":"2024-05-27T00:15:25.605403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.642810+0000","last_change":"2024-05-27T00:15:25.628646+0000","last_active":"2024-05-27T00:15:25.642810+0000","last_peered":"2024-05-27T00:15:25.642810+0000","last_clean":"2024-05-27T00:15:25.642810+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:15:25.642810+0000","last_undegraded":"2024-05-27T00:15:25.642810+0000","last_fullsized":"2024-05-27T00:15:25.642810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:34:15.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.590531+0000","last_change":"2024-05-27T00:15:24.605986+0000","last_active":"2024-05-27T00:15:25.590531+0000","last_peered":"2024-05-27T00:15:25.590531+0000","last_clean":"2024-05-27T00:15:25.590531+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:15:25.590531+0000","last_undegraded":"2024-05-27T00:15:25.590531+0000","last_fullsized":"2024-05-27T00:15:25.590531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.625031+0000","last_change":"2024-05-27T00:15:25.624136+0000","last_active":"2024-05-27T00:15:25.625031+0000","last_peered":"2024-05-27T00:15:25.625031+0000","last_clean":"2024-05-27T00:15:25.625031+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:15:25.625031+0000","last_undegraded":"2024-05-27T00:15:25.625031+0000","last_fullsized":"2024-05-27T00:15:25.625031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:34:15.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.613823+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:15:25.613823+0000","last_peered":"2024-05-27T00:15:25.613823+0000","last_clean":"2024-05-27T00:15:25.613823+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:15:25.613823+0000","last_undegraded":"2024-05-27T00:15:25.613823+0000","last_fullsized":"2024-05-27T00:15:25.613823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644760+0000","last_change":"2024-05-27T00:15:25.641965+0000","last_active":"2024-05-27T00:15:25.644760+0000","last_peered":"2024-05-27T00:15:25.644760+0000","last_clean":"2024-05-27T00:15:25.644760+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:15:25.644760+0000","last_undegraded":"2024-05-27T00:15:25.644760+0000","last_fullsized":"2024-05-27T00:15:25.644760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.603619+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:15:25.603619+0000","last_peered":"2024-05-27T00:15:25.603619+0000","last_clean":"2024-05-27T00:15:25.603619+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:15:25.603619+0000","last_undegraded":"2024-05-27T00:15:25.603619+0000","last_fullsized":"2024-05-27T00:15:25.603619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.624196+0000","last_change":"2024-05-27T00:15:25.608549+0000","last_active":"2024-05-27T00:15:25.624196+0000","last_peered":"2024-05-27T00:15:25.624196+0000","last_clean":"2024-05-27T00:15:25.624196+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:15:25.624196+0000","last_undegraded":"2024-05-27T00:15:25.624196+0000","last_fullsized":"2024-05-27T00:15:25.624196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.588573+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:15:25.588573+0000","last_peered":"2024-05-27T00:15:25.588573+0000","last_clean":"2024-05-27T00:15:25.588573+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:15:25.588573+0000","last_undegraded":"2024-05-27T00:15:25.588573+0000","last_fullsized":"2024-05-27T00:15:25.588573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644452+0000","last_change":"2024-05-27T00:15:25.638558+0000","last_active":"2024-05-27T00:15:25.644452+0000","last_peered":"2024-05-27T00:15:25.644452+0000","last_clean":"2024-05-27T00:15:25.644452+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:15:25.644452+0000","last_undegraded":"2024-05-27T00:15:25.644452+0000","last_fullsized":"2024-05-27T00:15:25.644452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.590059+0000","last_change":"2024-05-27T00:15:24.601924+0000","last_active":"2024-05-27T00:15:25.590059+0000","last_peered":"2024-05-27T00:15:25.590059+0000","last_clean":"2024-05-27T00:15:25.590059+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:15:25.590059+0000","last_undegraded":"2024-05-27T00:15:25.590059+0000","last_fullsized":"2024-05-27T00:15:25.590059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:32.185839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644521+0000","last_change":"2024-05-27T00:15:25.639140+0000","last_active":"2024-05-27T00:15:25.644521+0000","last_peered":"2024-05-27T00:15:25.644521+0000","last_clean":"2024-05-27T00:15:25.644521+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:15:25.644521+0000","last_undegraded":"2024-05-27T00:15:25.644521+0000","last_fullsized":"2024-05-27T00:15:25.644521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.588234+0000","last_change":"2024-05-27T00:15:24.600878+0000","last_active":"2024-05-27T00:15:25.588234+0000","last_peered":"2024-05-27T00:15:25.588234+0000","last_clean":"2024-05-27T00:15:25.588234+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:15:25.588234+0000","last_undegraded":"2024-05-27T00:15:25.588234+0000","last_fullsized":"2024-05-27T00:15:25.588234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.648649+0000","last_change":"2024-05-27T00:15:25.644249+0000","last_active":"2024-05-27T00:15:25.648649+0000","last_peered":"2024-05-27T00:15:25.648649+0000","last_clean":"2024-05-27T00:15:25.648649+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:15:25.648649+0000","last_undegraded":"2024-05-27T00:15:25.648649+0000","last_fullsized":"2024-05-27T00:15:25.648649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.582542+0000","last_change":"2024-05-27T00:15:24.603452+0000","last_active":"2024-05-27T00:15:25.582542+0000","last_peered":"2024-05-27T00:15:25.582542+0000","last_clean":"2024-05-27T00:15:25.582542+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:15:25.582542+0000","last_undegraded":"2024-05-27T00:15:25.582542+0000","last_fullsized":"2024-05-27T00:15:25.582542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636229+0000","last_change":"2024-05-27T00:15:25.619341+0000","last_active":"2024-05-27T00:15:25.636229+0000","last_peered":"2024-05-27T00:15:25.636229+0000","last_clean":"2024-05-27T00:15:25.636229+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:15:25.636229+0000","last_undegraded":"2024-05-27T00:15:25.636229+0000","last_fullsized":"2024-05-27T00:15:25.636229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:19.321535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.588785+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:15:25.588785+0000","last_peered":"2024-05-27T00:15:25.588785+0000","last_clean":"2024-05-27T00:15:25.588785+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:15:25.588785+0000","last_undegraded":"2024-05-27T00:15:25.588785+0000","last_fullsized":"2024-05-27T00:15:25.588785+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.644355+0000","last_change":"2024-05-27T00:15:25.639462+0000","last_active":"2024-05-27T00:15:25.644355+0000","last_peered":"2024-05-27T00:15:25.644355+0000","last_clean":"2024-05-27T00:15:25.644355+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:15:25.644355+0000","last_undegraded":"2024-05-27T00:15:25.644355+0000","last_fullsized":"2024-05-27T00:15:25.644355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":168,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.610911+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:15:25.610911+0000","last_peered":"2024-05-27T00:15:25.610911+0000","last_clean":"2024-05-27T00:15:25.610911+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:15:25.610911+0000","last_undegraded":"2024-05-27T00:15:25.610911+0000","last_fullsized":"2024-05-27T00:15:25.610911+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.596096+0000","last_change":"2024-05-27T00:15:24.603781+0000","last_active":"2024-05-27T00:15:25.596096+0000","last_peered":"2024-05-27T00:15:25.596096+0000","last_clean":"2024-05-27T00:15:25.596096+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:15:25.596096+0000","last_undegraded":"2024-05-27T00:15:25.596096+0000","last_fullsized":"2024-05-27T00:15:25.596096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.632804+0000","last_change":"2024-05-27T00:15:25.631978+0000","last_active":"2024-05-27T00:15:25.632804+0000","last_peered":"2024-05-27T00:15:25.632804+0000","last_clean":"2024-05-27T00:15:25.632804+0000","last_became_active":"2024-05-27T00:15:25.631697+0000","last_became_peered":"2024-05-27T00:15:25.631697+0000","last_unstale":"2024-05-27T00:15:25.632804+0000","last_undegraded":"2024-05-27T00:15:25.632804+0000","last_fullsized":"2024-05-27T00:15:25.632804+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:20:40.605562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.610777+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:15:25.610777+0000","last_peered":"2024-05-27T00:15:25.610777+0000","last_clean":"2024-05-27T00:15:25.610777+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:15:25.610777+0000","last_undegraded":"2024-05-27T00:15:25.610777+0000","last_fullsized":"2024-05-27T00:15:25.610777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612161+0000","last_change":"2024-05-27T00:15:25.610627+0000","last_active":"2024-05-27T00:15:25.612161+0000","last_peered":"2024-05-27T00:15:25.612161+0000","last_clean":"2024-05-27T00:15:25.612161+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:15:25.612161+0000","last_undegraded":"2024-05-27T00:15:25.612161+0000","last_fullsized":"2024-05-27T00:15:25.612161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.588094+0000","last_change":"2024-05-27T00:15:24.598182+0000","last_active":"2024-05-27T00:15:25.588094+0000","last_peered":"2024-05-27T00:15:25.588094+0000","last_clean":"2024-05-27T00:15:25.588094+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:15:25.588094+0000","last_undegraded":"2024-05-27T00:15:25.588094+0000","last_fullsized":"2024-05-27T00:15:25.588094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.594401+0000","last_change":"2024-05-27T00:15:25.591449+0000","last_active":"2024-05-27T00:15:25.594401+0000","last_peered":"2024-05-27T00:15:25.594401+0000","last_clean":"2024-05-27T00:15:25.594401+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:15:25.594401+0000","last_undegraded":"2024-05-27T00:15:25.594401+0000","last_fullsized":"2024-05-27T00:15:25.594401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:20.567777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.611803+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:15:25.611803+0000","last_peered":"2024-05-27T00:15:25.611803+0000","last_clean":"2024-05-27T00:15:25.611803+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:15:25.611803+0000","last_undegraded":"2024-05-27T00:15:25.611803+0000","last_fullsized":"2024-05-27T00:15:25.611803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.630803+0000","last_change":"2024-05-27T00:15:25.614073+0000","last_active":"2024-05-27T00:15:25.630803+0000","last_peered":"2024-05-27T00:15:25.630803+0000","last_clean":"2024-05-27T00:15:25.630803+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:15:25.630803+0000","last_undegraded":"2024-05-27T00:15:25.630803+0000","last_fullsized":"2024-05-27T00:15:25.630803+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:17:28.989259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.581600+0000","last_change":"2024-05-27T00:15:24.603518+0000","last_active":"2024-05-27T00:15:25.581600+0000","last_peered":"2024-05-27T00:15:25.581600+0000","last_clean":"2024-05-27T00:15:25.581600+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:15:25.581600+0000","last_undegraded":"2024-05-27T00:15:25.581600+0000","last_fullsized":"2024-05-27T00:15:25.581600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636656+0000","last_change":"2024-05-27T00:15:25.621519+0000","last_active":"2024-05-27T00:15:25.636656+0000","last_peered":"2024-05-27T00:15:25.636656+0000","last_clean":"2024-05-27T00:15:25.636656+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:15:25.636656+0000","last_undegraded":"2024-05-27T00:15:25.636656+0000","last_fullsized":"2024-05-27T00:15:25.636656+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.617116+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:15:25.617116+0000","last_peered":"2024-05-27T00:15:25.617116+0000","last_clean":"2024-05-27T00:15:25.617116+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:15:25.617116+0000","last_undegraded":"2024-05-27T00:15:25.617116+0000","last_fullsized":"2024-05-27T00:15:25.617116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.597110+0000","last_change":"2024-05-27T00:15:25.593180+0000","last_active":"2024-05-27T00:15:25.597110+0000","last_peered":"2024-05-27T00:15:25.597110+0000","last_clean":"2024-05-27T00:15:25.597110+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:15:25.597110+0000","last_undegraded":"2024-05-27T00:15:25.597110+0000","last_fullsized":"2024-05-27T00:15:25.597110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.582108+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:15:25.582108+0000","last_peered":"2024-05-27T00:15:25.582108+0000","last_clean":"2024-05-27T00:15:25.582108+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:15:25.582108+0000","last_undegraded":"2024-05-27T00:15:25.582108+0000","last_fullsized":"2024-05-27T00:15:25.582108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.632794+0000","last_change":"2024-05-27T00:15:25.631919+0000","last_active":"2024-05-27T00:15:25.632794+0000","last_peered":"2024-05-27T00:15:25.632794+0000","last_clean":"2024-05-27T00:15:25.632794+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:15:25.632794+0000","last_undegraded":"2024-05-27T00:15:25.632794+0000","last_fullsized":"2024-05-27T00:15:25.632794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:14:01.167988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.606237+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:15:25.606237+0000","last_peered":"2024-05-27T00:15:25.606237+0000","last_clean":"2024-05-27T00:15:25.606237+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:15:25.606237+0000","last_undegraded":"2024-05-27T00:15:25.606237+0000","last_fullsized":"2024-05-27T00:15:25.606237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.597970+0000","last_change":"2024-05-27T00:15:25.596792+0000","last_active":"2024-05-27T00:15:25.597970+0000","last_peered":"2024-05-27T00:15:25.597970+0000","last_clean":"2024-05-27T00:15:25.597970+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:15:25.597970+0000","last_undegraded":"2024-05-27T00:15:25.597970+0000","last_fullsized":"2024-05-27T00:15:25.597970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.610393+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:15:25.610393+0000","last_peered":"2024-05-27T00:15:25.610393+0000","last_clean":"2024-05-27T00:15:25.610393+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:15:25.610393+0000","last_undegraded":"2024-05-27T00:15:25.610393+0000","last_fullsized":"2024-05-27T00:15:25.610393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.616145+0000","last_change":"2024-05-27T00:15:25.610697+0000","last_active":"2024-05-27T00:15:25.616145+0000","last_peered":"2024-05-27T00:15:25.616145+0000","last_clean":"2024-05-27T00:15:25.616145+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:15:25.616145+0000","last_undegraded":"2024-05-27T00:15:25.616145+0000","last_fullsized":"2024-05-27T00:15:25.616145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:40:21.445109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.582789+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:15:25.582789+0000","last_peered":"2024-05-27T00:15:25.582789+0000","last_clean":"2024-05-27T00:15:25.582789+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:15:25.582789+0000","last_undegraded":"2024-05-27T00:15:25.582789+0000","last_fullsized":"2024-05-27T00:15:25.582789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.599907+0000","last_change":"2024-05-27T00:15:25.597164+0000","last_active":"2024-05-27T00:15:25.599907+0000","last_peered":"2024-05-27T00:15:25.599907+0000","last_clean":"2024-05-27T00:15:25.599907+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:15:25.599907+0000","last_undegraded":"2024-05-27T00:15:25.599907+0000","last_fullsized":"2024-05-27T00:15:25.599907+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:06:14.025980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.581442+0000","last_change":"2024-05-27T00:15:24.598238+0000","last_active":"2024-05-27T00:15:25.581442+0000","last_peered":"2024-05-27T00:15:25.581442+0000","last_clean":"2024-05-27T00:15:25.581442+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:15:25.581442+0000","last_undegraded":"2024-05-27T00:15:25.581442+0000","last_fullsized":"2024-05-27T00:15:25.581442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.601936+0000","last_change":"2024-05-27T00:15:25.601140+0000","last_active":"2024-05-27T00:15:25.601936+0000","last_peered":"2024-05-27T00:15:25.601936+0000","last_clean":"2024-05-27T00:15:25.601936+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:15:25.601936+0000","last_undegraded":"2024-05-27T00:15:25.601936+0000","last_fullsized":"2024-05-27T00:15:25.601936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:42:49.298822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.619554+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:15:25.619554+0000","last_peered":"2024-05-27T00:15:25.619554+0000","last_clean":"2024-05-27T00:15:25.619554+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:15:25.619554+0000","last_undegraded":"2024-05-27T00:15:25.619554+0000","last_fullsized":"2024-05-27T00:15:25.619554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.627220+0000","last_change":"2024-05-27T00:15:25.619694+0000","last_active":"2024-05-27T00:15:25.627220+0000","last_peered":"2024-05-27T00:15:25.627220+0000","last_clean":"2024-05-27T00:15:25.627220+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:15:25.627220+0000","last_undegraded":"2024-05-27T00:15:25.627220+0000","last_fullsized":"2024-05-27T00:15:25.627220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:20.567777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636921+0000","last_change":"2024-05-27T00:15:25.627578+0000","last_active":"2024-05-27T00:15:25.636921+0000","last_peered":"2024-05-27T00:15:25.636921+0000","last_clean":"2024-05-27T00:15:25.636921+0000","last_became_active":"2024-05-27T00:15:25.622421+0000","last_became_peered":"2024-05-27T00:15:25.622421+0000","last_unstale":"2024-05-27T00:15:25.636921+0000","last_undegraded":"2024-05-27T00:15:25.636921+0000","last_fullsized":"2024-05-27T00:15:25.636921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.610306+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:15:25.610306+0000","last_peered":"2024-05-27T00:15:25.610306+0000","last_clean":"2024-05-27T00:15:25.610306+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:15:25.610306+0000","last_undegraded":"2024-05-27T00:15:25.610306+0000","last_fullsized":"2024-05-27T00:15:25.610306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.636662+0000","last_change":"2024-05-27T00:15:25.623207+0000","last_active":"2024-05-27T00:15:25.636662+0000","last_peered":"2024-05-27T00:15:25.636662+0000","last_clean":"2024-05-27T00:15:25.636662+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:15:25.636662+0000","last_undegraded":"2024-05-27T00:15:25.636662+0000","last_fullsized":"2024-05-27T00:15:25.636662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.612125+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:15:25.612125+0000","last_peered":"2024-05-27T00:15:25.612125+0000","last_clean":"2024-05-27T00:15:25.612125+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:15:25.612125+0000","last_undegraded":"2024-05-27T00:15:25.612125+0000","last_fullsized":"2024-05-27T00:15:25.612125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.616696+0000","last_change":"2024-05-27T00:15:25.613018+0000","last_active":"2024-05-27T00:15:25.616696+0000","last_peered":"2024-05-27T00:15:25.616696+0000","last_clean":"2024-05-27T00:15:25.616696+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:15:25.616696+0000","last_undegraded":"2024-05-27T00:15:25.616696+0000","last_fullsized":"2024-05-27T00:15:25.616696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.587240+0000","last_change":"2024-05-27T00:15:24.600571+0000","last_active":"2024-05-27T00:15:25.587240+0000","last_peered":"2024-05-27T00:15:25.587240+0000","last_clean":"2024-05-27T00:15:25.587240+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:15:25.587240+0000","last_undegraded":"2024-05-27T00:15:25.587240+0000","last_fullsized":"2024-05-27T00:15:25.587240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:44:21.645223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.616804+0000","last_change":"2024-05-27T00:15:25.615649+0000","last_active":"2024-05-27T00:15:25.616804+0000","last_peered":"2024-05-27T00:15:25.616804+0000","last_clean":"2024-05-27T00:15:25.616804+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:15:25.616804+0000","last_undegraded":"2024-05-27T00:15:25.616804+0000","last_fullsized":"2024-05-27T00:15:25.616804+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:00:16.995327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T00:15:25.590461+0000","last_change":"2024-05-27T00:15:24.615066+0000","last_active":"2024-05-27T00:15:25.590461+0000","last_peered":"2024-05-27T00:15:25.590461+0000","last_clean":"2024-05-27T00:15:25.590461+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:15:25.590461+0000","last_undegraded":"2024-05-27T00:15:25.590461+0000","last_fullsized":"2024-05-27T00:15:25.590461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":806912,"data_stored":662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:15:31.094 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-27T00:15:31.094 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-27T00:15:31.094 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-27T00:15:31.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-27T00:15:31.095 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-27T00:15:31.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.265+0000 7fc22b400640 1 Processor -- start 2024-05-27T00:15:31.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.265+0000 7fc22b400640 1 -- start start 2024-05-27T00:15:31.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.265+0000 7fc22b400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2241511d0 0x7fc224070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.265+0000 7fc22b400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2240c1000 con 0x7fc2240cc8f0 2024-05-27T00:15:31.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.265+0000 7fc22b400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2241515b0 con 0x7fc2240c8670 2024-05-27T00:15:31.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.265+0000 7fc22b400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc2240b1dd0 con 0x7fc2241511d0 2024-05-27T00:15:31.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc228a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2241511d0 0x7fc224070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc228a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2241511d0 0x7fc224070e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53552/0 (socket says 172.21.15.98:53552) 2024-05-27T00:15:31.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc21be00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc2240cc8f0 0x7fc2240c1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43294/0 (socket says 172.21.15.98:43294) 2024-05-27T00:15:31.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc228a00640 1 -- 172.21.15.98:0/2205415465 learned_addr learned my addr 172.21.15.98:0/2205415465 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:31.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc228a00640 1 -- 172.21.15.98:0/2205415465 >> v1:172.21.15.204:6789/0 conn(0x7fc2240c8670 legacy=0x7fc2241617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc228a00640 1 -- 172.21.15.98:0/2205415465 >> v1:172.21.15.98:6789/0 conn(0x7fc2240cc8f0 legacy=0x7fc2240c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc228a00640 1 -- 172.21.15.98:0/2205415465 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2240c83c0 con 0x7fc2241511d0 2024-05-27T00:15:31.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc223e00640 1 -- 172.21.15.98:0/2205415465 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3757687837 0 0) 0x7fc2240c1000 con 0x7fc2240cc8f0 2024-05-27T00:15:31.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc228a00640 1 --2- 172.21.15.98:0/2205415465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2241511d0 0x7fc224070e10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc21400aa10 tx=0x7fc21401d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0ae6f243574741 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:31.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc223e00640 1 -- 172.21.15.98:0/2205415465 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21400b040 con 0x7fc2241511d0 2024-05-27T00:15:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc223e00640 1 -- 172.21.15.98:0/2205415465 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2140a7700 con 0x7fc2241511d0 2024-05-27T00:15:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.269+0000 7fc223e00640 1 -- 172.21.15.98:0/2205415465 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2140a7ad0 con 0x7fc2241511d0 2024-05-27T00:15:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 -- 172.21.15.98:0/2205415465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2241511d0 msgr2=0x7fc224070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 --2- 172.21.15.98:0/2205415465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2241511d0 0x7fc224070e10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc21400aa10 tx=0x7fc21401d8a0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 -- 172.21.15.98:0/2205415465 shutdown_connections 2024-05-27T00:15:31.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 --2- 172.21.15.98:0/2205415465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2241511d0 0x7fc224070e10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 -- 172.21.15.98:0/2205415465 >> 172.21.15.98:0/2205415465 conn(0x7fc224067530 msgr2=0x7fc22406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:31.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 -- 172.21.15.98:0/2205415465 shutdown_connections 2024-05-27T00:15:31.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 -- 172.21.15.98:0/2205415465 wait complete. 2024-05-27T00:15:31.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 Processor -- start 2024-05-27T00:15:31.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 -- start start 2024-05-27T00:15:31.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2240c8670 0x7fc2241416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2240cc8f0 0x7fc2241f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2241511d0 0x7fc22413bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2240c8e90 con 0x7fc2240cc8f0 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc2240c1a20 con 0x7fc2240c8670 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc22b400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc2240c2150 con 0x7fc2241511d0 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc21be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2240cc8f0 0x7fc2241f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc21be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2240cc8f0 0x7fc2241f55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53558/0 (socket says 172.21.15.98:53558) 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc21be00640 1 -- 172.21.15.98:0/942204621 learned_addr learned my addr 172.21.15.98:0/942204621 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc228a00640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2240c8670 0x7fc2241416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc229400640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2241511d0 0x7fc22413bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc21be00640 1 -- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2241511d0 msgr2=0x7fc22413bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc21be00640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2241511d0 0x7fc22413bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc21be00640 1 -- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2240c8670 msgr2=0x7fc2241416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc21be00640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2240c8670 0x7fc2241416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc21be00640 1 -- 172.21.15.98:0/942204621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc224141270 con 0x7fc2240cc8f0 2024-05-27T00:15:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.273+0000 7fc21be00640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2240cc8f0 0x7fc2241f55a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00c4a0 tx=0x7fc20c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.277+0000 7fc221600640 1 -- 172.21.15.98:0/942204621 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c018070 con 0x7fc2240cc8f0 2024-05-27T00:15:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.277+0000 7fc221600640 1 -- 172.21.15.98:0/942204621 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc20c0074e0 con 0x7fc2240cc8f0 2024-05-27T00:15:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.277+0000 7fc221600640 1 -- 172.21.15.98:0/942204621 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c004a30 con 0x7fc2240cc8f0 2024-05-27T00:15:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.277+0000 7fc22b400640 1 -- 172.21.15.98:0/942204621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc224140f90 con 0x7fc2240cc8f0 2024-05-27T00:15:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.277+0000 7fc22b400640 1 -- 172.21.15.98:0/942204621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22413b440 con 0x7fc2240cc8f0 2024-05-27T00:15:31.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.277+0000 7fc22b400640 1 -- 172.21.15.98:0/942204621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc224075cb0 con 0x7fc2240cc8f0 2024-05-27T00:15:31.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.277+0000 7fc221600640 1 -- 172.21.15.98:0/942204621 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc20c004bd0 con 0x7fc2240cc8f0 2024-05-27T00:15:31.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.277+0000 7fc221600640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2000678c0 0x7fc200069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.277+0000 7fc221600640 1 -- 172.21.15.98:0/942204621 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc20c0850a0 con 0x7fc2240cc8f0 2024-05-27T00:15:31.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.277+0000 7fc228a00640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2000678c0 0x7fc200069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.281+0000 7fc228a00640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2000678c0 0x7fc200069d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2140b0000 tx=0x7fc214008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:31.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.281+0000 7fc221600640 1 -- 172.21.15.98:0/942204621 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc20c055b40 con 0x7fc2240cc8f0 2024-05-27T00:15:31.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.473+0000 7fc22b400640 1 -- 172.21.15.98:0/942204621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc2240713c0 con 0x7fc2240cc8f0 2024-05-27T00:15:31.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.477+0000 7fc221600640 1 -- 172.21.15.98:0/942204621 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+166894 (secure 0 0 0) 0x7fc20c055440 con 0x7fc2240cc8f0 2024-05-27T00:15:31.482 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 -- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2000678c0 msgr2=0x7fc200069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2000678c0 0x7fc200069d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2140b0000 tx=0x7fc214008040 comp rx=0 tx=0).stop 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 -- 172.21.15.98:0/942204621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2240cc8f0 msgr2=0x7fc2241f55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2240cc8f0 0x7fc2241f55a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00c4a0 tx=0x7fc20c00c960 comp rx=0 tx=0).stop 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 -- 172.21.15.98:0/942204621 shutdown_connections 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2241511d0 0x7fc22413bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2240cc8f0 0x7fc2241f55a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2000678c0 0x7fc200069d80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 --2- 172.21.15.98:0/942204621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2240c8670 0x7fc2241416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 -- 172.21.15.98:0/942204621 >> 172.21.15.98:0/942204621 conn(0x7fc224067530 msgr2=0x7fc22414f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 -- 172.21.15.98:0/942204621 shutdown_connections 2024-05-27T00:15:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.481+0000 7fc22b400640 1 -- 172.21.15.98:0/942204621 wait complete. 2024-05-27T00:15:31.503 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3774298058},{"type":"v1","addr":"172.21.15.204:6833","nonce":3774298058}]},"active_addr":"172.21.15.204:6833/3774298058","active_change":"2024-05-27T00:15:05.119828+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":1159881367}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":2340538838}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":2588298997}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":2771577836}]}]} 2024-05-27T00:15:31.506 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-27T00:15:31.507 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-27T00:15:31.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:31.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.641+0000 7f17c3e00640 1 Processor -- start 2024-05-27T00:15:31.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.641+0000 7f17c3e00640 1 -- start start 2024-05-27T00:15:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc0735d0 0x7f17bc151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c3e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17bc0c17d0 con 0x7f17bc0c79f0 2024-05-27T00:15:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c3e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17bc070e10 con 0x7f17bc0731f0 2024-05-27T00:15:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f17bc071990 con 0x7f17bc0735d0 2024-05-27T00:15:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c1e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f17bc0c79f0 0x7f17bc157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43312/0 (socket says 172.21.15.98:43312) 2024-05-27T00:15:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c1e00640 1 -- 172.21.15.98:0/133877761 learned_addr learned my addr 172.21.15.98:0/133877761 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:31.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c0a00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc0735d0 0x7f17bc151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2318240269 0 0) 0x7f17bc0c17d0 con 0x7f17bc0c79f0 2024-05-27T00:15:31.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17a4003530 con 0x7f17bc0c79f0 2024-05-27T00:15:31.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1910788116 0 0) 0x7f17bc070e10 con 0x7f17bc0731f0 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17bc0c17d0 con 0x7f17bc0731f0 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2676596580 0 0) 0x7f17a4003530 con 0x7f17bc0c79f0 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc0735d0 msgr2=0x7f17bc151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc0735d0 0x7f17bc151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 >> v1:172.21.15.204:6789/0 conn(0x7f17bc0731f0 legacy=0x7f17bc0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17bc0710f0 con 0x7f17bc0c79f0 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f17b4001680 con 0x7f17bc0c79f0 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 >> v1:172.21.15.98:6789/0 conn(0x7f17bc0c79f0 legacy=0x7f17bc157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc0735d0 0x7f17a4003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17a40040e0 0x7f17a4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a4004d50 0x7f17a40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc071990 con 0x7f17a40040e0 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17a400dcc0 con 0x7f17a4004d50 2024-05-27T00:15:31.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f17a400dec0 con 0x7f17bc0735d0 2024-05-27T00:15:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c0a00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17a40040e0 0x7f17a4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c1e00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a4004d50 0x7f17a40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c1400640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc0735d0 0x7f17a4003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c0a00640 1 -- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc0735d0 msgr2=0x7f17a4003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c0a00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc0735d0 0x7f17a4003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c0a00640 1 -- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a4004d50 msgr2=0x7f17a40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c0a00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a4004d50 0x7f17a40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c0a00640 1 -- 172.21.15.98:0/133877761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a4003530 con 0x7f17a40040e0 2024-05-27T00:15:31.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.645+0000 7f17c1e00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a4004d50 0x7f17a40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:31.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17c0a00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17a40040e0 0x7f17a4004810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009ab0 tx=0x7f17a801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46251cf1448aff36 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:31.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a80b2070 con 0x7f17a40040e0 2024-05-27T00:15:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a80a5530 con 0x7f17a40040e0 2024-05-27T00:15:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17bbe00640 1 -- 172.21.15.98:0/133877761 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a80096b0 con 0x7f17a40040e0 2024-05-27T00:15:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17c3e00640 1 -- 172.21.15.98:0/133877761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17a40040e0 msgr2=0x7f17a4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17c3e00640 1 --2- 172.21.15.98:0/133877761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17a40040e0 0x7f17a4004810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009ab0 tx=0x7f17a801ca50 comp rx=0 tx=0).stop 2024-05-27T00:15:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17c1e00640 1 -- 172.21.15.98:0/133877761 reap_dead start 2024-05-27T00:15:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17c3e00640 1 -- 172.21.15.98:0/133877761 shutdown_connections 2024-05-27T00:15:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17c3e00640 1 -- 172.21.15.98:0/133877761 >> 172.21.15.98:0/133877761 conn(0x7f17bc067530 msgr2=0x7f17bc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17c3e00640 1 -- 172.21.15.98:0/133877761 shutdown_connections 2024-05-27T00:15:31.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.649+0000 7f17c3e00640 1 -- 172.21.15.98:0/133877761 wait complete. 2024-05-27T00:15:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 Processor -- start 2024-05-27T00:15:31.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 -- start start 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a40040e0 0x7f17bc141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc147500 0x7f17bc0cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17bc144fb0 0x7f17bc0cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc0c8400 con 0x7f17bc144fb0 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc0716b0 con 0x7f17a40040e0 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f17bc158080 con 0x7f17bc147500 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c1e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17bc144fb0 0x7f17bc0cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c1e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17bc144fb0 0x7f17bc0cb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53564/0 (socket says 172.21.15.98:53564) 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c1e00640 1 -- 172.21.15.98:0/2580250398 learned_addr learned my addr 172.21.15.98:0/2580250398 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c1400640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a40040e0 0x7f17bc141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c0a00640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc147500 0x7f17bc0cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c1e00640 1 -- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc147500 msgr2=0x7f17bc0cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c1e00640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc147500 0x7f17bc0cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c1e00640 1 -- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a40040e0 msgr2=0x7f17bc141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c1e00640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a40040e0 0x7f17bc141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c1e00640 1 -- 172.21.15.98:0/2580250398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17bc14dcd0 con 0x7f17bc144fb0 2024-05-27T00:15:31.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c1e00640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17bc144fb0 0x7f17bc0cb100 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0c81f0 tx=0x7f17b4011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17b9600640 1 -- 172.21.15.98:0/2580250398 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4010040 con 0x7f17bc144fb0 2024-05-27T00:15:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17b9600640 1 -- 172.21.15.98:0/2580250398 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17b401a070 con 0x7f17bc144fb0 2024-05-27T00:15:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17b9600640 1 -- 172.21.15.98:0/2580250398 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b400fb20 con 0x7f17bc144fb0 2024-05-27T00:15:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 -- 172.21.15.98:0/2580250398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17bc155630 con 0x7f17bc144fb0 2024-05-27T00:15:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 -- 172.21.15.98:0/2580250398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17bc1f0ee0 con 0x7f17bc144fb0 2024-05-27T00:15:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.653+0000 7f17c3e00640 1 -- 172.21.15.98:0/2580250398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17bc075cb0 con 0x7f17bc144fb0 2024-05-27T00:15:31.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.657+0000 7f17b9600640 1 -- 172.21.15.98:0/2580250398 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f17b400fcc0 con 0x7f17bc144fb0 2024-05-27T00:15:31.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.657+0000 7f17b9600640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678f0 0x7f179c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.657+0000 7f17b9600640 1 -- 172.21.15.98:0/2580250398 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f17b4088b20 con 0x7f17bc144fb0 2024-05-27T00:15:31.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.657+0000 7f17c1400640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678f0 0x7f179c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.657+0000 7f17c1400640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678f0 0x7f179c069db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f17b0007e70 tx=0x7f17b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:31.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.661+0000 7f17b9600640 1 -- 172.21.15.98:0/2580250398 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f17b40595c0 con 0x7f17bc144fb0 2024-05-27T00:15:31.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.817+0000 7f17c3e00640 1 -- 172.21.15.98:0/2580250398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f17bc147ec0 con 0x7f17bc144fb0 2024-05-27T00:15:31.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.817+0000 7f17b9600640 1 -- 172.21.15.98:0/2580250398 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16581 (secure 0 0 0) 0x7f17b4058ec0 con 0x7f17bc144fb0 2024-05-27T00:15:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:31.822 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:26.579229+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 -- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678f0 msgr2=0x7f179c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678f0 0x7f179c069db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f17b0007e70 tx=0x7f17b0009040 comp rx=0 tx=0).stop 2024-05-27T00:15:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 -- 172.21.15.98:0/2580250398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17bc144fb0 msgr2=0x7f17bc0cb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17bc144fb0 0x7f17bc0cb100 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0c81f0 tx=0x7f17b4011c20 comp rx=0 tx=0).stop 2024-05-27T00:15:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 -- 172.21.15.98:0/2580250398 shutdown_connections 2024-05-27T00:15:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17bc144fb0 0x7f17bc0cb100 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17bc147500 0x7f17bc0cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f179c0678f0 0x7f179c069db0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 --2- 172.21.15.98:0/2580250398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17a40040e0 0x7f17bc141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 -- 172.21.15.98:0/2580250398 >> 172.21.15.98:0/2580250398 conn(0x7f17bc067530 msgr2=0x7f17bc14e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:31.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 -- 172.21.15.98:0/2580250398 shutdown_connections 2024-05-27T00:15:31.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.821+0000 7f17c3e00640 1 -- 172.21.15.98:0/2580250398 wait complete. 2024-05-27T00:15:31.843 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-27T00:15:31.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f011a200640 1 Processor -- start 2024-05-27T00:15:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f011a200640 1 -- start start 2024-05-27T00:15:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f011a200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f01140b3260 0x7f01140c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f011a200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0114070660 0x7f01140c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f011a200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01140c36c0 0x7f01140c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f011a200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0114071010 con 0x7f0114070660 2024-05-27T00:15:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f011a200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f01140c88c0 con 0x7f01140b3260 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f011a200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f01140c8b80 con 0x7f01140c36c0 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0113e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0114070660 0x7f01140c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0113e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0114070660 0x7f01140c3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53568/0 (socket says 172.21.15.98:53568) 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0113e00640 1 -- 172.21.15.98:0/498165954 learned_addr learned my addr 172.21.15.98:0/498165954 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0118e00640 1 --2- 172.21.15.98:0/498165954 >> v2:172.21.15.204:3300/0 conn(0x7f01140b3260 0x7f01140c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0119800640 1 --2- 172.21.15.98:0/498165954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01140c36c0 0x7f01140c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0113e00640 1 -- 172.21.15.98:0/498165954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01140c36c0 msgr2=0x7f01140c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0113e00640 1 --2- 172.21.15.98:0/498165954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01140c36c0 0x7f01140c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0113e00640 1 -- 172.21.15.98:0/498165954 >> v2:172.21.15.204:3300/0 conn(0x7f01140b3260 msgr2=0x7f01140c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0113e00640 1 --2- 172.21.15.98:0/498165954 >> v2:172.21.15.204:3300/0 conn(0x7f01140b3260 0x7f01140c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0113e00640 1 -- 172.21.15.98:0/498165954 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01140c8e60 con 0x7f0114070660 2024-05-27T00:15:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.977+0000 7f0118e00640 1 --2- 172.21.15.98:0/498165954 >> v2:172.21.15.204:3300/0 conn(0x7f01140b3260 0x7f01140c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:31.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f0113e00640 1 --2- 172.21.15.98:0/498165954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0114070660 0x7f01140c3180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f01140cedc0 tx=0x7f010001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99eed771b74d4eb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:31.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f0113400640 1 -- 172.21.15.98:0/498165954 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01000b5070 con 0x7f0114070660 2024-05-27T00:15:31.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f0113400640 1 -- 172.21.15.98:0/498165954 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01000a89d0 con 0x7f0114070660 2024-05-27T00:15:31.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f0113400640 1 -- 172.21.15.98:0/498165954 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01000a8da0 con 0x7f0114070660 2024-05-27T00:15:31.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f011a200640 1 -- 172.21.15.98:0/498165954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0114070660 msgr2=0x7f01140c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f011a200640 1 --2- 172.21.15.98:0/498165954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0114070660 0x7f01140c3180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f01140cedc0 tx=0x7f010001ed10 comp rx=0 tx=0).stop 2024-05-27T00:15:31.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f011a200640 1 -- 172.21.15.98:0/498165954 shutdown_connections 2024-05-27T00:15:31.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f011a200640 1 --2- 172.21.15.98:0/498165954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01140c36c0 0x7f01140c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f011a200640 1 --2- 172.21.15.98:0/498165954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0114070660 0x7f01140c3180 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f011a200640 1 --2- 172.21.15.98:0/498165954 >> v2:172.21.15.204:3300/0 conn(0x7f01140b3260 0x7f01140c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f011a200640 1 -- 172.21.15.98:0/498165954 >> 172.21.15.98:0/498165954 conn(0x7f01140cdeb0 msgr2=0x7f01140ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:31.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f011a200640 1 -- 172.21.15.98:0/498165954 shutdown_connections 2024-05-27T00:15:31.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f011a200640 1 -- 172.21.15.98:0/498165954 wait complete. 2024-05-27T00:15:31.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.981+0000 7f011a200640 1 Processor -- start 2024-05-27T00:15:31.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f011a200640 1 -- start start 2024-05-27T00:15:31.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f011a200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0114070660 0x7f01140b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f011a200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01140c36c0 0x7f01141decb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f011a200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01140cc180 0x7f01141de160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f011a200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01140c8200 con 0x7f01140cc180 2024-05-27T00:15:31.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f011a200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f01140c9420 con 0x7f01140c36c0 2024-05-27T00:15:31.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f011a200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f01140c9940 con 0x7f0114070660 2024-05-27T00:15:31.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0119800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01140cc180 0x7f01141de160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0119800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01140cc180 0x7f01141de160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53574/0 (socket says 172.21.15.98:53574) 2024-05-27T00:15:31.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0119800640 1 -- 172.21.15.98:0/3056214512 learned_addr learned my addr 172.21.15.98:0/3056214512 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:31.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0118e00640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0114070660 0x7f01140b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0113e00640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01140c36c0 0x7f01141decb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0119800640 1 -- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0114070660 msgr2=0x7f01140b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0119800640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0114070660 0x7f01140b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0119800640 1 -- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01140c36c0 msgr2=0x7f01141decb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:31.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0119800640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01140c36c0 0x7f01141decb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:31.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0119800640 1 -- 172.21.15.98:0/3056214512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01141dfa30 con 0x7f01140cc180 2024-05-27T00:15:31.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0119800640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01140cc180 0x7f01141de160 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f010c007e40 tx=0x7f010c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:31.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0110c00640 1 -- 172.21.15.98:0/3056214512 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c014620 con 0x7f01140cc180 2024-05-27T00:15:31.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0110c00640 1 -- 172.21.15.98:0/3056214512 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f010c019070 con 0x7f01140cc180 2024-05-27T00:15:31.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f011a200640 1 -- 172.21.15.98:0/3056214512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01141f0e70 con 0x7f01140cc180 2024-05-27T00:15:31.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f0110c00640 1 -- 172.21.15.98:0/3056214512 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c014a20 con 0x7f01140cc180 2024-05-27T00:15:31.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f011a200640 1 -- 172.21.15.98:0/3056214512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01141df750 con 0x7f01140cc180 2024-05-27T00:15:31.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.985+0000 7f011a200640 1 -- 172.21.15.98:0/3056214512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0114075c40 con 0x7f01140cc180 2024-05-27T00:15:31.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.989+0000 7f0110c00640 1 -- 172.21.15.98:0/3056214512 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f010c014bc0 con 0x7f01140cc180 2024-05-27T00:15:31.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.989+0000 7f0110c00640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00f40678c0 0x7f00f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:31.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.989+0000 7f0110c00640 1 -- 172.21.15.98:0/3056214512 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f010c0864b0 con 0x7f01140cc180 2024-05-27T00:15:31.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.989+0000 7f0118e00640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00f40678c0 0x7f00f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:31.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.989+0000 7f0118e00640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00f40678c0 0x7f00f4069d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f01080098d0 tx=0x7f0108008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:31.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:31.993+0000 7f0110c00640 1 -- 172.21.15.98:0/3056214512 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f010c056900 con 0x7f01140cc180 2024-05-27T00:15:32.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.153+0000 7f011a200640 1 -- 172.21.15.98:0/3056214512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01140c5480 con 0x7f01140cc180 2024-05-27T00:15:32.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.153+0000 7f0110c00640 1 -- 172.21.15.98:0/3056214512 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16581 (secure 0 0 0) 0x7f010c05ac00 con 0x7f01140cc180 2024-05-27T00:15:32.156 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:32.156 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:26.579229+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.153+0000 7f011a200640 1 -- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00f40678c0 msgr2=0x7f00f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.153+0000 7f011a200640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00f40678c0 0x7f00f4069d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f01080098d0 tx=0x7f0108008040 comp rx=0 tx=0).stop 2024-05-27T00:15:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.153+0000 7f011a200640 1 -- 172.21.15.98:0/3056214512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01140cc180 msgr2=0x7f01141de160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.153+0000 7f011a200640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01140cc180 0x7f01141de160 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f010c007e40 tx=0x7f010c00e4b0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.153+0000 7f011a200640 1 -- 172.21.15.98:0/3056214512 shutdown_connections 2024-05-27T00:15:32.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.157+0000 7f011a200640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01140cc180 0x7f01141de160 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.157+0000 7f011a200640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00f40678c0 0x7f00f4069d80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.157+0000 7f011a200640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01140c36c0 0x7f01141decb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.157+0000 7f011a200640 1 --2- 172.21.15.98:0/3056214512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0114070660 0x7f01140b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.157+0000 7f011a200640 1 -- 172.21.15.98:0/3056214512 >> 172.21.15.98:0/3056214512 conn(0x7f01140cdeb0 msgr2=0x7f01141e2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.157+0000 7f011a200640 1 -- 172.21.15.98:0/3056214512 shutdown_connections 2024-05-27T00:15:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.157+0000 7f011a200640 1 -- 172.21.15.98:0/3056214512 wait complete. 2024-05-27T00:15:32.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-27T00:15:32.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-27T00:15:32.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-27T00:15:32.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-27T00:15:32.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-27T00:15:32.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-27T00:15:32.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-27T00:15:32.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-27T00:15:32.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.413+0000 7fb563600640 1 Processor -- start 2024-05-27T00:15:32.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.413+0000 7fb563600640 1 -- start start 2024-05-27T00:15:32.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb563600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb55c070c00 0x7fb55c162d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb563600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb55c070fe0 0x7fb55c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb563600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb55c163260 con 0x7fb55c15edf0 2024-05-27T00:15:32.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb563600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb55c0c8a00 con 0x7fb55c070fe0 2024-05-27T00:15:32.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb563600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb55c0c8ce0 con 0x7fb55c070c00 2024-05-27T00:15:32.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb560c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb55c070c00 0x7fb55c162d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb560c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb55c070c00 0x7fb55c162d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:34728/0 (socket says 172.21.15.98:34728) 2024-05-27T00:15:32.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb560c00640 1 -- 172.21.15.98:0/512995221 learned_addr learned my addr 172.21.15.98:0/512995221 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb55be00640 1 --2- 172.21.15.98:0/512995221 >> v2:172.21.15.98:3300/0 conn(0x7fb55c070fe0 0x7fb55c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb55be00640 1 -- 172.21.15.98:0/512995221 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb55c070c00 msgr2=0x7fb55c162d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb55be00640 1 --2- 172.21.15.98:0/512995221 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb55c070c00 0x7fb55c162d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb55be00640 1 -- 172.21.15.98:0/512995221 >> v1:172.21.15.98:6789/0 conn(0x7fb55c15edf0 legacy=0x7fb55c158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7fb55be00640 1 -- 172.21.15.98:0/512995221 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb55c15f310 con 0x7fb55c070fe0 2024-05-27T00:15:32.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7f5587200640 1 Processor -- start 2024-05-27T00:15:32.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.417+0000 7f5587200640 1 -- start start 2024-05-27T00:15:32.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5587200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5588158570 0x7f55880b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5587200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55880b3ca0 0x7f55880c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5587200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5588152a30 con 0x7f5588152650 2024-05-27T00:15:32.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5587200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f558806f170 con 0x7f5588158570 2024-05-27T00:15:32.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5587200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f55880c53b0 con 0x7f55880b3ca0 2024-05-27T00:15:32.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5585400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5588158570 0x7f55880b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5585400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5588158570 0x7f55880b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57660/0 (socket says 172.21.15.98:57660) 2024-05-27T00:15:32.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5585400640 1 -- 172.21.15.98:0/4157120746 learned_addr learned my addr 172.21.15.98:0/4157120746 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5586800640 1 --2- 172.21.15.98:0/4157120746 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55880b3ca0 0x7f55880c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7fb55be00640 1 --2- 172.21.15.98:0/512995221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55c070fe0 0x7fb55c0c9170 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb5480097c0 tx=0x7fb54801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b670922e57753f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5585400640 1 -- 172.21.15.98:0/4157120746 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55880b3ca0 msgr2=0x7f55880c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5585400640 1 --2- 172.21.15.98:0/4157120746 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55880b3ca0 0x7f55880c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5585400640 1 -- 172.21.15.98:0/4157120746 >> v1:172.21.15.204:6789/0 conn(0x7f5588152650 legacy=0x7f558806d8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T00:15:32.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.421+0000 7f5585400640 1 -- 172.21.15.98:0/4157120746 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5588158ab0 con 0x7f5588158570 2024-05-27T00:15:32.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb55b400640 1 -- 172.21.15.98:0/512995221 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5480b2070 con 0x7fb55c070fe0 2024-05-27T00:15:32.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb55b400640 1 -- 172.21.15.98:0/512995221 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5480ad640 con 0x7fb55c070fe0 2024-05-27T00:15:32.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb55b400640 1 -- 172.21.15.98:0/512995221 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5480ada10 con 0x7fb55c070fe0 2024-05-27T00:15:32.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb563600640 1 -- 172.21.15.98:0/512995221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55c070fe0 msgr2=0x7fb55c0c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb563600640 1 --2- 172.21.15.98:0/512995221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55c070fe0 0x7fb55c0c9170 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb5480097c0 tx=0x7fb54801c4c0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f3cbfe00640 1 Processor -- start 2024-05-27T00:15:32.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f3cbfe00640 1 -- start start 2024-05-27T00:15:32.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f5585400640 1 --2- 172.21.15.98:0/4157120746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5588158570 0x7f55880b3760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f55880627b0 tx=0x7f557001c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9459cd58d60c4a71 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb563600640 1 -- 172.21.15.98:0/512995221 shutdown_connections 2024-05-27T00:15:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb563600640 1 --2- 172.21.15.98:0/512995221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55c070fe0 0x7fb55c0c9170 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb563600640 1 --2- 172.21.15.98:0/512995221 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb55c070c00 0x7fb55c162d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb563600640 1 -- 172.21.15.98:0/512995221 >> 172.21.15.98:0/512995221 conn(0x7fb55c072f90 msgr2=0x7fb55c0733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f5584a00640 1 -- 172.21.15.98:0/4157120746 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55700b1070 con 0x7f5588158570 2024-05-27T00:15:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fcaefc00640 1 Processor -- start 2024-05-27T00:15:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb23c000640 1 Processor -- start 2024-05-27T00:15:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f5584a00640 1 -- 172.21.15.98:0/4157120746 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55700ac630 con 0x7f5588158570 2024-05-27T00:15:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f5584a00640 1 -- 172.21.15.98:0/4157120746 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55700a8050 con 0x7f5588158570 2024-05-27T00:15:32.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb563600640 1 -- 172.21.15.98:0/512995221 shutdown_connections 2024-05-27T00:15:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f3cbfe00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3cb806e660 0x7f3cb8150b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f3cbfe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cb80c7bc0 0x7f3cb80c8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f3cbfe00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cb8151050 con 0x7f3cb806da00 2024-05-27T00:15:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f3cbfe00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f3cb806d220 con 0x7f3cb806e660 2024-05-27T00:15:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7f3cbfe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb8156970 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f5587200640 1 -- 172.21.15.98:0/4157120746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5588158570 msgr2=0x7f55880b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f5587200640 1 --2- 172.21.15.98:0/4157120746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5588158570 0x7f55880b3760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f55880627b0 tx=0x7f557001c520 comp rx=0 tx=0).stop 2024-05-27T00:15:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb563600640 1 -- 172.21.15.98:0/512995221 wait complete. 2024-05-27T00:15:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f5587200640 1 -- 172.21.15.98:0/4157120746 shutdown_connections 2024-05-27T00:15:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f5587200640 1 --2- 172.21.15.98:0/4157120746 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55880b3ca0 0x7f55880c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f5587200640 1 --2- 172.21.15.98:0/4157120746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5588158570 0x7f55880b3760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f5587200640 1 -- 172.21.15.98:0/4157120746 >> 172.21.15.98:0/4157120746 conn(0x7f55880cdeb0 msgr2=0x7f55880ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f3cbd400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3cb806e660 0x7f3cb8150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f3cbd400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3cb806e660 0x7f3cb8150b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57662/0 (socket says 172.21.15.98:57662) 2024-05-27T00:15:32.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f3cbde00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cb80c7bc0 0x7f3cb80c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f3cbd400640 1 -- 172.21.15.98:0/3316096926 learned_addr learned my addr 172.21.15.98:0/3316096926 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fb23c000640 1 -- start start 2024-05-27T00:15:32.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.425+0000 7fcaefc00640 1 -- start start 2024-05-27T00:15:32.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f3cbd400640 1 -- 172.21.15.98:0/3316096926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cb80c7bc0 msgr2=0x7f3cb80c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f3cbd400640 1 --2- 172.21.15.98:0/3316096926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cb80c7bc0 0x7f3cb80c8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f3cbd400640 1 -- 172.21.15.98:0/3316096926 >> v1:172.21.15.98:6789/0 conn(0x7f3cb806da00 legacy=0x7f3cb806c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f3cbd400640 1 -- 172.21.15.98:0/3316096926 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb8156c50 con 0x7f3cb806e660 2024-05-27T00:15:32.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcaefc00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcae8162db0 0x7fcae80c1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcaefc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae80c1b50 0x7fcae806e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcaefc00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcae815fb30 con 0x7fcae81590f0 2024-05-27T00:15:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcaefc00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcae80c7700 con 0x7fcae8162db0 2024-05-27T00:15:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcaefc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcae815edb0 con 0x7fcae80c1b50 2024-05-27T00:15:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb23c000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb23406e660 0x7fb23406da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb23c000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb234150a80 0x7fb23406c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb23c000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb23406d220 con 0x7fb2340c7bc0 2024-05-27T00:15:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb23c000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb234156970 con 0x7fb23406e660 2024-05-27T00:15:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb23c000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb234156c50 con 0x7fb234150a80 2024-05-27T00:15:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb239600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb23406e660 0x7fb23406da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb239600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb23406e660 0x7fb23406da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57670/0 (socket says 172.21.15.98:57670) 2024-05-27T00:15:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb239600640 1 -- 172.21.15.98:0/4004891630 learned_addr learned my addr 172.21.15.98:0/4004891630 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb238c00640 1 --2- 172.21.15.98:0/4004891630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb234150a80 0x7fb23406c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcae7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcae8162db0 0x7fcae80c1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcaedc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae80c1b50 0x7fcae806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcaed200640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fcae81590f0 0x7fcae80c7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:42842/0 (socket says 172.21.15.98:42842) 2024-05-27T00:15:32.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcaed200640 1 -- 172.21.15.98:0/1501626881 learned_addr learned my addr 172.21.15.98:0/1501626881 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:32.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f5587200640 1 -- 172.21.15.98:0/4157120746 shutdown_connections 2024-05-27T00:15:32.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb239600640 1 -- 172.21.15.98:0/4004891630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb234150a80 msgr2=0x7fb23406c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb239600640 1 --2- 172.21.15.98:0/4004891630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb234150a80 0x7fb23406c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb239600640 1 -- 172.21.15.98:0/4004891630 >> v1:172.21.15.98:6789/0 conn(0x7fb2340c7bc0 legacy=0x7fb2340c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fb239600640 1 -- 172.21.15.98:0/4004891630 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb234156f30 con 0x7fb23406e660 2024-05-27T00:15:32.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcae7e00640 1 -- 172.21.15.98:0/1501626881 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae80c1b50 msgr2=0x7fcae806e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcae7e00640 1 --2- 172.21.15.98:0/1501626881 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae80c1b50 0x7fcae806e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcae7e00640 1 -- 172.21.15.98:0/1501626881 >> v1:172.21.15.204:6789/0 conn(0x7fcae81590f0 legacy=0x7fcae80c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7fcae7e00640 1 -- 172.21.15.98:0/1501626881 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae80c7420 con 0x7fcae8162db0 2024-05-27T00:15:32.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5538800640 1 Processor -- start 2024-05-27T00:15:32.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7fb563600640 1 Processor -- start 2024-05-27T00:15:32.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7fb563600640 1 -- start start 2024-05-27T00:15:32.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5538800640 1 -- start start 2024-05-27T00:15:32.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f7b98a00640 1 Processor -- start 2024-05-27T00:15:32.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.429+0000 7f5587200640 1 -- 172.21.15.98:0/4157120746 wait complete. 2024-05-27T00:15:32.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7fb563600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55c070c00 0x7fb55c0cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5587200640 1 Processor -- start 2024-05-27T00:15:32.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f3cbd400640 1 --2- 172.21.15.98:0/3316096926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cb806e660 0x7f3cb8150b10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009e50 tx=0x7f3cac01f210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a83d0bc9deb6a7c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f3cb7e00640 1 -- 172.21.15.98:0/3316096926 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0b4070 con 0x7f3cb806e660 2024-05-27T00:15:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f3cb7e00640 1 -- 172.21.15.98:0/3316096926 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cac0af400 con 0x7f3cb806e660 2024-05-27T00:15:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f3cb7e00640 1 -- 172.21.15.98:0/3316096926 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0af7d0 con 0x7f3cb806e660 2024-05-27T00:15:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5538800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 0x7f55300c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5538800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5530162db0 0x7f55300c1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5538800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f553015fb30 con 0x7f55300c1b50 2024-05-27T00:15:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5538800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f55300c7700 con 0x7f5530162db0 2024-05-27T00:15:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5538800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f553015edb0 con 0x7f55301590f0 2024-05-27T00:15:32.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5535400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5530162db0 0x7f55300c1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f7b98a00640 1 -- start start 2024-05-27T00:15:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7fb563600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb55c070fe0 0x7fb55c1f8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7fb563600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb55c15edf0 0x7fb55c14eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5536800640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f55300c1b50 0x7f553006e890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:42856/0 (socket says 172.21.15.98:42856) 2024-05-27T00:15:32.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5536800640 1 -- 172.21.15.98:0/449850887 learned_addr learned my addr 172.21.15.98:0/449850887 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5535e00640 1 --2- 172.21.15.98:0/449850887 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 0x7f55300c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7fb563600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb55c0c9830 con 0x7fb55c070c00 2024-05-27T00:15:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7fb563600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb55c0c96b0 con 0x7fb55c070fe0 2024-05-27T00:15:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7fb563600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb55c0c99b0 con 0x7fb55c15edf0 2024-05-27T00:15:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3316096926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cb806e660 msgr2=0x7f3cb8150b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f3cbfe00640 1 --2- 172.21.15.98:0/3316096926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cb806e660 0x7f3cb8150b10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009e50 tx=0x7f3cac01f210 comp rx=0 tx=0).stop 2024-05-27T00:15:32.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7fb560c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55c070c00 0x7fb55c0cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb560c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55c070c00 0x7fb55c0cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53588/0 (socket says 172.21.15.98:53588) 2024-05-27T00:15:32.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb560c00640 1 -- 172.21.15.98:0/3585745284 learned_addr learned my addr 172.21.15.98:0/3585745284 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.433+0000 7f5587200640 1 -- start start 2024-05-27T00:15:32.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b98a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7b9406e890 0x7f7b9406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b98a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9415fb30 con 0x7f7b941590f0 2024-05-27T00:15:32.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5535400640 1 -- 172.21.15.98:0/449850887 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 msgr2=0x7f55300c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b98a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b940c7700 con 0x7f7b94162db0 2024-05-27T00:15:32.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb55be00640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb55c070fe0 0x7fb55c1f8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b98a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7b9415edb0 con 0x7f7b9406e890 2024-05-27T00:15:32.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb239600640 1 --2- 172.21.15.98:0/4004891630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb23406e660 0x7fb23406da00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb220009870 tx=0x7fb22001ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8899038c490e0220 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b89400640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f7b94162db0 0x7f7b940c1610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:42864/0 (socket says 172.21.15.98:42864) 2024-05-27T00:15:32.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b89400640 1 -- 172.21.15.98:0/3724666024 learned_addr learned my addr 172.21.15.98:0/3724666024 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:32.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb561600640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb55c15edf0 0x7fb55c14eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb560c00640 1 -- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb55c15edf0 msgr2=0x7fb55c14eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb560c00640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb55c15edf0 0x7fb55c14eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb560c00640 1 -- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb55c070fe0 msgr2=0x7fb55c1f8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5587200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55880b3ca0 0x7f55881fe360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb560c00640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb55c070fe0 0x7fb55c1f8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb560c00640 1 -- 172.21.15.98:0/3585745284 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb55c1f9170 con 0x7fb55c070c00 2024-05-27T00:15:32.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5534a00640 1 -- 172.21.15.98:0/449850887 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047167876 0 0) 0x7f553015fb30 con 0x7f55300c1b50 2024-05-27T00:15:32.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5535400640 1 --2- 172.21.15.98:0/449850887 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 0x7f55300c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5535400640 1 -- 172.21.15.98:0/449850887 >> v1:172.21.15.204:6789/0 conn(0x7f55300c1b50 legacy=0x7f553006e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5535400640 1 -- 172.21.15.98:0/449850887 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55300c7420 con 0x7f5530162db0 2024-05-27T00:15:32.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5535e00640 1 --2- 172.21.15.98:0/449850887 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 0x7f55300c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:32.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fcae7e00640 1 --2- 172.21.15.98:0/1501626881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcae8162db0 0x7fcae80c1610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcad40097c0 tx=0x7fcad401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56636511d5ae3db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb233e00640 1 -- 172.21.15.98:0/4004891630 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2200b5070 con 0x7fb23406e660 2024-05-27T00:15:32.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b92800640 1 --2- 172.21.15.98:0/3724666024 >> v2:172.21.15.98:3300/0 conn(0x7f7b9406e890 0x7f7b9406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb233e00640 1 -- 172.21.15.98:0/4004891630 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2200a89d0 con 0x7fb23406e660 2024-05-27T00:15:32.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb233e00640 1 -- 172.21.15.98:0/4004891630 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2200a8da0 con 0x7fb23406e660 2024-05-27T00:15:32.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5587200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5588152650 0x7f55880c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3316096926 shutdown_connections 2024-05-27T00:15:32.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f3cbfe00640 1 --2- 172.21.15.98:0/3316096926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cb80c7bc0 0x7f3cb80c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f3cbfe00640 1 --2- 172.21.15.98:0/3316096926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cb806e660 0x7f3cb8150b10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3316096926 >> 172.21.15.98:0/3316096926 conn(0x7f3cb80cc200 msgr2=0x7f3cb80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3316096926 shutdown_connections 2024-05-27T00:15:32.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5587200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5588158570 0x7f55880c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb560c00640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55c070c00 0x7fb55c0cd0d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb55000d7d0 tx=0x7fb55000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb558c00640 1 -- 172.21.15.98:0/3585745284 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb550018020 con 0x7fb55c070c00 2024-05-27T00:15:32.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb563600640 1 -- 172.21.15.98:0/3585745284 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb55c158710 con 0x7fb55c070c00 2024-05-27T00:15:32.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb563600640 1 -- 172.21.15.98:0/3585745284 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb55c156c10 con 0x7fb55c070c00 2024-05-27T00:15:32.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb558c00640 1 -- 172.21.15.98:0/3585745284 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5500096a0 con 0x7fb55c070c00 2024-05-27T00:15:32.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb558c00640 1 -- 172.21.15.98:0/3585745284 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb550009a50 con 0x7fb55c070c00 2024-05-27T00:15:32.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5587200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f558806e630 con 0x7f5588152650 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5587200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f55880c5690 con 0x7f55880b3ca0 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5587200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5588165710 con 0x7f5588158570 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5585e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55880b3ca0 0x7f55881fe360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5586800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5588158570 0x7f55880c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5585400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5588152650 0x7f55880c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5586800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5588158570 0x7f55880c3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:34770/0 (socket says 172.21.15.98:34770) 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5586800640 1 -- 172.21.15.98:0/371725388 learned_addr learned my addr 172.21.15.98:0/371725388 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5586800640 1 -- 172.21.15.98:0/371725388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55880b3ca0 msgr2=0x7f55881fe360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5586800640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55880b3ca0 0x7f55881fe360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5586800640 1 -- 172.21.15.98:0/371725388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5588152650 msgr2=0x7f55880c3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5586800640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5588152650 0x7f55880c3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5586800640 1 -- 172.21.15.98:0/371725388 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55880c2eb0 con 0x7f5588158570 2024-05-27T00:15:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f5586800640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5588158570 0x7f55880c3c90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f558000e890 tx=0x7f558000ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f557e000640 1 -- 172.21.15.98:0/371725388 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5580017070 con 0x7f5588158570 2024-05-27T00:15:32.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f557e000640 1 -- 172.21.15.98:0/371725388 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f558000c940 con 0x7f5588158570 2024-05-27T00:15:32.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f557e000640 1 -- 172.21.15.98:0/371725388 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f558000ccf0 con 0x7f5588158570 2024-05-27T00:15:32.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3316096926 wait complete. 2024-05-27T00:15:32.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbfe00640 1 Processor -- start 2024-05-27T00:15:32.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fcae7400640 1 -- 172.21.15.98:0/1501626881 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad40b5070 con 0x7fcae8162db0 2024-05-27T00:15:32.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fcae7400640 1 -- 172.21.15.98:0/1501626881 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcad40a8910 con 0x7fcae8162db0 2024-05-27T00:15:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fcae7400640 1 -- 172.21.15.98:0/1501626881 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad40a8ce0 con 0x7fcae8162db0 2024-05-27T00:15:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fcaefc00640 1 -- 172.21.15.98:0/1501626881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcae8162db0 msgr2=0x7fcae80c1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fcaefc00640 1 --2- 172.21.15.98:0/1501626881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcae8162db0 0x7fcae80c1610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcad40097c0 tx=0x7fcad401ec50 comp rx=0 tx=0).stop 2024-05-27T00:15:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fcaefc00640 1 -- 172.21.15.98:0/1501626881 shutdown_connections 2024-05-27T00:15:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fcaefc00640 1 --2- 172.21.15.98:0/1501626881 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae80c1b50 0x7fcae806e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fcaefc00640 1 --2- 172.21.15.98:0/1501626881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcae8162db0 0x7fcae80c1610 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcad40097c0 tx=0x7fcad401ec50 comp rx=0 tx=0).stop 2024-05-27T00:15:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fcaefc00640 1 -- 172.21.15.98:0/1501626881 >> 172.21.15.98:0/1501626881 conn(0x7fcae80cc200 msgr2=0x7fcae80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb23c000640 1 -- 172.21.15.98:0/4004891630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb23406e660 msgr2=0x7fb23406da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7fb23c000640 1 --2- 172.21.15.98:0/4004891630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb23406e660 0x7fb23406da00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb220009870 tx=0x7fb22001ed10 comp rx=0 tx=0).stop 2024-05-27T00:15:32.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fb23c000640 1 -- 172.21.15.98:0/4004891630 shutdown_connections 2024-05-27T00:15:32.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fb23c000640 1 --2- 172.21.15.98:0/4004891630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb234150a80 0x7fb23406c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fb23c000640 1 --2- 172.21.15.98:0/4004891630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb23406e660 0x7fb23406da00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fb23c000640 1 -- 172.21.15.98:0/4004891630 >> 172.21.15.98:0/4004891630 conn(0x7fb2340cc200 msgr2=0x7fb2340cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f07eb200640 1 Processor -- start 2024-05-27T00:15:32.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07eb200640 1 -- start start 2024-05-27T00:15:32.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b91400640 1 -- 172.21.15.98:0/3724666024 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 843672001 0 0) 0x7f7b940c7700 con 0x7f7b94162db0 2024-05-27T00:15:32.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b91400640 1 -- 172.21.15.98:0/3724666024 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b78003530 con 0x7f7b94162db0 2024-05-27T00:15:32.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b92800640 1 -- 172.21.15.98:0/3724666024 >> v1:172.21.15.204:6789/0 conn(0x7f7b94162db0 legacy=0x7f7b940c1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b92800640 1 -- 172.21.15.98:0/3724666024 >> v1:172.21.15.98:6789/0 conn(0x7f7b941590f0 legacy=0x7f7b940c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.437+0000 7f7b92800640 1 -- 172.21.15.98:0/3724666024 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b940c7420 con 0x7f7b9406e890 2024-05-27T00:15:32.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fb23c000640 1 -- 172.21.15.98:0/4004891630 shutdown_connections 2024-05-27T00:15:32.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55880c3270 con 0x7f5588158570 2024-05-27T00:15:32.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5588155580 con 0x7f5588158570 2024-05-27T00:15:32.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbfe00640 1 -- start start 2024-05-27T00:15:32.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f5535400640 1 --2- 172.21.15.98:0/449850887 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5530162db0 0x7f55300c1610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f55240097c0 tx=0x7f552401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac31efd3542b4b75 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fb563600640 1 -- 172.21.15.98:0/3585745284 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb538001040 con 0x7fb55c070c00 2024-05-27T00:15:32.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fcaefc00640 1 -- 172.21.15.98:0/1501626881 shutdown_connections 2024-05-27T00:15:32.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fcaefc00640 1 -- 172.21.15.98:0/1501626881 wait complete. 2024-05-27T00:15:32.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07eb200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f07e4036260 0x7f07e40460d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07eb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07e4045300 0x7f07e4046610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fb558c00640 1 -- 172.21.15.98:0/3585745284 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb55001b020 con 0x7fb55c070c00 2024-05-27T00:15:32.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f558804f7e0 con 0x7f5588158570 2024-05-27T00:15:32.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f5534a00640 1 -- 172.21.15.98:0/449850887 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55240b5070 con 0x7f5530162db0 2024-05-27T00:15:32.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f5534a00640 1 -- 172.21.15.98:0/449850887 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55240a8990 con 0x7f5530162db0 2024-05-27T00:15:32.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f5534a00640 1 -- 172.21.15.98:0/449850887 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55240a8d60 con 0x7f5530162db0 2024-05-27T00:15:32.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fb23c000640 1 -- 172.21.15.98:0/4004891630 wait complete. 2024-05-27T00:15:32.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7fb23c000640 1 Processor -- start 2024-05-27T00:15:32.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07eb200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e40dc730 con 0x7f07e40d5c90 2024-05-27T00:15:32.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07eb200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f07e4033610 con 0x7f07e4036260 2024-05-27T00:15:32.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07e9e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f07e40d5c90 0x7f07e4034fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43364/0 (socket says 172.21.15.98:43364) 2024-05-27T00:15:32.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07eb200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f07e40338f0 con 0x7f07e4045300 2024-05-27T00:15:32.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07e9e00640 1 -- 172.21.15.98:0/176612240 learned_addr learned my addr 172.21.15.98:0/176612240 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:32.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07e9400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f07e4036260 0x7f07e40460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07f0a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07e4045300 0x7f07e4046610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f7b92800640 1 --2- 172.21.15.98:0/3724666024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b9406e890 0x7f7b9406ece0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c0097c0 tx=0x7f7b8c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1821ddb44e76bc1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbfe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb806da00 0x7f3cb81514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbfe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cb806e660 0x7f3cb8141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cb80c7bc0 0x7f3cb8141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbfe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb806de20 con 0x7f3cb806da00 2024-05-27T00:15:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbfe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb8157210 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbfe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb806dfa0 con 0x7f3cb806e660 2024-05-27T00:15:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cb80c7bc0 0x7f3cb8141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cb80c7bc0 0x7f3cb8141f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57696/0 (socket says 172.21.15.98:57696) 2024-05-27T00:15:32.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f3cbde00640 1 -- 172.21.15.98:0/3859597685 learned_addr learned my addr 172.21.15.98:0/3859597685 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f5538800640 1 -- 172.21.15.98:0/449850887 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5530162db0 msgr2=0x7f55300c1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb558c00640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb53c0678c0 0x7fb53c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f5538800640 1 --2- 172.21.15.98:0/449850887 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5530162db0 0x7f55300c1610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f55240097c0 tx=0x7f552401ec90 comp rx=0 tx=0).stop 2024-05-27T00:15:32.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07e9400640 1 -- 172.21.15.98:0/176612240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07e4045300 msgr2=0x7f07e4046610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07e9400640 1 --2- 172.21.15.98:0/176612240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07e4045300 0x7f07e4046610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07e9400640 1 -- 172.21.15.98:0/176612240 >> v1:172.21.15.98:6789/0 conn(0x7f07e40d5c90 legacy=0x7f07e4034fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.441+0000 7f07e9400640 1 -- 172.21.15.98:0/176612240 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e40dc400 con 0x7f07e4036260 2024-05-27T00:15:32.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f5538800640 1 -- 172.21.15.98:0/449850887 shutdown_connections 2024-05-27T00:15:32.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f557e000640 1 -- 172.21.15.98:0/371725388 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f558001a070 con 0x7f5588158570 2024-05-27T00:15:32.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f5538800640 1 --2- 172.21.15.98:0/449850887 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5530162db0 0x7f55300c1610 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f5538800640 1 --2- 172.21.15.98:0/449850887 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 0x7f55300c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f5538800640 1 -- 172.21.15.98:0/449850887 >> 172.21.15.98:0/449850887 conn(0x7f55300cc200 msgr2=0x7f55300cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f3cbca00640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cb806e660 0x7f3cb8141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb558c00640 1 -- 172.21.15.98:0/3585745284 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fb550086230 con 0x7fb55c070c00 2024-05-27T00:15:32.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb55be00640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb53c0678c0 0x7fb53c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb558c00640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] conn(0x7fb53c06fac0 0x7fb53c071ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb558c00640 1 -- 172.21.15.98:0/3585745284 --> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb55000af30 con 0x7fb53c06fac0 2024-05-27T00:15:32.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f557e000640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f55640678f0 0x7f5564069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f557e000640 1 -- 172.21.15.98:0/371725388 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5580085cc0 con 0x7f5588158570 2024-05-27T00:15:32.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f3cbd400640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb806da00 0x7f3cb81514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f3cbde00640 1 -- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cb806e660 msgr2=0x7f3cb8141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fcaefc00640 1 Processor -- start 2024-05-27T00:15:32.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fcaefc00640 1 -- start start 2024-05-27T00:15:32.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb23c000640 1 -- start start 2024-05-27T00:15:32.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb23c000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23406e660 0x7fb234200f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb23c000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2340c7bc0 0x7fb234148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb23c000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb234150a80 0x7fb2340c5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb23c000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb23406ceb0 con 0x7fb23406e660 2024-05-27T00:15:32.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb23c000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb23406df40 con 0x7fb234150a80 2024-05-27T00:15:32.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb23c000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb234150ea0 con 0x7fb2340c7bc0 2024-05-27T00:15:32.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb238c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2340c7bc0 0x7fb234148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb239600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23406e660 0x7fb234200f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb238c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2340c7bc0 0x7fb234148630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:34784/0 (socket says 172.21.15.98:34784) 2024-05-27T00:15:32.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb238c00640 1 -- 172.21.15.98:0/981426920 learned_addr learned my addr 172.21.15.98:0/981426920 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb561600640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] conn(0x7fb53c06fac0 0x7fb53c071ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb558c00640 1 -- 172.21.15.98:0/3585745284 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb55008d050 con 0x7fb55c070c00 2024-05-27T00:15:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb561600640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] conn(0x7fb53c06fac0 0x7fb53c071ee0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f557e000640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] conn(0x7f556406faf0 0x7f5564071f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f5585e00640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f55640678f0 0x7f5564069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f5585400640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] conn(0x7f556406faf0 0x7f5564071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f557e000640 1 -- 172.21.15.98:0/371725388 --> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f558000ae90 con 0x7f556406faf0 2024-05-27T00:15:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f557e000640 1 -- 172.21.15.98:0/371725388 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f55800882e0 con 0x7f5588158570 2024-05-27T00:15:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f3cbde00640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cb806e660 0x7f3cb8141a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f3cbde00640 1 -- 172.21.15.98:0/3859597685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb806da00 msgr2=0x7f3cb81514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f3cbde00640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb806da00 0x7f3cb81514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f3cbde00640 1 -- 172.21.15.98:0/3859597685 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb8140690 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f3cbd400640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cb806da00 0x7f3cb81514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f5538800640 1 -- 172.21.15.98:0/449850887 shutdown_connections 2024-05-27T00:15:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f5538800640 1 -- 172.21.15.98:0/449850887 wait complete. 2024-05-27T00:15:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fcaefc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae80c1b50 0x7fcae80c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb23a000640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb234150a80 0x7fb2340c5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb239600640 1 -- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2340c7bc0 msgr2=0x7fb234148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb239600640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2340c7bc0 0x7fb234148630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb239600640 1 -- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb234150a80 msgr2=0x7fb2340c5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb239600640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb234150a80 0x7fb2340c5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb239600640 1 -- 172.21.15.98:0/981426920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb234201440 con 0x7fb23406e660 2024-05-27T00:15:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb239600640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23406e660 0x7fb234200f00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb2200af000 tx=0x7fb22001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f7b91400640 1 -- 172.21.15.98:0/3724666024 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0b5070 con 0x7f7b9406e890 2024-05-27T00:15:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f7b91400640 1 -- 172.21.15.98:0/3724666024 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b8c0a8950 con 0x7f7b9406e890 2024-05-27T00:15:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f7b91400640 1 -- 172.21.15.98:0/3724666024 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0a8d20 con 0x7f7b9406e890 2024-05-27T00:15:32.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fb55be00640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb53c0678c0 0x7fb53c069d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb5480af000 tx=0x7fb5480a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5585400640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] conn(0x7f556406faf0 0x7f5564071f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f3cbca00640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cb806e660 0x7f3cb8141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:32.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fcaefc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcae81590f0 0x7fcae80b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7fcaefc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae81f5110 0x7fcae80c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f7b98a00640 1 -- 172.21.15.98:0/3724666024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b9406e890 msgr2=0x7f7b9406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.445+0000 7f7b98a00640 1 --2- 172.21.15.98:0/3724666024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b9406e890 0x7f7b9406ece0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c0097c0 tx=0x7f7b8c01ec50 comp rx=0 tx=0).stop 2024-05-27T00:15:32.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f7b98a00640 1 -- 172.21.15.98:0/3724666024 shutdown_connections 2024-05-27T00:15:32.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f7b98a00640 1 --2- 172.21.15.98:0/3724666024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b9406e890 0x7f7b9406ece0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f7b98a00640 1 -- 172.21.15.98:0/3724666024 >> 172.21.15.98:0/3724666024 conn(0x7f7b940cc200 msgr2=0x7f7b940cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5585e00640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f55640678f0 0x7f5564069db0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5578009f60 tx=0x7f55780094f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f3cbde00640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cb80c7bc0 0x7f3cb8141f50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000e7f0 tx=0x7f3cb000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5538800640 1 Processor -- start 2024-05-27T00:15:32.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaefc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcae815f610 con 0x7fcae81590f0 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaefc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcae815f330 con 0x7fcae81f5110 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaefc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcae80c6fe0 con 0x7fcae80c1b50 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaed200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae80c1b50 0x7fcae80c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaedc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae81f5110 0x7fcae80c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaedc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae81f5110 0x7fcae80c3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57706/0 (socket says 172.21.15.98:57706) 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaedc00640 1 -- 172.21.15.98:0/1505591963 learned_addr learned my addr 172.21.15.98:0/1505591963 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcae7e00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcae81590f0 0x7fcae80b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fb231600640 1 -- 172.21.15.98:0/981426920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2200c0070 con 0x7fb23406e660 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fb231600640 1 -- 172.21.15.98:0/981426920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb220003830 con 0x7fb23406e660 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fb231600640 1 -- 172.21.15.98:0/981426920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2200b5070 con 0x7fb23406e660 2024-05-27T00:15:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2340c5b30 con 0x7fb23406e660 2024-05-27T00:15:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f7b98a00640 1 -- 172.21.15.98:0/3724666024 shutdown_connections 2024-05-27T00:15:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f7b98a00640 1 -- 172.21.15.98:0/3724666024 wait complete. 2024-05-27T00:15:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f3cb5600640 1 -- 172.21.15.98:0/3859597685 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0017070 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f3cb5600640 1 -- 172.21.15.98:0/3859597685 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cb000c730 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f3cb5600640 1 -- 172.21.15.98:0/3859597685 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb000cb40 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb8140970 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb8140f30 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaedc00640 1 -- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae80c1b50 msgr2=0x7fcae80c58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaedc00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae80c1b50 0x7fcae80c58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaedc00640 1 -- 172.21.15.98:0/1505591963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcae81590f0 msgr2=0x7fcae80b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaedc00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcae81590f0 0x7fcae80b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaedc00640 1 -- 172.21.15.98:0/1505591963 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae81f8950 con 0x7fcae81f5110 2024-05-27T00:15:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2341fd450 con 0x7fb23406e660 2024-05-27T00:15:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f07e9400640 1 --2- 172.21.15.98:0/176612240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07e4036260 0x7f07e40460d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f07d800ba60 tx=0x7f07d8021360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e24261c75f5635f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5538800640 1 -- start start 2024-05-27T00:15:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f07e8a00640 1 -- 172.21.15.98:0/176612240 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d80b6070 con 0x7f07e4036260 2024-05-27T00:15:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f07e8a00640 1 -- 172.21.15.98:0/176612240 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d80b1600 con 0x7f07e4036260 2024-05-27T00:15:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f07e8a00640 1 -- 172.21.15.98:0/176612240 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d80aae60 con 0x7f07e4036260 2024-05-27T00:15:32.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f7b98a00640 1 Processor -- start 2024-05-27T00:15:32.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fb558c00640 1 -- 172.21.15.98:0/3585745284 <== osd.0 v2:172.21.15.98:6816/4085423032 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb55000af30 con 0x7fb53c06fac0 2024-05-27T00:15:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3cb8075cb0 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f557e000640 1 -- 172.21.15.98:0/371725388 <== osd.1 v2:172.21.15.98:6808/1799281170 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f558000ae90 con 0x7f556406faf0 2024-05-27T00:15:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb234075cb0 con 0x7fb23406e660 2024-05-27T00:15:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcaedc00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae81f5110 0x7fcae80c3c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcae0007da0 tx=0x7fcae000e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f07eb200640 1 -- 172.21.15.98:0/176612240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07e4036260 msgr2=0x7f07e40460d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f07eb200640 1 --2- 172.21.15.98:0/176612240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07e4036260 0x7f07e40460d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f07d800ba60 tx=0x7f07d8021360 comp rx=0 tx=0).stop 2024-05-27T00:15:32.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcae4c00640 1 -- 172.21.15.98:0/1505591963 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0014420 con 0x7fcae81f5110 2024-05-27T00:15:32.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcae4c00640 1 -- 172.21.15.98:0/1505591963 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae0019070 con 0x7fcae81f5110 2024-05-27T00:15:32.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5538800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55300c1b50 0x7f55300c2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7fcae4c00640 1 -- 172.21.15.98:0/1505591963 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0014820 con 0x7fcae81f5110 2024-05-27T00:15:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5538800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 0x7f55300c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5538800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5530162db0 0x7f55300c5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5538800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f553015f610 con 0x7f5530162db0 2024-05-27T00:15:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5538800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f553015f330 con 0x7f55300c1b50 2024-05-27T00:15:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5538800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f55300c8050 con 0x7f55301590f0 2024-05-27T00:15:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f07eb200640 1 -- 172.21.15.98:0/176612240 shutdown_connections 2024-05-27T00:15:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f07eb200640 1 --2- 172.21.15.98:0/176612240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07e4045300 0x7f07e4046610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f07eb200640 1 --2- 172.21.15.98:0/176612240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07e4036260 0x7f07e40460d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f07eb200640 1 -- 172.21.15.98:0/176612240 >> 172.21.15.98:0/176612240 conn(0x7f07e4050360 msgr2=0x7f07e4050770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5535e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55300c1b50 0x7f55300c2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f5535400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 0x7f55300c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f5535400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 0x7f55300c4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:34794/0 (socket says 172.21.15.98:34794) 2024-05-27T00:15:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f5535400640 1 -- 172.21.15.98:0/3984233427 learned_addr learned my addr 172.21.15.98:0/3984233427 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae81f8c30 con 0x7fcae81f5110 2024-05-27T00:15:32.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.449+0000 7f7b98a00640 1 -- start start 2024-05-27T00:15:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae81f91f0 con 0x7fcae81f5110 2024-05-27T00:15:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f07eb200640 1 -- 172.21.15.98:0/176612240 shutdown_connections 2024-05-27T00:15:32.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f5535400640 1 -- 172.21.15.98:0/3984233427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55300c1b50 msgr2=0x7f55300c2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f5535400640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55300c1b50 0x7f55300c2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f5535400640 1 -- 172.21.15.98:0/3984233427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5530162db0 msgr2=0x7f55300c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f5535400640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5530162db0 0x7f55300c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f5535400640 1 -- 172.21.15.98:0/3984233427 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55300c6270 con 0x7f55301590f0 2024-05-27T00:15:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcae81f8700 con 0x7fcae81f5110 2024-05-27T00:15:32.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f07eb200640 1 -- 172.21.15.98:0/176612240 wait complete. 2024-05-27T00:15:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b98a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b9406e890 0x7f7b94159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b98a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b941590f0 0x7f7b9414f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b98a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b94162db0 0x7f7b9415e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b98a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9415f610 con 0x7f7b941590f0 2024-05-27T00:15:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b98a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9415f330 con 0x7f7b9406e890 2024-05-27T00:15:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b98a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7b940c1d20 con 0x7f7b94162db0 2024-05-27T00:15:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b91e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b9406e890 0x7f7b94159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b91e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b9406e890 0x7f7b94159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57726/0 (socket says 172.21.15.98:57726) 2024-05-27T00:15:32.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b91e00640 1 -- 172.21.15.98:0/238353670 learned_addr learned my addr 172.21.15.98:0/238353670 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f5535400640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 0x7f55300c4f20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f552401d340 tx=0x7f552401ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b89400640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b941590f0 0x7f7b9414f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b92800640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b94162db0 0x7f7b9415e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f3cb5600640 1 -- 172.21.15.98:0/3859597685 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3cb001a070 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f552a000640 1 -- 172.21.15.98:0/3984233427 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55240b5040 con 0x7f55301590f0 2024-05-27T00:15:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f552a000640 1 -- 172.21.15.98:0/3984233427 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55240c0070 con 0x7f55301590f0 2024-05-27T00:15:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b89400640 1 -- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b94162db0 msgr2=0x7f7b9415e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b89400640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b94162db0 0x7f7b9415e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b89400640 1 -- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b9406e890 msgr2=0x7f7b94159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b89400640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b9406e890 0x7f7b94159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b89400640 1 -- 172.21.15.98:0/238353670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b941568a0 con 0x7f7b941590f0 2024-05-27T00:15:32.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b92800640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b94162db0 0x7f7b9415e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f7b89400640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b941590f0 0x7f7b9414f9b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00c780 tx=0x7f7b7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7fb231600640 1 -- 172.21.15.98:0/981426920 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb2200c3070 con 0x7fb23406e660 2024-05-27T00:15:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7fb231600640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb214067990 0x7fb214069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7fb231600640 1 -- 172.21.15.98:0/981426920 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fb220122ea0 con 0x7fb23406e660 2024-05-27T00:15:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fb231600640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] conn(0x7fb21406fb30 0x7fb214071f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fb231600640 1 -- 172.21.15.98:0/981426920 --> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb214072600 con 0x7fb21406fb30 2024-05-27T00:15:32.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f7b8aa00640 1 -- 172.21.15.98:0/238353670 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c018070 con 0x7f7b941590f0 2024-05-27T00:15:32.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f7b8aa00640 1 -- 172.21.15.98:0/238353670 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7c00a640 con 0x7f7b941590f0 2024-05-27T00:15:32.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f7b8aa00640 1 -- 172.21.15.98:0/238353670 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c018070 con 0x7f7b941590f0 2024-05-27T00:15:32.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f553015a5d0 con 0x7f55301590f0 2024-05-27T00:15:32.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55300c4160 con 0x7f55301590f0 2024-05-27T00:15:32.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f552a000640 1 -- 172.21.15.98:0/3984233427 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55240b0980 con 0x7f55301590f0 2024-05-27T00:15:32.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07eb200640 1 Processor -- start 2024-05-27T00:15:32.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f3cb5600640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3c980678c0 0x7f3c98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f3cb5600640 1 -- 172.21.15.98:0/3859597685 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f3cb0086330 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f3cb5600640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] conn(0x7f3c9806fa60 0x7f3c98071e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f3cb5600640 1 -- 172.21.15.98:0/3859597685 --> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c98072530 con 0x7f3c9806fa60 2024-05-27T00:15:32.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.453+0000 7f3cb5600640 1 -- 172.21.15.98:0/3859597685 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3cb00867c0 con 0x7f3cb80c7bc0 2024-05-27T00:15:32.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fb231600640 1 -- 172.21.15.98:0/981426920 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb2200ef610 con 0x7fb23406e660 2024-05-27T00:15:32.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fb23a000640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] conn(0x7fb21406fb30 0x7fb214071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fcae4c00640 1 -- 172.21.15.98:0/1505591963 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcae00149c0 con 0x7fcae81f5110 2024-05-27T00:15:32.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f3cbd400640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3c980678c0 0x7f3c98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f3cbca00640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] conn(0x7f3c9806fa60 0x7f3c98071e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fb238c00640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb214067990 0x7fb214069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07eb200640 1 -- start start 2024-05-27T00:15:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f7b98a00640 1 -- 172.21.15.98:0/238353670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b94160490 con 0x7f7b941590f0 2024-05-27T00:15:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f7b98a00640 1 -- 172.21.15.98:0/238353670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b941f81b0 con 0x7f7b941590f0 2024-05-27T00:15:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fcae4c00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcac80678c0 0x7fcac8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fcae4c00640 1 -- 172.21.15.98:0/1505591963 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fcae0087230 con 0x7fcae81f5110 2024-05-27T00:15:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07eb200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07e4036260 0x7f07e40d9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07eb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07e4045300 0x7f07e40d7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07eb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07e40d5c90 0x7f07e40d8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07eb200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07e40d6070 con 0x7f07e4036260 2024-05-27T00:15:32.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07eb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f07e400e4a0 con 0x7f07e40d5c90 2024-05-27T00:15:32.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07eb200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f07e40356e0 con 0x7f07e4045300 2024-05-27T00:15:32.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f7b83400640 1 -- 172.21.15.98:0/238353670 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b64001040 con 0x7f7b941590f0 2024-05-27T00:15:32.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fb23a000640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] conn(0x7fb21406fb30 0x7fb214071f50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f55300c3f10 con 0x7f55301590f0 2024-05-27T00:15:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fcae4c00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] conn(0x7fcac806fb40 0x7fcac8071f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fcae4c00640 1 -- 172.21.15.98:0/1505591963 --> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcae0016e70 con 0x7fcac806fb40 2024-05-27T00:15:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fcae4c00640 1 -- 172.21.15.98:0/1505591963 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fcae0087620 con 0x7fcae81f5110 2024-05-27T00:15:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fcaed200640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcac80678c0 0x7fcac8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7fb238c00640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb214067990 0x7fb214069e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb234156430 tx=0x7fb228008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07e9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07e4036260 0x7f07e40d9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07e9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07e4036260 0x7f07e40d9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53660/0 (socket says 172.21.15.98:53660) 2024-05-27T00:15:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07e9e00640 1 -- 172.21.15.98:0/3668629299 learned_addr learned my addr 172.21.15.98:0/3668629299 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07f0a00640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07e40d5c90 0x7f07e40d8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f07e9400640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07e4045300 0x7f07e40d7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f3cbca00640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] conn(0x7f3c9806fa60 0x7f3c98071e80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.457+0000 7f3cbd400640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3c980678c0 0x7f3c98069d80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0b1000 tx=0x7f3cac0aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07f0a00640 1 -- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07e4045300 msgr2=0x7f07e40d7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07f0a00640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07e4045300 0x7f07e40d7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07f0a00640 1 -- 172.21.15.98:0/3668629299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07e4036260 msgr2=0x7f07e40d9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7fcae7e00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] conn(0x7fcac806fb40 0x7fcac8071f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07f0a00640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07e4036260 0x7f07e40d9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07f0a00640 1 -- 172.21.15.98:0/3668629299 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e40da0a0 con 0x7f07e40d5c90 2024-05-27T00:15:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07e9e00640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07e4036260 0x7f07e40d9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7fcaed200640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcac80678c0 0x7fcac8069d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcad8006fd0 tx=0x7fcad8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07f0a00640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07e40d5c90 0x7f07e40d8510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0505c0 tx=0x7f07ec069a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7fcae7e00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] conn(0x7fcac806fb40 0x7fcac8071f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f552a000640 1 -- 172.21.15.98:0/3984233427 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f55240b8070 con 0x7f55301590f0 2024-05-27T00:15:32.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7fb231600640 1 -- 172.21.15.98:0/981426920 <== osd.5 v2:172.21.15.204:6808/858542033 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb214072600 con 0x7fb21406fb30 2024-05-27T00:15:32.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f7b8aa00640 1 -- 172.21.15.98:0/238353670 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7b7c00a830 con 0x7f7b941590f0 2024-05-27T00:15:32.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07e2000640 1 -- 172.21.15.98:0/3668629299 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec092030 con 0x7f07e40d5c90 2024-05-27T00:15:32.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07e2000640 1 -- 172.21.15.98:0/3668629299 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ec050700 con 0x7f07e40d5c90 2024-05-27T00:15:32.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07e2000640 1 -- 172.21.15.98:0/3668629299 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec073830 con 0x7f07e40d5c90 2024-05-27T00:15:32.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f3cb5600640 1 -- 172.21.15.98:0/3859597685 <== osd.2 v2:172.21.15.98:6800/2594065140 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c98072530 con 0x7f3c9806fa60 2024-05-27T00:15:32.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07eb200640 1 -- 172.21.15.98:0/3668629299 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e4182140 con 0x7f07e40d5c90 2024-05-27T00:15:32.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07eb200640 1 -- 172.21.15.98:0/3668629299 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e40dcb00 con 0x7f07e40d5c90 2024-05-27T00:15:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f7b8aa00640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b70067cc0 0x7f7b7006a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f7b8aa00640 1 -- 172.21.15.98:0/238353670 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f7b7c004d80 con 0x7f7b941590f0 2024-05-27T00:15:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f7b91e00640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b70067cc0 0x7f7b7006a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f552a000640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f55140678c0 0x7f5514069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f552a000640 1 -- 172.21.15.98:0/3984233427 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5524123110 con 0x7f55301590f0 2024-05-27T00:15:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f552a000640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] conn(0x7f551406fac0 0x7f5514071ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f552a000640 1 -- 172.21.15.98:0/3984233427 --> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5514072590 con 0x7f551406fac0 2024-05-27T00:15:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f552a000640 1 -- 172.21.15.98:0/3984233427 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f552412a050 con 0x7f55301590f0 2024-05-27T00:15:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f5535e00640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f55140678c0 0x7f5514069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f7b8aa00640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] conn(0x7f7b7006fec0 0x7f7b700722e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f7b8aa00640 1 -- 172.21.15.98:0/238353670 --> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b7c014e50 con 0x7f7b7006fec0 2024-05-27T00:15:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f7b8aa00640 1 -- 172.21.15.98:0/238353670 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7b7c085650 con 0x7f7b941590f0 2024-05-27T00:15:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.461+0000 7f07d7400640 1 -- 172.21.15.98:0/3668629299 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f07c0001040 con 0x7f07e40d5c90 2024-05-27T00:15:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f5536800640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] conn(0x7f551406fac0 0x7f5514071ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7fcae4c00640 1 -- 172.21.15.98:0/1505591963 <== osd.4 v2:172.21.15.204:6800/4073643582 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcae0016e70 con 0x7fcac806fb40 2024-05-27T00:15:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f7b92800640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] conn(0x7f7b7006fec0 0x7f7b700722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f5535e00640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f55140678c0 0x7f5514069d80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f552c003b90 tx=0x7f552c009890 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f7b91e00640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b70067cc0 0x7f7b7006a180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b84007fd0 tx=0x7f7b84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f5536800640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] conn(0x7f551406fac0 0x7f5514071ee0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f07e2000640 1 -- 172.21.15.98:0/3668629299 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f07ec095030 con 0x7f07e40d5c90 2024-05-27T00:15:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f7b92800640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] conn(0x7f7b7006fec0 0x7f7b700722e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f552a000640 1 -- 172.21.15.98:0/3984233427 <== osd.3 v2:172.21.15.98:6817/1805101132 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5514072590 con 0x7f551406fac0 2024-05-27T00:15:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f07e2000640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07cc0678f0 0x7f07cc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f07e2000640 1 -- 172.21.15.98:0/3668629299 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f07ec100fc0 con 0x7f07e40d5c90 2024-05-27T00:15:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f07e2000640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] conn(0x7f07cc06fb70 0x7f07cc071f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f07e2000640 1 -- 172.21.15.98:0/3668629299 --> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07cc072640 con 0x7f07cc06fb70 2024-05-27T00:15:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.465+0000 7f07e2000640 1 -- 172.21.15.98:0/3668629299 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f07ec1013b0 con 0x7f07e40d5c90 2024-05-27T00:15:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.469+0000 7f07e9400640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] conn(0x7f07cc06fb70 0x7f07cc071f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.469+0000 7f07e9e00640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07cc0678f0 0x7f07cc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.469+0000 7f7b8aa00640 1 -- 172.21.15.98:0/238353670 <== osd.7 v2:172.21.15.204:6816/874852314 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b7c014e50 con 0x7f7b7006fec0 2024-05-27T00:15:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.469+0000 7f07e9400640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] conn(0x7f07cc06fb70 0x7f07cc071f90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.469+0000 7f07e2000640 1 -- 172.21.15.98:0/3668629299 <== osd.6 v2:172.21.15.204:6824/469091443 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07cc072640 con 0x7f07cc06fb70 2024-05-27T00:15:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.473+0000 7f07e9e00640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07cc0678f0 0x7f07cc069db0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f07dc005890 tx=0x7f07dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.477+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 --> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f55880c3440 con 0x7f556406faf0 2024-05-27T00:15:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.477+0000 7f557e000640 1 -- 172.21.15.98:0/371725388 <== osd.1 v2:172.21.15.98:6808/1799281170 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f55880c3440 con 0x7f556406faf0 2024-05-27T00:15:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.481+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 >> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] conn(0x7f556406faf0 msgr2=0x7f5564071f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.481+0000 7f5587200640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] conn(0x7f556406faf0 0x7f5564071f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.481+0000 7fb563600640 1 -- 172.21.15.98:0/3585745284 --> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb538002de0 con 0x7fb53c06fac0 2024-05-27T00:15:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.481+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f55640678f0 msgr2=0x7f5564069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.481+0000 7f5587200640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f55640678f0 0x7f5564069db0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5578009f60 tx=0x7f55780094f0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.481+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5588158570 msgr2=0x7f55880c3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.481+0000 7f5587200640 1 --2- 172.21.15.98:0/371725388 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5588158570 0x7f55880c3c90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f558000e890 tx=0x7f558000ed50 comp rx=0 tx=0).stop 2024-05-27T00:15:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.481+0000 7f5586800640 1 -- 172.21.15.98:0/371725388 reap_dead start 2024-05-27T00:15:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.481+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 shutdown_connections 2024-05-27T00:15:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.481+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 >> 172.21.15.98:0/371725388 conn(0x7f55880cdeb0 msgr2=0x7f5588156a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.485+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 shutdown_connections 2024-05-27T00:15:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.485+0000 7f5587200640 1 -- 172.21.15.98:0/371725388 wait complete. 2024-05-27T00:15:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.485+0000 7fb558c00640 1 -- 172.21.15.98:0/3585745284 <== osd.0 v2:172.21.15.98:6816/4085423032 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb538002de0 con 0x7fb53c06fac0 2024-05-27T00:15:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.485+0000 7fb54e000640 1 -- 172.21.15.98:0/3585745284 >> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] conn(0x7fb53c06fac0 msgr2=0x7fb53c071ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.485+0000 7fb54e000640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] conn(0x7fb53c06fac0 0x7fb53c071ee0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.485+0000 7fb54e000640 1 -- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb53c0678c0 msgr2=0x7fb53c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.485+0000 7fb54e000640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb53c0678c0 0x7fb53c069d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb5480af000 tx=0x7fb5480a7070 comp rx=0 tx=0).stop 2024-05-27T00:15:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.485+0000 7fb54e000640 1 -- 172.21.15.98:0/3585745284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55c070c00 msgr2=0x7fb55c0cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.485+0000 7fb54e000640 1 --2- 172.21.15.98:0/3585745284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb55c070c00 0x7fb55c0cd0d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb55000d7d0 tx=0x7fb55000dc90 comp rx=0 tx=0).stop 2024-05-27T00:15:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 --> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3cb8062720 con 0x7f3c9806fa60 2024-05-27T00:15:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cb5600640 1 -- 172.21.15.98:0/3859597685 <== osd.2 v2:172.21.15.98:6800/2594065140 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3cb8062720 con 0x7f3c9806fa60 2024-05-27T00:15:32.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7fb561600640 1 -- 172.21.15.98:0/3585745284 reap_dead start 2024-05-27T00:15:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7fb54e000640 1 -- 172.21.15.98:0/3585745284 shutdown_connections 2024-05-27T00:15:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7fb54e000640 1 -- 172.21.15.98:0/3585745284 >> 172.21.15.98:0/3585745284 conn(0x7fb55c072f90 msgr2=0x7fb55c0c4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 --> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb234149360 con 0x7fb21406fb30 2024-05-27T00:15:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 >> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] conn(0x7f3c9806fa60 msgr2=0x7f3c98071e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cbfe00640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] conn(0x7f3c9806fa60 0x7f3c98071e80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7fb54e000640 1 -- 172.21.15.98:0/3585745284 shutdown_connections 2024-05-27T00:15:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7fb54e000640 1 -- 172.21.15.98:0/3585745284 wait complete. 2024-05-27T00:15:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3c980678c0 msgr2=0x7f3c98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 --> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcae8075cb0 con 0x7fcac806fb40 2024-05-27T00:15:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7fb231600640 1 -- 172.21.15.98:0/981426920 <== osd.5 v2:172.21.15.204:6808/858542033 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb234149360 con 0x7fb21406fb30 2024-05-27T00:15:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cbfe00640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3c980678c0 0x7f3c98069d80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0b1000 tx=0x7f3cac0aa000 comp rx=0 tx=0).stop 2024-05-27T00:15:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cb80c7bc0 msgr2=0x7f3cb8141f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cbfe00640 1 --2- 172.21.15.98:0/3859597685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cb80c7bc0 0x7f3cb8141f50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000e7f0 tx=0x7f3cb000ecb0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cbde00640 1 -- 172.21.15.98:0/3859597685 reap_dead start 2024-05-27T00:15:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 shutdown_connections 2024-05-27T00:15:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 >> 172.21.15.98:0/3859597685 conn(0x7f3cb80cc200 msgr2=0x7f3cb80ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.489+0000 7fcae4c00640 1 -- 172.21.15.98:0/1505591963 <== osd.4 v2:172.21.15.204:6800/4073643582 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcae8075cb0 con 0x7fcac806fb40 2024-05-27T00:15:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 shutdown_connections 2024-05-27T00:15:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] conn(0x7fb21406fb30 msgr2=0x7fb214071f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23c000640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] conn(0x7fb21406fb30 0x7fb214071f50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7f3cbfe00640 1 -- 172.21.15.98:0/3859597685 wait complete. 2024-05-27T00:15:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb214067990 msgr2=0x7fb214069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23c000640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb214067990 0x7fb214069e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb234156430 tx=0x7fb228008040 comp rx=0 tx=0).stop 2024-05-27T00:15:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23406e660 msgr2=0x7fb234200f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23c000640 1 --2- 172.21.15.98:0/981426920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb23406e660 0x7fb234200f00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb2200af000 tx=0x7fb22001bab0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23a000640 1 -- 172.21.15.98:0/981426920 reap_dead start 2024-05-27T00:15:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 shutdown_connections 2024-05-27T00:15:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 >> 172.21.15.98:0/981426920 conn(0x7fb2340cc200 msgr2=0x7fb234155100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 shutdown_connections 2024-05-27T00:15:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fb23c000640 1 -- 172.21.15.98:0/981426920 wait complete. 2024-05-27T00:15:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 --> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5530075cb0 con 0x7f551406fac0 2024-05-27T00:15:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7f552a000640 1 -- 172.21.15.98:0/3984233427 <== osd.3 v2:172.21.15.98:6817/1805101132 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5530075cb0 con 0x7f551406fac0 2024-05-27T00:15:32.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] conn(0x7fcac806fb40 msgr2=0x7fcac8071f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fcaefc00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] conn(0x7fcac806fb40 0x7fcac8071f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcac80678c0 msgr2=0x7fcac8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fcaefc00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcac80678c0 0x7fcac8069d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcad8006fd0 tx=0x7fcad8008040 comp rx=0 tx=0).stop 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae81f5110 msgr2=0x7fcae80c3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fcaefc00640 1 --2- 172.21.15.98:0/1505591963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae81f5110 0x7fcae80c3c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcae0007da0 tx=0x7fcae000e4b0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.493+0000 7fcaedc00640 1 -- 172.21.15.98:0/1505591963 reap_dead start 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 shutdown_connections 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 >> 172.21.15.98:0/1505591963 conn(0x7fcae80cc200 msgr2=0x7fcae80cb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 shutdown_connections 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7fcaefc00640 1 -- 172.21.15.98:0/1505591963 wait complete. 2024-05-27T00:15:32.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f7b83400640 1 -- 172.21.15.98:0/238353670 --> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7b64002de0 con 0x7f7b7006fec0 2024-05-27T00:15:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f07d7400640 1 -- 172.21.15.98:0/3668629299 --> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f07c0002de0 con 0x7f07cc06fb70 2024-05-27T00:15:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 >> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] conn(0x7f551406fac0 msgr2=0x7f5514071ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f7b8aa00640 1 -- 172.21.15.98:0/238353670 <== osd.7 v2:172.21.15.204:6816/874852314 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7b64002de0 con 0x7f7b7006fec0 2024-05-27T00:15:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f5538800640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] conn(0x7f551406fac0 0x7f5514071ee0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f55140678c0 msgr2=0x7f5514069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f5538800640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f55140678c0 0x7f5514069d80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f552c003b90 tx=0x7f552c009890 comp rx=0 tx=0).stop 2024-05-27T00:15:32.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 msgr2=0x7f55300c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f5538800640 1 --2- 172.21.15.98:0/3984233427 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55301590f0 0x7f55300c4f20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f552401d340 tx=0x7f552401ba30 comp rx=0 tx=0).stop 2024-05-27T00:15:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f5536800640 1 -- 172.21.15.98:0/3984233427 reap_dead start 2024-05-27T00:15:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f7b83400640 1 -- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] conn(0x7f7b7006fec0 msgr2=0x7f7b700722e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 shutdown_connections 2024-05-27T00:15:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 >> 172.21.15.98:0/3984233427 conn(0x7f55300cc200 msgr2=0x7f553015e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f7b83400640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] conn(0x7f7b7006fec0 0x7f7b700722e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f7b83400640 1 -- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b70067cc0 msgr2=0x7f7b7006a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.497+0000 7f7b83400640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b70067cc0 0x7f7b7006a180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b84007fd0 tx=0x7f7b84009040 comp rx=0 tx=0).stop 2024-05-27T00:15:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f7b83400640 1 -- 172.21.15.98:0/238353670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b941590f0 msgr2=0x7f7b9414f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f7b83400640 1 --2- 172.21.15.98:0/238353670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b941590f0 0x7f7b9414f9b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00c780 tx=0x7f7b7c00cc40 comp rx=0 tx=0).stop 2024-05-27T00:15:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f7b92800640 1 -- 172.21.15.98:0/238353670 reap_dead start 2024-05-27T00:15:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 shutdown_connections 2024-05-27T00:15:32.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f5538800640 1 -- 172.21.15.98:0/3984233427 wait complete. 2024-05-27T00:15:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f07e2000640 1 -- 172.21.15.98:0/3668629299 <== osd.6 v2:172.21.15.204:6824/469091443 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f07c0002de0 con 0x7f07cc06fb70 2024-05-27T00:15:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f7b83400640 1 -- 172.21.15.98:0/238353670 shutdown_connections 2024-05-27T00:15:32.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f7b83400640 1 -- 172.21.15.98:0/238353670 >> 172.21.15.98:0/238353670 conn(0x7f7b940cc200 msgr2=0x7f7b940c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f07d7400640 1 -- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] conn(0x7f07cc06fb70 msgr2=0x7f07cc071f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f07d7400640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] conn(0x7f07cc06fb70 0x7f07cc071f90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f07d7400640 1 -- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07cc0678f0 msgr2=0x7f07cc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f07d7400640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07cc0678f0 0x7f07cc069db0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f07dc005890 tx=0x7f07dc009040 comp rx=0 tx=0).stop 2024-05-27T00:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f07d7400640 1 -- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07e40d5c90 msgr2=0x7f07e40d8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.501+0000 7f07d7400640 1 --2- 172.21.15.98:0/3668629299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07e40d5c90 0x7f07e40d8510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f07ec0505c0 tx=0x7f07ec069a70 comp rx=0 tx=0).stop 2024-05-27T00:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.505+0000 7f07f0a00640 1 -- 172.21.15.98:0/3668629299 reap_dead start 2024-05-27T00:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.505+0000 7f07d7400640 1 -- 172.21.15.98:0/3668629299 shutdown_connections 2024-05-27T00:15:32.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.505+0000 7f07d7400640 1 -- 172.21.15.98:0/3668629299 >> 172.21.15.98:0/3668629299 conn(0x7f07e4050360 msgr2=0x7f07e404a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.505+0000 7f7b83400640 1 -- 172.21.15.98:0/238353670 shutdown_connections 2024-05-27T00:15:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.505+0000 7f7b83400640 1 -- 172.21.15.98:0/238353670 wait complete. 2024-05-27T00:15:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.505+0000 7f07d7400640 1 -- 172.21.15.98:0/3668629299 shutdown_connections 2024-05-27T00:15:32.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.505+0000 7f07d7400640 1 -- 172.21.15.98:0/3668629299 wait complete. 2024-05-27T00:15:32.535 INFO:teuthology.orchestra.run.smithi098.stdout:60129542152 2024-05-27T00:15:32.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-27T00:15:32.536 INFO:teuthology.orchestra.run.smithi098.stdout:60129542151 2024-05-27T00:15:32.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-27T00:15:32.538 INFO:teuthology.orchestra.run.smithi098.stdout:60129542151 2024-05-27T00:15:32.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-27T00:15:32.541 INFO:teuthology.orchestra.run.smithi098.stdout:60129542151 2024-05-27T00:15:32.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-27T00:15:32.543 INFO:teuthology.orchestra.run.smithi098.stdout:60129542152 2024-05-27T00:15:32.544 INFO:teuthology.orchestra.run.smithi098.stdout:60129542153 2024-05-27T00:15:32.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-27T00:15:32.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-27T00:15:32.555 INFO:teuthology.orchestra.run.smithi098.stdout:60129542152 2024-05-27T00:15:32.555 INFO:teuthology.orchestra.run.smithi098.stdout:60129542151 2024-05-27T00:15:32.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-27T00:15:32.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-27T00:15:32.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.745+0000 7f2520800640 1 Processor -- start 2024-05-27T00:15:32.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.749+0000 7f2520800640 1 -- start start 2024-05-27T00:15:32.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.749+0000 7f2520800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f25180c1a60 0x7f2518159e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.749+0000 7f251d400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2518070b60 0x7f251815f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43374/0 (socket says 172.21.15.98:43374) 2024-05-27T00:15:32.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.749+0000 7f251d400640 1 -- 172.21.15.98:0/884969380 learned_addr learned my addr 172.21.15.98:0/884969380 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:32.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.749+0000 7f2520800640 1 -- 172.21.15.98:0/884969380 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25180c7630 con 0x7f2518070b60 2024-05-27T00:15:32.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.749+0000 7f2520800640 1 -- 172.21.15.98:0/884969380 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25180c81d0 con 0x7f2518150b30 2024-05-27T00:15:32.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.753+0000 7f251e800640 1 --2- 172.21.15.98:0/884969380 >> v2:172.21.15.204:3300/0 conn(0x7f25180c1a60 0x7f2518159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.753+0000 7f2520800640 1 -- 172.21.15.98:0/884969380 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f25180c7100 con 0x7f25180c1a60 2024-05-27T00:15:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.753+0000 7f251ca00640 1 -- 172.21.15.98:0/884969380 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776246017 0 0) 0x7f25180c7630 con 0x7f2518070b60 2024-05-27T00:15:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.753+0000 7f251ca00640 1 -- 172.21.15.98:0/884969380 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2500003530 con 0x7f2518070b60 2024-05-27T00:15:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.753+0000 7f251ca00640 1 -- 172.21.15.98:0/884969380 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2363737320 0 0) 0x7f25180c81d0 con 0x7f2518150b30 2024-05-27T00:15:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.753+0000 7f251ca00640 1 -- 172.21.15.98:0/884969380 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25180c7630 con 0x7f2518150b30 2024-05-27T00:15:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.753+0000 7f251e800640 1 -- 172.21.15.98:0/884969380 >> v1:172.21.15.204:6789/0 conn(0x7f2518150b30 legacy=0x7f25180c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.753+0000 7f251e800640 1 -- 172.21.15.98:0/884969380 >> v1:172.21.15.98:6789/0 conn(0x7f2518070b60 legacy=0x7f251815f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.753+0000 7f251e800640 1 -- 172.21.15.98:0/884969380 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25180c7350 con 0x7f25180c1a60 2024-05-27T00:15:32.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.757+0000 7f251e800640 1 --2- 172.21.15.98:0/884969380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25180c1a60 0x7f2518159e70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f251400c2e0 tx=0x7f251401de50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e68b2a115ac49a87 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.757+0000 7f251ca00640 1 -- 172.21.15.98:0/884969380 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25140b2070 con 0x7f25180c1a60 2024-05-27T00:15:32.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.757+0000 7f251ca00640 1 -- 172.21.15.98:0/884969380 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251400a2d0 con 0x7f25180c1a60 2024-05-27T00:15:32.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.757+0000 7f251ca00640 1 -- 172.21.15.98:0/884969380 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251400a6a0 con 0x7f25180c1a60 2024-05-27T00:15:32.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.761+0000 7f2520800640 1 -- 172.21.15.98:0/884969380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25180c1a60 msgr2=0x7f2518159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.761+0000 7f2520800640 1 --2- 172.21.15.98:0/884969380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25180c1a60 0x7f2518159e70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f251400c2e0 tx=0x7f251401de50 comp rx=0 tx=0).stop 2024-05-27T00:15:32.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.761+0000 7f2520800640 1 -- 172.21.15.98:0/884969380 shutdown_connections 2024-05-27T00:15:32.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.761+0000 7f2520800640 1 --2- 172.21.15.98:0/884969380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25180c1a60 0x7f2518159e70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.761+0000 7f2520800640 1 -- 172.21.15.98:0/884969380 >> 172.21.15.98:0/884969380 conn(0x7f251806e750 msgr2=0x7f251806eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.761+0000 7f2520800640 1 -- 172.21.15.98:0/884969380 shutdown_connections 2024-05-27T00:15:32.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.761+0000 7f2520800640 1 -- 172.21.15.98:0/884969380 wait complete. 2024-05-27T00:15:32.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.761+0000 7f2520800640 1 Processor -- start 2024-05-27T00:15:32.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.765+0000 7f2520800640 1 -- start start 2024-05-27T00:15:32.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2520800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2518070b60 0x7f25180c52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f251de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2518070b60 0x7f25180c52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f251de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2518070b60 0x7f25180c52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53668/0 (socket says 172.21.15.98:53668) 2024-05-27T00:15:32.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f251de00640 1 -- 172.21.15.98:0/1995365115 learned_addr learned my addr 172.21.15.98:0/1995365115 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2520800640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25180c1a60 0x7f2518166180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2520800640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2518150b30 0x7f251814cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2520800640 1 -- 172.21.15.98:0/1995365115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25180c7e20 con 0x7f2518070b60 2024-05-27T00:15:32.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2520800640 1 -- 172.21.15.98:0/1995365115 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f25180c15c0 con 0x7f25180c1a60 2024-05-27T00:15:32.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2520800640 1 -- 172.21.15.98:0/1995365115 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f25180c5f90 con 0x7f2518150b30 2024-05-27T00:15:32.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f251de00640 1 -- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2518150b30 msgr2=0x7f251814cc40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T00:15:32.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f251de00640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2518150b30 0x7f251814cc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f251de00640 1 -- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25180c1a60 msgr2=0x7f2518166180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T00:15:32.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f251de00640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25180c1a60 0x7f2518166180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f251de00640 1 -- 172.21.15.98:0/1995365115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25181f8b60 con 0x7f2518070b60 2024-05-27T00:15:32.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f251de00640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2518070b60 0x7f25180c52f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f250c00c480 tx=0x7f250c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2512000640 1 -- 172.21.15.98:0/1995365115 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c0052d0 con 0x7f2518070b60 2024-05-27T00:15:32.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2512000640 1 -- 172.21.15.98:0/1995365115 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f250c005470 con 0x7f2518070b60 2024-05-27T00:15:32.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2512000640 1 -- 172.21.15.98:0/1995365115 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c012870 con 0x7f2518070b60 2024-05-27T00:15:32.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2520800640 1 -- 172.21.15.98:0/1995365115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25181650e0 con 0x7f2518070b60 2024-05-27T00:15:32.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.769+0000 7f2520800640 1 -- 172.21.15.98:0/1995365115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25181656a0 con 0x7f2518070b60 2024-05-27T00:15:32.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.773+0000 7f2520800640 1 -- 172.21.15.98:0/1995365115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2518075c40 con 0x7f2518070b60 2024-05-27T00:15:32.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.773+0000 7f1002000640 1 Processor -- start 2024-05-27T00:15:32.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.773+0000 7f1002000640 1 -- start start 2024-05-27T00:15:32.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.773+0000 7f2512000640 1 -- 172.21.15.98:0/1995365115 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f250c012a10 con 0x7f2518070b60 2024-05-27T00:15:32.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7fa4ac000640 1 Processor -- start 2024-05-27T00:15:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f1002000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0ffc0c7ad0 0x7f0ffc15f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f0ffb400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0ffc158ed0 0x7f0ffc06ec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43378/0 (socket says 172.21.15.98:43378) 2024-05-27T00:15:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7fa4ac000640 1 -- start start 2024-05-27T00:15:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f1002000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0ffc0c1610 0x7f0ffc162b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f1002000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ffc0c7050 con 0x7f0ffc158ed0 2024-05-27T00:15:32.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f1002000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0ffc1630d0 con 0x7f0ffc0c7ad0 2024-05-27T00:15:32.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f1002000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f0ffc15f2d0 con 0x7f0ffc0c1610 2024-05-27T00:15:32.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f0ffb400640 1 -- 172.21.15.98:0/1588289744 learned_addr learned my addr 172.21.15.98:0/1588289744 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:32.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f2180000640 1 Processor -- start 2024-05-27T00:15:32.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f2180000640 1 -- start start 2024-05-27T00:15:32.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f0ffaa00640 1 --2- 172.21.15.98:0/1588289744 >> v2:172.21.15.98:3300/0 conn(0x7f0ffc0c7ad0 0x7f0ffc15f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7fa4ac000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa4a40c19c0 0x7fa4a40c7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f2512000640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24fc0679c0 0x7f24fc069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f2512000640 1 -- 172.21.15.98:0/1995365115 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f250c085970 con 0x7f2518070b60 2024-05-27T00:15:32.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f2180000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f21781590f0 0x7f21780c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f2180000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2178162db0 0x7f21780c1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f2180000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f217815fb30 con 0x7f21780c1b50 2024-05-27T00:15:32.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f2180000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f21780c7700 con 0x7f21781590f0 2024-05-27T00:15:32.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7f2180000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f217815edb0 con 0x7f2178162db0 2024-05-27T00:15:32.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7fa4ac000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 0x7fa4a415e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7fa4ac000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4a415b740 con 0x7fa4a4154ae0 2024-05-27T00:15:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7fa4ac000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa4a40c0eb0 con 0x7fa4a40c19c0 2024-05-27T00:15:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7fa4ac000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a415de60 con 0x7fa4a415b310 2024-05-27T00:15:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7fa4a9600640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa4a4154ae0 0x7fa4a406ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:42876/0 (socket says 172.21.15.98:42876) 2024-05-27T00:15:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7fa4a9600640 1 -- 172.21.15.98:0/2364327658 learned_addr learned my addr 172.21.15.98:0/2364327658 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.777+0000 7fa4aa000640 1 --2- 172.21.15.98:0/2364327658 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 0x7fa4a415e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f0ffaa00640 1 -- 172.21.15.98:0/1588289744 >> v2:172.21.15.204:3300/0 conn(0x7f0ffc0c1610 msgr2=0x7f0ffc162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f0ffbe00640 1 --2- 172.21.15.98:0/1588289744 >> v2:172.21.15.204:3300/0 conn(0x7f0ffc0c1610 0x7f0ffc162b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f217cc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2178162db0 0x7f21780c1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f251d400640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24fc0679c0 0x7f24fc069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f217cc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2178162db0 0x7f21780c1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:34834/0 (socket says 172.21.15.98:34834) 2024-05-27T00:15:32.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f217cc00640 1 -- 172.21.15.98:0/433860945 learned_addr learned my addr 172.21.15.98:0/433860945 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f0ffaa00640 1 --2- 172.21.15.98:0/1588289744 >> v2:172.21.15.204:3300/0 conn(0x7f0ffc0c1610 0x7f0ffc162b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f251d400640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24fc0679c0 0x7f24fc069e80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2508007fd0 tx=0x7f2508007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f217d600640 1 --2- 172.21.15.98:0/433860945 >> v2:172.21.15.98:3300/0 conn(0x7f21781590f0 0x7f21780c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f2177e00640 1 -- 172.21.15.98:0/433860945 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130119990 0 0) 0x7f217815fb30 con 0x7f21780c1b50 2024-05-27T00:15:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f2177e00640 1 -- 172.21.15.98:0/433860945 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2160003530 con 0x7f21780c1b50 2024-05-27T00:15:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f217d600640 1 -- 172.21.15.98:0/433860945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2178162db0 msgr2=0x7f21780c1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f217d600640 1 --2- 172.21.15.98:0/433860945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2178162db0 0x7f21780c1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f217d600640 1 -- 172.21.15.98:0/433860945 >> v1:172.21.15.204:6789/0 conn(0x7f21780c1b50 legacy=0x7f217806e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7f217d600640 1 -- 172.21.15.98:0/433860945 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21780c7420 con 0x7f21781590f0 2024-05-27T00:15:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7fa4a3e00640 1 -- 172.21.15.98:0/2364327658 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340647547 0 0) 0x7fa490001120 con 0x7fa4a4154ae0 2024-05-27T00:15:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7fa4a3e00640 1 -- 172.21.15.98:0/2364327658 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa48c003530 con 0x7fa4a4154ae0 2024-05-27T00:15:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7fa4a8c00640 1 --2- 172.21.15.98:0/2364327658 >> v2:172.21.15.98:3300/0 conn(0x7fa4a40c19c0 0x7fa4a40c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7fa4aa000640 1 -- 172.21.15.98:0/2364327658 >> v2:172.21.15.98:3300/0 conn(0x7fa4a40c19c0 msgr2=0x7fa4a40c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7fa4aa000640 1 --2- 172.21.15.98:0/2364327658 >> v2:172.21.15.98:3300/0 conn(0x7fa4a40c19c0 0x7fa4a40c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7fa4aa000640 1 -- 172.21.15.98:0/2364327658 >> v1:172.21.15.204:6789/0 conn(0x7fa4a4154ae0 legacy=0x7fa4a406ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7fa4aa000640 1 -- 172.21.15.98:0/2364327658 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a40c1100 con 0x7fa4a415b310 2024-05-27T00:15:32.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.781+0000 7fa4a8c00640 1 --2- 172.21.15.98:0/2364327658 >> v2:172.21.15.98:3300/0 conn(0x7fa4a40c19c0 0x7fa4a40c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.785+0000 7f0ffaa00640 1 -- 172.21.15.98:0/1588289744 >> v1:172.21.15.98:6789/0 conn(0x7f0ffc158ed0 legacy=0x7f0ffc06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.785+0000 7f0ffaa00640 1 -- 172.21.15.98:0/1588289744 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ffc0c72f0 con 0x7f0ffc0c7ad0 2024-05-27T00:15:32.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.785+0000 7f0ffbe00640 1 --2- 172.21.15.98:0/1588289744 >> v2:172.21.15.204:3300/0 conn(0x7f0ffc0c1610 0x7f0ffc162b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:32.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.785+0000 7f217d600640 1 --2- 172.21.15.98:0/433860945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21781590f0 0x7f21780c7b10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f216c0097c0 tx=0x7f216c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de924abd328eda4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.785+0000 7f2512000640 1 -- 172.21.15.98:0/1995365115 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f250c056520 con 0x7f2518070b60 2024-05-27T00:15:32.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.785+0000 7fa4aa000640 1 --2- 172.21.15.98:0/2364327658 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 0x7fa4a415e8b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009d90 tx=0x7fa49c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ecf429cd700efd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.785+0000 7f2177e00640 1 -- 172.21.15.98:0/433860945 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c0b5070 con 0x7f21781590f0 2024-05-27T00:15:32.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.785+0000 7fa4a3e00640 1 -- 172.21.15.98:0/2364327658 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0b2070 con 0x7fa4a415b310 2024-05-27T00:15:32.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.785+0000 7fa4a3e00640 1 -- 172.21.15.98:0/2364327658 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49c0a5940 con 0x7fa4a415b310 2024-05-27T00:15:32.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f2177e00640 1 -- 172.21.15.98:0/433860945 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f216c0a8990 con 0x7f21781590f0 2024-05-27T00:15:32.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f2177e00640 1 -- 172.21.15.98:0/433860945 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c0a8d60 con 0x7f21781590f0 2024-05-27T00:15:32.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f0ffaa00640 1 --2- 172.21.15.98:0/1588289744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c7ad0 0x7f0ffc15f760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc15f240 tx=0x7f0fec01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6213470238b26e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7fa4ac000640 1 -- 172.21.15.98:0/2364327658 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 msgr2=0x7fa4a415e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7fa4a3e00640 1 -- 172.21.15.98:0/2364327658 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0b2070 con 0x7fa4a415b310 2024-05-27T00:15:32.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7fa4ac000640 1 --2- 172.21.15.98:0/2364327658 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 0x7fa4a415e8b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009d90 tx=0x7fa49c01c970 comp rx=0 tx=0).stop 2024-05-27T00:15:32.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7fa4ac000640 1 -- 172.21.15.98:0/2364327658 shutdown_connections 2024-05-27T00:15:32.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7fa4ac000640 1 --2- 172.21.15.98:0/2364327658 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 0x7fa4a415e8b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f2180000640 1 -- 172.21.15.98:0/433860945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21781590f0 msgr2=0x7f21780c7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7fa4ac000640 1 --2- 172.21.15.98:0/2364327658 >> v2:172.21.15.98:3300/0 conn(0x7fa4a40c19c0 0x7fa4a40c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7fa4ac000640 1 -- 172.21.15.98:0/2364327658 >> 172.21.15.98:0/2364327658 conn(0x7fa4a40cc200 msgr2=0x7fa4a40cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f2180000640 1 --2- 172.21.15.98:0/433860945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21781590f0 0x7f21780c7b10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f216c0097c0 tx=0x7f216c01ec90 comp rx=0 tx=0).stop 2024-05-27T00:15:32.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f0ffa000640 1 -- 172.21.15.98:0/1588289744 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fec0b1070 con 0x7f0ffc0c7ad0 2024-05-27T00:15:32.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7fa4ac000640 1 -- 172.21.15.98:0/2364327658 shutdown_connections 2024-05-27T00:15:32.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7fa4ac000640 1 -- 172.21.15.98:0/2364327658 wait complete. 2024-05-27T00:15:32.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f0ffa000640 1 -- 172.21.15.98:0/1588289744 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fec0ac5f0 con 0x7f0ffc0c7ad0 2024-05-27T00:15:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f0ffa000640 1 -- 172.21.15.98:0/1588289744 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fec0b1070 con 0x7f0ffc0c7ad0 2024-05-27T00:15:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f1002000640 1 -- 172.21.15.98:0/1588289744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c7ad0 msgr2=0x7f0ffc15f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f1002000640 1 --2- 172.21.15.98:0/1588289744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c7ad0 0x7f0ffc15f760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc15f240 tx=0x7f0fec01c480 comp rx=0 tx=0).stop 2024-05-27T00:15:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f1002000640 1 -- 172.21.15.98:0/1588289744 shutdown_connections 2024-05-27T00:15:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f1002000640 1 --2- 172.21.15.98:0/1588289744 >> v2:172.21.15.204:3300/0 conn(0x7f0ffc0c1610 0x7f0ffc162b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f1002000640 1 --2- 172.21.15.98:0/1588289744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c7ad0 0x7f0ffc15f760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f1002000640 1 -- 172.21.15.98:0/1588289744 >> 172.21.15.98:0/1588289744 conn(0x7f0ffc0cc200 msgr2=0x7f0ffc0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f1002000640 1 -- 172.21.15.98:0/1588289744 shutdown_connections 2024-05-27T00:15:32.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f2180000640 1 -- 172.21.15.98:0/433860945 shutdown_connections 2024-05-27T00:15:32.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f2180000640 1 --2- 172.21.15.98:0/433860945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2178162db0 0x7f21780c1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f2180000640 1 --2- 172.21.15.98:0/433860945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21781590f0 0x7f21780c7b10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f2180000640 1 -- 172.21.15.98:0/433860945 >> 172.21.15.98:0/433860945 conn(0x7f21780cc200 msgr2=0x7f21780cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f2180000640 1 -- 172.21.15.98:0/433860945 shutdown_connections 2024-05-27T00:15:32.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.789+0000 7f2180000640 1 -- 172.21.15.98:0/433860945 wait complete. 2024-05-27T00:15:32.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7f1002000640 1 -- 172.21.15.98:0/1588289744 wait complete. 2024-05-27T00:15:32.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4ac000640 1 Processor -- start 2024-05-27T00:15:32.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4ac000640 1 -- start start 2024-05-27T00:15:32.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa81d200640 1 Processor -- start 2024-05-27T00:15:32.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7f2180000640 1 Processor -- start 2024-05-27T00:15:32.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa81d200640 1 -- start start 2024-05-27T00:15:32.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7f2180000640 1 -- start start 2024-05-27T00:15:32.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4ac000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4a40c19c0 0x7fa4a4159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4ac000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4a4154ae0 0x7fa4a4151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4ac000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 0x7fa4a4152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7f1002000640 1 Processor -- start 2024-05-27T00:15:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4ac000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a406f410 con 0x7fa4a4154ae0 2024-05-27T00:15:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4ac000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a4154ec0 con 0x7fa4a40c19c0 2024-05-27T00:15:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4ac000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a406f590 con 0x7fa4a415b310 2024-05-27T00:15:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4a8c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4a4154ae0 0x7fa4a4151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa81d200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa818152930 0x7fa8180c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4a8c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4a4154ae0 0x7fa4a4151cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53700/0 (socket says 172.21.15.98:53700) 2024-05-27T00:15:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa4a8c00640 1 -- 172.21.15.98:0/3206709806 learned_addr learned my addr 172.21.15.98:0/3206709806 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.793+0000 7fa81d200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8180ca550 con 0x7fa8180c3020 2024-05-27T00:15:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa81d200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8180c51e0 con 0x7fa81806e450 2024-05-27T00:15:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa81d200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa8180ce3a0 con 0x7fa818152930 2024-05-27T00:15:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa4aa000640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 0x7fa4a4152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa816800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa8180c3020 0x7fa81806e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43392/0 (socket says 172.21.15.98:43392) 2024-05-27T00:15:32.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa816800640 1 -- 172.21.15.98:0/1945346731 learned_addr learned my addr 172.21.15.98:0/1945346731 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:32.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f2180000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f21780c1b50 0x7f21781e81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f0b72200640 1 Processor -- start 2024-05-27T00:15:32.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f2180000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21781590f0 0x7f21781f5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa4a8c00640 1 -- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 msgr2=0x7fa4a4152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa4a8c00640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 0x7fa4a4152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa4a8c00640 1 -- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4a40c19c0 msgr2=0x7fa4a4159820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T00:15:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa4a8c00640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4a40c19c0 0x7fa4a4159820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa4a8c00640 1 -- 172.21.15.98:0/3206709806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a41f83f0 con 0x7fa4a4154ae0 2024-05-27T00:15:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f0b72200640 1 -- start start 2024-05-27T00:15:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f217d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f21780c1b50 0x7f21781e81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f217d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f21780c1b50 0x7f21781e81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:34854/0 (socket says 172.21.15.98:34854) 2024-05-27T00:15:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 164189515 0 0) 0x7fa8180ca550 con 0x7fa8180c3020 2024-05-27T00:15:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f217d600640 1 -- 172.21.15.98:0/2816848726 learned_addr learned my addr 172.21.15.98:0/2816848726 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7fc003530 con 0x7fa8180c3020 2024-05-27T00:15:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f217cc00640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21781590f0 0x7f21781f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f2180000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2178162db0 0x7f21781f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa4a8c00640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4a4154ae0 0x7fa4a4151cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa49800e980 tx=0x7fa49800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f217815f610 con 0x7f2178162db0 2024-05-27T00:15:32.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f217815f330 con 0x7f21781590f0 2024-05-27T00:15:32.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f217e000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2178162db0 0x7f21781f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f21780c8050 con 0x7f21780c1b50 2024-05-27T00:15:32.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f1002000640 1 -- start start 2024-05-27T00:15:32.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa815e00640 1 --2- 172.21.15.98:0/1945346731 >> v2:172.21.15.204:3300/0 conn(0x7fa818152930 0x7fa8180c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3009718104 0 0) 0x7fa7fc003530 con 0x7fa8180c3020 2024-05-27T00:15:32.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 >> v2:172.21.15.204:3300/0 conn(0x7fa818152930 msgr2=0x7fa8180c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa814a00640 1 --2- 172.21.15.98:0/1945346731 >> v2:172.21.15.204:3300/0 conn(0x7fa818152930 0x7fa8180c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 >> v1:172.21.15.204:6789/0 conn(0x7fa81806e450 legacy=0x7fa8180c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8180c8aa0 con 0x7fa8180c3020 2024-05-27T00:15:32.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f0b72200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c154a00 0x7f0b6c0b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f0b72200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0b6c0d0070 0x7f0b6c0d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f0b72200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b6c0b3f20 con 0x7f0b6c154de0 2024-05-27T00:15:32.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f0b72200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f0b6c0b4200 con 0x7f0b6c0d0070 2024-05-27T00:15:32.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7f0b72200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6c0b44e0 con 0x7f0b6c154a00 2024-05-27T00:15:32.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f1002000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c1610 0x7f0ffc15a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0b6be00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0b6c0d0070 0x7f0b6c0d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.797+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fa808001a00 con 0x7fa8180c3020 2024-05-27T00:15:32.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa4a1600640 1 -- 172.21.15.98:0/3206709806 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498017070 con 0x7fa4a4154ae0 2024-05-27T00:15:32.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa4a1600640 1 -- 172.21.15.98:0/3206709806 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49800c9a0 con 0x7fa4a4154ae0 2024-05-27T00:15:32.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa4a1600640 1 -- 172.21.15.98:0/3206709806 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49800cd50 con 0x7fa4a4154ae0 2024-05-27T00:15:32.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa4ac000640 1 -- 172.21.15.98:0/3206709806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a4153360 con 0x7fa4a4154ae0 2024-05-27T00:15:32.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f1002000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ffc158ed0 0x7f0ffc0ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f1002000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ffc155150 0x7f0ffc1fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f1002000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ffc06f390 con 0x7f0ffc0c1610 2024-05-27T00:15:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f1002000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0ffc1592b0 con 0x7f0ffc158ed0 2024-05-27T00:15:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f1002000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0ffc06f510 con 0x7f0ffc155150 2024-05-27T00:15:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f217e000640 1 -- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f21780c1b50 msgr2=0x7f21781e81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f217e000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f21780c1b50 0x7f21781e81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c1610 0x7f0ffc15a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c1610 0x7f0ffc15a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53722/0 (socket says 172.21.15.98:53722) 2024-05-27T00:15:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffb400640 1 -- 172.21.15.98:0/2048138786 learned_addr learned my addr 172.21.15.98:0/2048138786 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 >> v1:172.21.15.98:6789/0 conn(0x7fa8180c3020 legacy=0x7fa81806e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f217e000640 1 -- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21781590f0 msgr2=0x7f21781f5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f217e000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21781590f0 0x7f21781f5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffbe00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ffc155150 0x7f0ffc1fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffaa00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ffc158ed0 0x7f0ffc0ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f217e000640 1 -- 172.21.15.98:0/2816848726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2178159a50 con 0x7f2178162db0 2024-05-27T00:15:32.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa814a00640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc0039c0 0x7fa7fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa4ac000640 1 -- 172.21.15.98:0/3206709806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a4153920 con 0x7fa4a4154ae0 2024-05-27T00:15:32.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0b6be00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0b6c0d0070 0x7f0b6c0d04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57756/0 (socket says 172.21.15.98:57756) 2024-05-27T00:15:32.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0b6be00640 1 -- 172.21.15.98:0/1422649710 learned_addr learned my addr 172.21.15.98:0/1422649710 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0b6b400640 1 --2- 172.21.15.98:0/1422649710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c154a00 0x7f0b6c0b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffbe00640 1 -- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ffc158ed0 msgr2=0x7f0ffc0ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa814a00640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa7fc004810 0x7fa7fc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa814a00640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa7fc0051d0 0x7fa7fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8180ce3a0 con 0x7fa7fc0039c0 2024-05-27T00:15:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa7fc00c100 con 0x7fa7fc0051d0 2024-05-27T00:15:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa7fc00c300 con 0x7fa7fc004810 2024-05-27T00:15:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa815e00640 1 --2- 172.21.15.98:0/1945346731 >> v2:172.21.15.204:3300/0 conn(0x7fa818152930 0x7fa8180c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa496a00640 1 -- 172.21.15.98:0/3206709806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a4153b20 con 0x7fa4a4154ae0 2024-05-27T00:15:32.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffbe00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ffc158ed0 0x7f0ffc0ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffbe00640 1 -- 172.21.15.98:0/2048138786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c1610 msgr2=0x7f0ffc15a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffbe00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c1610 0x7f0ffc15a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffbe00640 1 -- 172.21.15.98:0/2048138786 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ffc15ac30 con 0x7f0ffc155150 2024-05-27T00:15:32.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffb400640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c1610 0x7f0ffc15a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:32.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0ffbe00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ffc155150 0x7f0ffc1fb2f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400ce00 tx=0x7f0ff400a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa815e00640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc0039c0 0x7fa7fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa816800640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa7fc0051d0 0x7fa7fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa815400640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa7fc004810 0x7fa7fc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f217e000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2178162db0 0x7f21781f5640 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f21780c8580 tx=0x7f217000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0b6be00640 1 -- 172.21.15.98:0/1422649710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c154a00 msgr2=0x7f0b6c0b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0b6be00640 1 --2- 172.21.15.98:0/1422649710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c154a00 0x7f0b6c0b2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0b6a000640 1 -- 172.21.15.98:0/1422649710 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 140349673 0 0) 0x7f0b6c0b3f20 con 0x7f0b6c154de0 2024-05-27T00:15:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0b6be00640 1 -- 172.21.15.98:0/1422649710 >> v1:172.21.15.204:6789/0 conn(0x7f0b6c154de0 legacy=0x7f0b6c15b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f0b6be00640 1 -- 172.21.15.98:0/1422649710 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6c0b47c0 con 0x7f0b6c0d0070 2024-05-27T00:15:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa815e00640 1 -- 172.21.15.98:0/1945346731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa7fc004810 msgr2=0x7fa7fc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa815e00640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa7fc004810 0x7fa7fc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa815e00640 1 -- 172.21.15.98:0/1945346731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa7fc0051d0 msgr2=0x7fa7fc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa815e00640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa7fc0051d0 0x7fa7fc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7fa815e00640 1 -- 172.21.15.98:0/1945346731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7fc003530 con 0x7fa7fc0039c0 2024-05-27T00:15:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f2175600640 1 -- 172.21.15.98:0/2816848726 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170003b80 con 0x7f2178162db0 2024-05-27T00:15:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21781582c0 con 0x7f2178162db0 2024-05-27T00:15:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7f0ff3400640 1 -- 172.21.15.98:0/2048138786 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff401a070 con 0x7f0ffc155150 2024-05-27T00:15:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7f0ff3400640 1 -- 172.21.15.98:0/2048138786 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff400d880 con 0x7f0ffc155150 2024-05-27T00:15:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7f0ff3400640 1 -- 172.21.15.98:0/2048138786 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff400dc70 con 0x7f0ffc155150 2024-05-27T00:15:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7f1002000640 1 -- 172.21.15.98:0/2048138786 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ffc154b30 con 0x7f0ffc155150 2024-05-27T00:15:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7f1002000640 1 -- 172.21.15.98:0/2048138786 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ffc159580 con 0x7f0ffc155150 2024-05-27T00:15:32.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.801+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21781f4be0 con 0x7f2178162db0 2024-05-27T00:15:32.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21781f4de0 con 0x7f2178162db0 2024-05-27T00:15:32.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7fa816800640 1 -- 172.21.15.98:0/1945346731 reap_dead start 2024-05-27T00:15:32.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7fa815e00640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc0039c0 0x7fa7fc003de0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa8100098b0 tx=0x7fa81001c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55721c90979b9183 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7f0b6be00640 1 --2- 172.21.15.98:0/1422649710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c0d0070 0x7f0b6c0d04c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009e50 tx=0x7f0b6401f210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=894ac5cb72d6a7b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b6a000640 1 -- 172.21.15.98:0/1422649710 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b640b4070 con 0x7f0b6c0d0070 2024-05-27T00:15:32.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7fa4a1600640 1 -- 172.21.15.98:0/3206709806 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa49801a070 con 0x7fa4a4154ae0 2024-05-27T00:15:32.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.805+0000 7fa4a1600640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4840679c0 0x7fa484069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7fa4a1600640 1 -- 172.21.15.98:0/3206709806 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa498086230 con 0x7fa4a4154ae0 2024-05-27T00:15:32.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b6a000640 1 -- 172.21.15.98:0/1422649710 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b640af400 con 0x7f0b6c0d0070 2024-05-27T00:15:32.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b6a000640 1 -- 172.21.15.98:0/1422649710 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b640af7d0 con 0x7f0b6c0d0070 2024-05-27T00:15:32.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0ff3400640 1 -- 172.21.15.98:0/2048138786 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0ff401d070 con 0x7f0ffc155150 2024-05-27T00:15:32.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7fa4a9600640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4840679c0 0x7fa484069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0ff3400640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0fdc067940 0x7f0fdc069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b72200640 1 -- 172.21.15.98:0/1422649710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c0d0070 msgr2=0x7f0b6c0d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b72200640 1 --2- 172.21.15.98:0/1422649710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c0d0070 0x7f0b6c0d04c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009e50 tx=0x7f0b6401f210 comp rx=0 tx=0).stop 2024-05-27T00:15:32.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7fa4a9600640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4840679c0 0x7fa484069e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa490000f70 tx=0x7fa490007860 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0ff3400640 1 -- 172.21.15.98:0/2048138786 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f0ff4086130 con 0x7f0ffc155150 2024-05-27T00:15:32.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0ffb400640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0fdc067940 0x7f0fdc069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b72200640 1 -- 172.21.15.98:0/1422649710 shutdown_connections 2024-05-27T00:15:32.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b72200640 1 --2- 172.21.15.98:0/1422649710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c0d0070 0x7f0b6c0d04c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b72200640 1 --2- 172.21.15.98:0/1422649710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c154a00 0x7f0b6c0b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b72200640 1 -- 172.21.15.98:0/1422649710 >> 172.21.15.98:0/1422649710 conn(0x7f0b6c067530 msgr2=0x7f0b6c06c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b72200640 1 -- 172.21.15.98:0/1422649710 shutdown_connections 2024-05-27T00:15:32.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0ff0c00640 1 -- 172.21.15.98:0/2048138786 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ffc075cb0 con 0x7f0ffc155150 2024-05-27T00:15:32.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0b72200640 1 -- 172.21.15.98:0/1422649710 wait complete. 2024-05-27T00:15:32.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.809+0000 7f0ffb400640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0fdc067940 0x7f0fdc069e00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8007fd0 tx=0x7f0fe8007f20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b72200640 1 Processor -- start 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b72200640 1 -- start start 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b72200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0d0070 0x7f0b6c15ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b72200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c154a00 0x7f0b6c155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b72200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c154de0 0x7f0b6c0c4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b72200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6c0b37a0 con 0x7f0b6c154a00 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b72200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6c0b3620 con 0x7f0b6c154de0 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b72200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6c0cc7e0 con 0x7f0b6c0d0070 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c154a00 0x7f0b6c155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c154a00 0x7f0b6c155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53740/0 (socket says 172.21.15.98:53740) 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6aa00640 1 -- 172.21.15.98:0/3165930608 learned_addr learned my addr 172.21.15.98:0/3165930608 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6b400640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0d0070 0x7f0b6c15ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6be00640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c154de0 0x7f0b6c0c4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6aa00640 1 -- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0d0070 msgr2=0x7f0b6c15ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6aa00640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0d0070 0x7f0b6c15ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6aa00640 1 -- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c154de0 msgr2=0x7f0b6c0c4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6aa00640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c154de0 0x7f0b6c0c4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6aa00640 1 -- 172.21.15.98:0/3165930608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6c1eff30 con 0x7f0b6c154a00 2024-05-27T00:15:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7f0b6aa00640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c154a00 0x7f0b6c155460 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800c3c0 tx=0x7f0b5800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81000f040 con 0x7fa7fc0039c0 2024-05-27T00:15:32.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8100a6040 con 0x7fa7fc0039c0 2024-05-27T00:15:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7fa814a00640 1 -- 172.21.15.98:0/1945346731 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81000f040 con 0x7fa7fc0039c0 2024-05-27T00:15:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.813+0000 7fa4a1600640 1 -- 172.21.15.98:0/3206709806 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa498056700 con 0x7fa4a4154ae0 2024-05-27T00:15:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7fa81d200640 1 -- 172.21.15.98:0/1945346731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc0039c0 msgr2=0x7fa7fc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7fa81d200640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc0039c0 0x7fa7fc003de0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa8100098b0 tx=0x7fa81001c650 comp rx=0 tx=0).stop 2024-05-27T00:15:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7f0b63400640 1 -- 172.21.15.98:0/3165930608 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58017070 con 0x7f0b6c154a00 2024-05-27T00:15:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7f0b63400640 1 -- 172.21.15.98:0/3165930608 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b58003600 con 0x7f0b6c154a00 2024-05-27T00:15:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7f0b63400640 1 -- 172.21.15.98:0/3165930608 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b580039f0 con 0x7f0b6c154a00 2024-05-27T00:15:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7fa81d200640 1 -- 172.21.15.98:0/1945346731 shutdown_connections 2024-05-27T00:15:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7fa81d200640 1 --2- 172.21.15.98:0/1945346731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc0039c0 0x7fa7fc003de0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7fa81d200640 1 -- 172.21.15.98:0/1945346731 >> 172.21.15.98:0/1945346731 conn(0x7fa81806d680 msgr2=0x7fa81806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7f0b72200640 1 -- 172.21.15.98:0/3165930608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b6c0c55b0 con 0x7f0b6c154a00 2024-05-27T00:15:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7fa81d200640 1 -- 172.21.15.98:0/1945346731 shutdown_connections 2024-05-27T00:15:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7fa81d200640 1 -- 172.21.15.98:0/1945346731 wait complete. 2024-05-27T00:15:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7f0b72200640 1 -- 172.21.15.98:0/3165930608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b6c0c5b70 con 0x7f0b6c154a00 2024-05-27T00:15:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7f0b72200640 1 -- 172.21.15.98:0/3165930608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b6c075cb0 con 0x7f0b6c154a00 2024-05-27T00:15:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.817+0000 7fa81d200640 1 Processor -- start 2024-05-27T00:15:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa81d200640 1 -- start start 2024-05-27T00:15:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7f2175600640 1 -- 172.21.15.98:0/2816848726 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2170003d20 con 0x7f2178162db0 2024-05-27T00:15:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7f2175600640 1 -- 172.21.15.98:0/2816848726 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21700052d0 con 0x7f2178162db0 2024-05-27T00:15:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7f2175600640 1 -- 172.21.15.98:0/2816848726 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f21700054f0 con 0x7f2178162db0 2024-05-27T00:15:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa81d200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa7fc0039c0 0x7fa8180cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa81d200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81806c330 0x7fa8181e9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa81d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa81806c770 0x7fa8181ea0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa81d200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8180ce030 con 0x7fa81806c770 2024-05-27T00:15:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa81d200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa8180c9060 con 0x7fa81806c330 2024-05-27T00:15:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa81d200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa818152d10 con 0x7fa7fc0039c0 2024-05-27T00:15:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7f2175600640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f21580678f0 0x7f2158069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7f2175600640 1 -- 172.21.15.98:0/2816848726 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2170086530 con 0x7f2178162db0 2024-05-27T00:15:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa815e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa7fc0039c0 0x7fa8180cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa816800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa81806c770 0x7fa8181ea0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa815e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa7fc0039c0 0x7fa8180cc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:34910/0 (socket says 172.21.15.98:34910) 2024-05-27T00:15:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa815e00640 1 -- 172.21.15.98:0/2558713170 learned_addr learned my addr 172.21.15.98:0/2558713170 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa815400640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81806c330 0x7fa8181e9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7f217d600640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f21580678f0 0x7f2158069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa816800640 1 -- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa7fc0039c0 msgr2=0x7fa8180cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa816800640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa7fc0039c0 0x7fa8180cc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa816800640 1 -- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81806c330 msgr2=0x7fa8181e9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa816800640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81806c330 0x7fa8181e9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa816800640 1 -- 172.21.15.98:0/2558713170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8180cd500 con 0x7fa81806c770 2024-05-27T00:15:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.821+0000 7fa816800640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa81806c770 0x7fa8181ea0f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa818072600 tx=0x7fa808007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.825+0000 7f217d600640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f21580678f0 0x7f2158069db0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f216c01d340 tx=0x7f216c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.825+0000 7f0ff3400640 1 -- 172.21.15.98:0/2048138786 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f0ff4056790 con 0x7f0ffc155150 2024-05-27T00:15:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.829+0000 7f2175600640 1 -- 172.21.15.98:0/2816848726 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f21781f4de0 con 0x7f2178162db0 2024-05-27T00:15:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.829+0000 7f0b63400640 1 -- 172.21.15.98:0/3165930608 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0b5801a070 con 0x7f0b6c154a00 2024-05-27T00:15:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.833+0000 7f0b63400640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b4c067910 0x7f0b4c069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.833+0000 7f0b63400640 1 -- 172.21.15.98:0/3165930608 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f0b58053950 con 0x7f0b6c154a00 2024-05-27T00:15:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.833+0000 7f0b6b400640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b4c067910 0x7f0b4c069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.833+0000 7f0b6b400640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b4c067910 0x7f0b4c069dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c009b60 tx=0x7f0b5c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.833+0000 7f0b63400640 1 -- 172.21.15.98:0/3165930608 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f0b58021020 con 0x7f0b6c154a00 2024-05-27T00:15:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.837+0000 7fa80e000640 1 -- 172.21.15.98:0/2558713170 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80801a070 con 0x7fa81806c770 2024-05-27T00:15:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.837+0000 7fa80e000640 1 -- 172.21.15.98:0/2558713170 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8080155f0 con 0x7fa81806c770 2024-05-27T00:15:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.837+0000 7fa80e000640 1 -- 172.21.15.98:0/2558713170 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808011040 con 0x7fa81806c770 2024-05-27T00:15:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.837+0000 7fa81d200640 1 -- 172.21.15.98:0/2558713170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8180cd7a0 con 0x7fa81806c770 2024-05-27T00:15:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.837+0000 7fa81d200640 1 -- 172.21.15.98:0/2558713170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8181e90d0 con 0x7fa81806c770 2024-05-27T00:15:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.837+0000 7fa81d200640 1 -- 172.21.15.98:0/2558713170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7f00052a0 con 0x7fa81806c770 2024-05-27T00:15:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.841+0000 7fa80e000640 1 -- 172.21.15.98:0/2558713170 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa80801d070 con 0x7fa81806c770 2024-05-27T00:15:32.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.841+0000 7fa80e000640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7f8067910 0x7fa7f8069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.841+0000 7fa80e000640 1 -- 172.21.15.98:0/2558713170 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa808087940 con 0x7fa81806c770 2024-05-27T00:15:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.841+0000 7fa815e00640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7f8067910 0x7fa7f8069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.841+0000 7faf32000640 1 Processor -- start 2024-05-27T00:15:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.841+0000 7faf32000640 1 -- start start 2024-05-27T00:15:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7fa815e00640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7f8067910 0x7fa7f8069dd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa81001c690 tx=0x7fa8100b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf32000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7faf2c06e660 0x7faf2c150b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf32000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faf2c0c6800 0x7faf2c0c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf32000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf2c151050 con 0x7faf2c06da00 2024-05-27T00:15:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf32000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7faf2c0c71d0 con 0x7faf2c0c6800 2024-05-27T00:15:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf32000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7faf2c06d220 con 0x7faf2c06e660 2024-05-27T00:15:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf2b400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7faf2c06e660 0x7faf2c150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf2b400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7faf2c06e660 0x7faf2c150b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57806/0 (socket says 172.21.15.98:57806) 2024-05-27T00:15:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf2b400640 1 -- 172.21.15.98:0/3989734086 learned_addr learned my addr 172.21.15.98:0/3989734086 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf2a000640 1 -- 172.21.15.98:0/3989734086 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509945608 0 0) 0x7faf2c151050 con 0x7faf2c06da00 2024-05-27T00:15:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf2a000640 1 -- 172.21.15.98:0/3989734086 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf10003530 con 0x7faf2c06da00 2024-05-27T00:15:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf2b400640 1 -- 172.21.15.98:0/3989734086 >> v2:172.21.15.98:3300/0 conn(0x7faf2c0c6800 msgr2=0x7faf2c0c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf2b400640 1 --2- 172.21.15.98:0/3989734086 >> v2:172.21.15.98:3300/0 conn(0x7faf2c0c6800 0x7faf2c0c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf2b400640 1 -- 172.21.15.98:0/3989734086 >> v1:172.21.15.204:6789/0 conn(0x7faf2c06da00 legacy=0x7faf2c06c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.845+0000 7faf2b400640 1 -- 172.21.15.98:0/3989734086 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c156970 con 0x7faf2c06e660 2024-05-27T00:15:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.849+0000 7fa80e000640 1 -- 172.21.15.98:0/2558713170 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa80804f6d0 con 0x7fa81806c770 2024-05-27T00:15:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.857+0000 7faf2b400640 1 --2- 172.21.15.98:0/3989734086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf2c06e660 0x7faf2c150b10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faf2c06f770 tx=0x7faf1401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14920f24aff9eec server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.857+0000 7faf2a000640 1 -- 172.21.15.98:0/3989734086 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf140b5070 con 0x7faf2c06e660 2024-05-27T00:15:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.857+0000 7faf2a000640 1 -- 172.21.15.98:0/3989734086 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf140a8a10 con 0x7faf2c06e660 2024-05-27T00:15:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.857+0000 7faf2a000640 1 -- 172.21.15.98:0/3989734086 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf140b5070 con 0x7faf2c06e660 2024-05-27T00:15:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.861+0000 7faf32000640 1 -- 172.21.15.98:0/3989734086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf2c06e660 msgr2=0x7faf2c150b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.861+0000 7faf32000640 1 --2- 172.21.15.98:0/3989734086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf2c06e660 0x7faf2c150b10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faf2c06f770 tx=0x7faf1401ed30 comp rx=0 tx=0).stop 2024-05-27T00:15:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.861+0000 7faf32000640 1 -- 172.21.15.98:0/3989734086 shutdown_connections 2024-05-27T00:15:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.861+0000 7faf32000640 1 --2- 172.21.15.98:0/3989734086 >> v2:172.21.15.98:3300/0 conn(0x7faf2c0c6800 0x7faf2c0c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.861+0000 7faf32000640 1 --2- 172.21.15.98:0/3989734086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf2c06e660 0x7faf2c150b10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.861+0000 7faf32000640 1 -- 172.21.15.98:0/3989734086 >> 172.21.15.98:0/3989734086 conn(0x7faf2c0cc200 msgr2=0x7faf2c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.861+0000 7faf32000640 1 -- 172.21.15.98:0/3989734086 shutdown_connections 2024-05-27T00:15:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.861+0000 7faf32000640 1 -- 172.21.15.98:0/3989734086 wait complete. 2024-05-27T00:15:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf32000640 1 Processor -- start 2024-05-27T00:15:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf32000640 1 -- start start 2024-05-27T00:15:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf32000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf2c06da00 0x7faf2c14f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf32000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf2c06e660 0x7faf2c1f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf32000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf2c0c6800 0x7faf2c14dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf32000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c157210 con 0x7faf2c06e660 2024-05-27T00:15:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf32000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c156f30 con 0x7faf2c06da00 2024-05-27T00:15:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf32000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faf2c06de20 con 0x7faf2c0c6800 2024-05-27T00:15:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf2aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf2c06e660 0x7faf2c1f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf2aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf2c06e660 0x7faf2c1f3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53774/0 (socket says 172.21.15.98:53774) 2024-05-27T00:15:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf2aa00640 1 -- 172.21.15.98:0/2652784737 learned_addr learned my addr 172.21.15.98:0/2652784737 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf2be00640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf2c0c6800 0x7faf2c14dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.865+0000 7faf2b400640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf2c06da00 0x7faf2c14f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf2aa00640 1 -- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf2c0c6800 msgr2=0x7faf2c14dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf2aa00640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf2c0c6800 0x7faf2c14dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf2aa00640 1 -- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf2c06da00 msgr2=0x7faf2c14f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf2aa00640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf2c06da00 0x7faf2c14f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf2aa00640 1 -- 172.21.15.98:0/2652784737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c1fd000 con 0x7faf2c06e660 2024-05-27T00:15:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7f0a40000640 1 Processor -- start 2024-05-27T00:15:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf2aa00640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf2c06e660 0x7faf2c1f3ca0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7faf2400c430 tx=0x7faf2400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf23400640 1 -- 172.21.15.98:0/2652784737 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24017070 con 0x7faf2c06e660 2024-05-27T00:15:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf23400640 1 -- 172.21.15.98:0/2652784737 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf240052f0 con 0x7faf2c06e660 2024-05-27T00:15:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf23400640 1 -- 172.21.15.98:0/2652784737 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24003840 con 0x7faf2c06e660 2024-05-27T00:15:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf32000640 1 -- 172.21.15.98:0/2652784737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2c1fd2e0 con 0x7faf2c06e660 2024-05-27T00:15:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7faf32000640 1 -- 172.21.15.98:0/2652784737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2c1fd8a0 con 0x7faf2c06e660 2024-05-27T00:15:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7f0a40000640 1 -- start start 2024-05-27T00:15:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7f0a40000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0a380cba10 0x7f0a380d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7f0a40000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a380d0970 con 0x7f0a380cb630 2024-05-27T00:15:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7f0a40000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a380b4110 con 0x7f0a380cc340 2024-05-27T00:15:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.869+0000 7f0a40000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f0a380b43f0 con 0x7f0a380cba10 2024-05-27T00:15:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.873+0000 7f0a3d600640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0a380cb630 0x7f0a38162d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43402/0 (socket says 172.21.15.98:43402) 2024-05-27T00:15:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.873+0000 7f0a3cc00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0a380cba10 0x7f0a380d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.873+0000 7f0a3cc00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0a380cba10 0x7f0a380d0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57826/0 (socket says 172.21.15.98:57826) 2024-05-27T00:15:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.873+0000 7f0a3cc00640 1 -- 172.21.15.98:0/3493544951 learned_addr learned my addr 172.21.15.98:0/3493544951 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.873+0000 7faf32000640 1 -- 172.21.15.98:0/2652784737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf2c075cb0 con 0x7faf2c06e660 2024-05-27T00:15:32.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.873+0000 7f0a3cc00640 1 -- 172.21.15.98:0/3493544951 >> v1:172.21.15.204:6789/0 conn(0x7f0a380cc340 legacy=0x7f0a380cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.873+0000 7f0a3cc00640 1 -- 172.21.15.98:0/3493544951 >> v1:172.21.15.98:6789/0 conn(0x7f0a380cb630 legacy=0x7f0a38162d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.873+0000 7faf23400640 1 -- 172.21.15.98:0/2652784737 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf2401a070 con 0x7faf2c06e660 2024-05-27T00:15:32.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.873+0000 7f0a3cc00640 1 -- 172.21.15.98:0/3493544951 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a380b46d0 con 0x7f0a380cba10 2024-05-27T00:15:32.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.877+0000 7faf23400640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf080678c0 0x7faf08069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.877+0000 7faf2b400640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf080678c0 0x7faf08069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.877+0000 7faf23400640 1 -- 172.21.15.98:0/2652784737 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7faf24053590 con 0x7faf2c06e660 2024-05-27T00:15:32.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.881+0000 7f0a3cc00640 1 --2- 172.21.15.98:0/3493544951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 0x7f0a380d0070 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a24006fd0 tx=0x7f0a2401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b44f52f0b6846fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.881+0000 7f0a37e00640 1 -- 172.21.15.98:0/3493544951 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a240ab430 con 0x7f0a380cba10 2024-05-27T00:15:32.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.881+0000 7f0a37e00640 1 -- 172.21.15.98:0/3493544951 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a240b0070 con 0x7f0a380cba10 2024-05-27T00:15:32.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.881+0000 7f0a37e00640 1 -- 172.21.15.98:0/3493544951 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a240ab7f0 con 0x7f0a380cba10 2024-05-27T00:15:32.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.885+0000 7faf2b400640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf080678c0 0x7faf08069d80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faf140b2000 tx=0x7faf14009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.885+0000 7f0a40000640 1 -- 172.21.15.98:0/3493544951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 msgr2=0x7f0a380d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.885+0000 7f0a40000640 1 --2- 172.21.15.98:0/3493544951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 0x7f0a380d0070 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a24006fd0 tx=0x7f0a2401c540 comp rx=0 tx=0).stop 2024-05-27T00:15:32.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.885+0000 7f0a40000640 1 -- 172.21.15.98:0/3493544951 shutdown_connections 2024-05-27T00:15:32.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.885+0000 7f0a40000640 1 --2- 172.21.15.98:0/3493544951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 0x7f0a380d0070 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.885+0000 7f0a40000640 1 -- 172.21.15.98:0/3493544951 >> 172.21.15.98:0/3493544951 conn(0x7f0a38067530 msgr2=0x7f0a3806c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:32.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.885+0000 7f0a40000640 1 -- 172.21.15.98:0/3493544951 shutdown_connections 2024-05-27T00:15:32.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.885+0000 7f0a40000640 1 -- 172.21.15.98:0/3493544951 wait complete. 2024-05-27T00:15:32.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.885+0000 7faf23400640 1 -- 172.21.15.98:0/2652784737 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7faf240554f0 con 0x7faf2c06e660 2024-05-27T00:15:32.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.885+0000 7f0a40000640 1 Processor -- start 2024-05-27T00:15:32.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a40000640 1 -- start start 2024-05-27T00:15:32.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a40000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a380cb630 0x7f0a38161480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a40000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 0x7f0a38153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a40000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a380cc340 0x7f0a38153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a3cc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 0x7f0a38153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a3cc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 0x7f0a38153200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:57842/0 (socket says 172.21.15.98:57842) 2024-05-27T00:15:32.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a3d600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a380cb630 0x7f0a38161480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a3d600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a380cb630 0x7f0a38161480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53784/0 (socket says 172.21.15.98:53784) 2024-05-27T00:15:32.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a3cc00640 1 -- 172.21.15.98:0/458490795 learned_addr learned my addr 172.21.15.98:0/458490795 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:32.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a40000640 1 -- 172.21.15.98:0/458490795 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a380cd990 con 0x7f0a380cb630 2024-05-27T00:15:32.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a40000640 1 -- 172.21.15.98:0/458490795 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0a380cd810 con 0x7f0a380cba10 2024-05-27T00:15:32.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a40000640 1 -- 172.21.15.98:0/458490795 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0a380b51b0 con 0x7f0a380cc340 2024-05-27T00:15:32.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.889+0000 7f0a3e000640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a380cc340 0x7f0a38153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a3e000640 1 -- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 msgr2=0x7f0a38153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a3e000640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 0x7f0a38153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a3e000640 1 -- 172.21.15.98:0/458490795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a380cb630 msgr2=0x7f0a38161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a3e000640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a380cb630 0x7f0a38161480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a3e000640 1 -- 172.21.15.98:0/458490795 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a38154e00 con 0x7f0a380cc340 2024-05-27T00:15:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a3cc00640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 0x7f0a38153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a3d600640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a380cb630 0x7f0a38161480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a3e000640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a380cc340 0x7f0a38153740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000c760 tx=0x7f0a3000cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a35600640 1 -- 172.21.15.98:0/458490795 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30016020 con 0x7f0a380cc340 2024-05-27T00:15:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a35600640 1 -- 172.21.15.98:0/458490795 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a30006d60 con 0x7f0a380cc340 2024-05-27T00:15:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a35600640 1 -- 172.21.15.98:0/458490795 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30003bc0 con 0x7f0a380cc340 2024-05-27T00:15:32.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a40000640 1 -- 172.21.15.98:0/458490795 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3815d4b0 con 0x7f0a380cc340 2024-05-27T00:15:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.893+0000 7f0a40000640 1 -- 172.21.15.98:0/458490795 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a381622b0 con 0x7f0a380cc340 2024-05-27T00:15:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.897+0000 7f0a2aa00640 1 -- 172.21.15.98:0/458490795 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a38075cb0 con 0x7f0a380cc340 2024-05-27T00:15:32.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.901+0000 7f0a35600640 1 -- 172.21.15.98:0/458490795 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0a300050d0 con 0x7f0a380cc340 2024-05-27T00:15:32.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.901+0000 7f0a35600640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0a18067990 0x7f0a18069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:32.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.901+0000 7f0a35600640 1 -- 172.21.15.98:0/458490795 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f0a30085780 con 0x7f0a380cc340 2024-05-27T00:15:32.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.909+0000 7f0a3d600640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0a18067990 0x7f0a18069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:32.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.909+0000 7f0a3d600640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0a18067990 0x7f0a18069e50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c000f70 tx=0x7f0a2c007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:32.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:32.913+0000 7f0a35600640 1 -- 172.21.15.98:0/458490795 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f0a30020020 con 0x7f0a380cc340 2024-05-27T00:15:33.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.113+0000 7f2520800640 1 -- 172.21.15.98:0/1995365115 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f25180b17c0 con 0x7f2518070b60 2024-05-27T00:15:33.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.113+0000 7f2512000640 1 -- 172.21.15.98:0/1995365115 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f250c055e20 con 0x7f2518070b60 2024-05-27T00:15:33.118 INFO:teuthology.orchestra.run.smithi098.stdout:60129542150 2024-05-27T00:15:33.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.117+0000 7f2507400640 1 -- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24fc0679c0 msgr2=0x7f24fc069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.117+0000 7f2507400640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24fc0679c0 0x7f24fc069e80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2508007fd0 tx=0x7f2508007900 comp rx=0 tx=0).stop 2024-05-27T00:15:33.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.117+0000 7f2507400640 1 -- 172.21.15.98:0/1995365115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2518070b60 msgr2=0x7f25180c52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.117+0000 7f2507400640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2518070b60 0x7f25180c52f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f250c00c480 tx=0x7f250c00c940 comp rx=0 tx=0).stop 2024-05-27T00:15:33.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.117+0000 7f2507400640 1 -- 172.21.15.98:0/1995365115 shutdown_connections 2024-05-27T00:15:33.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.117+0000 7f2507400640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2518150b30 0x7f251814cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.117+0000 7f2507400640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24fc0679c0 0x7f24fc069e80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.117+0000 7f2507400640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25180c1a60 0x7f2518166180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.117+0000 7f2507400640 1 --2- 172.21.15.98:0/1995365115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2518070b60 0x7f25180c52f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.117+0000 7f2507400640 1 -- 172.21.15.98:0/1995365115 >> 172.21.15.98:0/1995365115 conn(0x7f251806e750 msgr2=0x7f2518072f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:33.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.121+0000 7f2507400640 1 -- 172.21.15.98:0/1995365115 shutdown_connections 2024-05-27T00:15:33.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.121+0000 7f2507400640 1 -- 172.21.15.98:0/1995365115 wait complete. 2024-05-27T00:15:33.152 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.1 2024-05-27T00:15:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.153+0000 7fa496a00640 1 -- 172.21.15.98:0/3206709806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa4a40c2f10 con 0x7fa4a4154ae0 2024-05-27T00:15:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.157+0000 7fa4a1600640 1 -- 172.21.15.98:0/3206709806 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa49805aa00 con 0x7fa4a4154ae0 2024-05-27T00:15:33.159 INFO:teuthology.orchestra.run.smithi098.stdout:60129542149 2024-05-27T00:15:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.157+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f2178075cb0 con 0x7f2178162db0 2024-05-27T00:15:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.157+0000 7f2175600640 1 -- 172.21.15.98:0/2816848726 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f21700570e0 con 0x7f2178162db0 2024-05-27T00:15:33.162 INFO:teuthology.orchestra.run.smithi098.stdout:60129542149 2024-05-27T00:15:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 -- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4840679c0 msgr2=0x7fa484069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4840679c0 0x7fa484069e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa490000f70 tx=0x7fa490007860 comp rx=0 tx=0).stop 2024-05-27T00:15:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 -- 172.21.15.98:0/3206709806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4a4154ae0 msgr2=0x7fa4a4151cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4a4154ae0 0x7fa4a4151cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa49800e980 tx=0x7fa49800ee40 comp rx=0 tx=0).stop 2024-05-27T00:15:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 -- 172.21.15.98:0/3206709806 shutdown_connections 2024-05-27T00:15:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4a415b310 0x7fa4a4152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4a4154ae0 0x7fa4a4151cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4840679c0 0x7fa484069e80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 --2- 172.21.15.98:0/3206709806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4a40c19c0 0x7fa4a4159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 -- 172.21.15.98:0/3206709806 >> 172.21.15.98:0/3206709806 conn(0x7fa4a40cc200 msgr2=0x7fa4a40ca300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f21580678f0 msgr2=0x7f2158069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 -- 172.21.15.98:0/3206709806 shutdown_connections 2024-05-27T00:15:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7fa496a00640 1 -- 172.21.15.98:0/3206709806 wait complete. 2024-05-27T00:15:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f21580678f0 0x7f2158069db0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f216c01d340 tx=0x7f216c0aa070 comp rx=0 tx=0).stop 2024-05-27T00:15:33.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f0b72200640 1 -- 172.21.15.98:0/3165930608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0b6c15d3f0 con 0x7f0b6c154a00 2024-05-27T00:15:33.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2178162db0 msgr2=0x7f21781f5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2178162db0 0x7f21781f5640 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f21780c8580 tx=0x7f217000cb90 comp rx=0 tx=0).stop 2024-05-27T00:15:33.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 shutdown_connections 2024-05-27T00:15:33.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2178162db0 0x7f21781f5640 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f21580678f0 0x7f2158069db0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21781590f0 0x7f21781f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 --2- 172.21.15.98:0/2816848726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f21780c1b50 0x7f21781e81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 >> 172.21.15.98:0/2816848726 conn(0x7f21780cc200 msgr2=0x7f21780cb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:33.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 shutdown_connections 2024-05-27T00:15:33.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f0b63400640 1 -- 172.21.15.98:0/3165930608 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0b58021020 con 0x7f0b6c154a00 2024-05-27T00:15:33.169 INFO:teuthology.orchestra.run.smithi098.stdout:60129542150 2024-05-27T00:15:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.161+0000 7f2180000640 1 -- 172.21.15.98:0/2816848726 wait complete. 2024-05-27T00:15:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.165+0000 7f0ff0c00640 1 -- 172.21.15.98:0/2048138786 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0ffc0c78b0 con 0x7f0ffc155150 2024-05-27T00:15:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.165+0000 7f0b72200640 1 -- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b4c067910 msgr2=0x7f0b4c069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.165+0000 7f0b72200640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b4c067910 0x7f0b4c069dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c009b60 tx=0x7f0b5c009190 comp rx=0 tx=0).stop 2024-05-27T00:15:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.165+0000 7f0b72200640 1 -- 172.21.15.98:0/3165930608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c154a00 msgr2=0x7f0b6c155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.165+0000 7f0b72200640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c154a00 0x7f0b6c155460 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800c3c0 tx=0x7f0b5800c880 comp rx=0 tx=0).stop 2024-05-27T00:15:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.165+0000 7f0ff3400640 1 -- 172.21.15.98:0/2048138786 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0ff405aa90 con 0x7f0ffc155150 2024-05-27T00:15:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0b72200640 1 -- 172.21.15.98:0/3165930608 shutdown_connections 2024-05-27T00:15:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0b72200640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b4c067910 0x7f0b4c069dd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0b72200640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c154de0 0x7f0b6c0c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0b72200640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b6c154a00 0x7f0b6c155460 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0b72200640 1 --2- 172.21.15.98:0/3165930608 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0d0070 0x7f0b6c15ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0b72200640 1 -- 172.21.15.98:0/3165930608 >> 172.21.15.98:0/3165930608 conn(0x7f0b6c067530 msgr2=0x7f0b6c15b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:33.172 INFO:teuthology.orchestra.run.smithi098.stdout:60129542149 2024-05-27T00:15:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0b72200640 1 -- 172.21.15.98:0/3165930608 shutdown_connections 2024-05-27T00:15:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0b72200640 1 -- 172.21.15.98:0/3165930608 wait complete. 2024-05-27T00:15:33.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 -- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0fdc067940 msgr2=0x7f0fdc069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0fdc067940 0x7f0fdc069e00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8007fd0 tx=0x7f0fe8007f20 comp rx=0 tx=0).stop 2024-05-27T00:15:33.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 -- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ffc155150 msgr2=0x7f0ffc1fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ffc155150 0x7f0ffc1fb2f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400ce00 tx=0x7f0ff400a710 comp rx=0 tx=0).stop 2024-05-27T00:15:33.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 -- 172.21.15.98:0/2048138786 shutdown_connections 2024-05-27T00:15:33.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ffc155150 0x7f0ffc1fb2f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0fdc067940 0x7f0fdc069e00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ffc158ed0 0x7f0ffc0ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 --2- 172.21.15.98:0/2048138786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ffc0c1610 0x7f0ffc15a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 -- 172.21.15.98:0/2048138786 >> 172.21.15.98:0/2048138786 conn(0x7f0ffc0cc200 msgr2=0x7f0ffc072ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:33.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 -- 172.21.15.98:0/2048138786 shutdown_connections 2024-05-27T00:15:33.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.169+0000 7f0ff0c00640 1 -- 172.21.15.98:0/2048138786 wait complete. 2024-05-27T00:15:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.189+0000 7fa81d200640 1 -- 172.21.15.98:0/2558713170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa7f0005860 con 0x7fa81806c770 2024-05-27T00:15:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.189+0000 7fa80e000640 1 -- 172.21.15.98:0/2558713170 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa808057fa0 con 0x7fa81806c770 2024-05-27T00:15:33.192 INFO:teuthology.orchestra.run.smithi098.stdout:60129542151 2024-05-27T00:15:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 -- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7f8067910 msgr2=0x7fa7f8069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7f8067910 0x7fa7f8069dd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa81001c690 tx=0x7fa8100b5040 comp rx=0 tx=0).stop 2024-05-27T00:15:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 -- 172.21.15.98:0/2558713170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa81806c770 msgr2=0x7fa8181ea0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa81806c770 0x7fa8181ea0f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa818072600 tx=0x7fa808007710 comp rx=0 tx=0).stop 2024-05-27T00:15:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 -- 172.21.15.98:0/2558713170 shutdown_connections 2024-05-27T00:15:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa81806c770 0x7fa8181ea0f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7f8067910 0x7fa7f8069dd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa81806c330 0x7fa8181e9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 --2- 172.21.15.98:0/2558713170 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa7fc0039c0 0x7fa8180cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 -- 172.21.15.98:0/2558713170 >> 172.21.15.98:0/2558713170 conn(0x7fa81806d680 msgr2=0x7fa81806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 -- 172.21.15.98:0/2558713170 shutdown_connections 2024-05-27T00:15:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.193+0000 7fa81d200640 1 -- 172.21.15.98:0/2558713170 wait complete. 2024-05-27T00:15:33.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.0 2024-05-27T00:15:33.205 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.2 2024-05-27T00:15:33.207 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.4 2024-05-27T00:15:33.208 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.3 2024-05-27T00:15:33.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 -- 172.21.15.98:0/2652784737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7faf2c1566b0 con 0x7faf2c06e660 2024-05-27T00:15:33.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf23400640 1 -- 172.21.15.98:0/2652784737 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faf240597f0 con 0x7faf2c06e660 2024-05-27T00:15:33.219 INFO:teuthology.orchestra.run.smithi098.stdout:60129542150 2024-05-27T00:15:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 -- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf080678c0 msgr2=0x7faf08069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf080678c0 0x7faf08069d80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faf140b2000 tx=0x7faf14009040 comp rx=0 tx=0).stop 2024-05-27T00:15:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 -- 172.21.15.98:0/2652784737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf2c06e660 msgr2=0x7faf2c1f3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf2c06e660 0x7faf2c1f3ca0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7faf2400c430 tx=0x7faf2400c8f0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 -- 172.21.15.98:0/2652784737 shutdown_connections 2024-05-27T00:15:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf2c0c6800 0x7faf2c14dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf2c06e660 0x7faf2c1f3ca0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf080678c0 0x7faf08069d80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 --2- 172.21.15.98:0/2652784737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf2c06da00 0x7faf2c14f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 -- 172.21.15.98:0/2652784737 >> 172.21.15.98:0/2652784737 conn(0x7faf2c0cc200 msgr2=0x7faf2c0c9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:33.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 -- 172.21.15.98:0/2652784737 shutdown_connections 2024-05-27T00:15:33.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.217+0000 7faf32000640 1 -- 172.21.15.98:0/2652784737 wait complete. 2024-05-27T00:15:33.222 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542151 for osd.5 2024-05-27T00:15:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.221+0000 7f0a2aa00640 1 -- 172.21.15.98:0/458490795 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f0a381553d0 con 0x7f0a380cc340 2024-05-27T00:15:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.221+0000 7f0a35600640 1 -- 172.21.15.98:0/458490795 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0a30056330 con 0x7f0a380cc340 2024-05-27T00:15:33.225 INFO:teuthology.orchestra.run.smithi098.stdout:60129542149 2024-05-27T00:15:33.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 -- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0a18067990 msgr2=0x7f0a18069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0a18067990 0x7f0a18069e50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c000f70 tx=0x7f0a2c007d50 comp rx=0 tx=0).stop 2024-05-27T00:15:33.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 -- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a380cc340 msgr2=0x7f0a38153740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:33.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a380cc340 0x7f0a38153740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0a3000c760 tx=0x7f0a3000cc20 comp rx=0 tx=0).stop 2024-05-27T00:15:33.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 -- 172.21.15.98:0/458490795 shutdown_connections 2024-05-27T00:15:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a380cc340 0x7f0a38153740 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0a18067990 0x7f0a18069e50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a380cba10 0x7f0a38153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 --2- 172.21.15.98:0/458490795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a380cb630 0x7f0a38161480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 -- 172.21.15.98:0/458490795 >> 172.21.15.98:0/458490795 conn(0x7f0a38067530 msgr2=0x7f0a38157a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 -- 172.21.15.98:0/458490795 shutdown_connections 2024-05-27T00:15:33.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:33.225+0000 7f0a2aa00640 1 -- 172.21.15.98:0/458490795 wait complete. 2024-05-27T00:15:33.245 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.6 2024-05-27T00:15:33.248 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.7 2024-05-27T00:15:34.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-27T00:15:34.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-27T00:15:34.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-27T00:15:34.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-27T00:15:34.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-27T00:15:34.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-27T00:15:34.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-27T00:15:34.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-27T00:15:34.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.325+0000 7f44f3200640 1 Processor -- start 2024-05-27T00:15:34.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.325+0000 7f44f3200640 1 -- start start 2024-05-27T00:15:34.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.325+0000 7f44f1e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f44e80d5000 0x7f44e804c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47052/0 (socket says 172.21.15.98:47052) 2024-05-27T00:15:34.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.325+0000 7f44f1e00640 1 -- 172.21.15.98:0/353524178 learned_addr learned my addr 172.21.15.98:0/353524178 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:34.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.325+0000 7f44f3200640 1 --2- 172.21.15.98:0/353524178 >> v2:172.21.15.204:3300/0 conn(0x7f44e80347a0 0x7f44e80dad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f3200640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44e80db270 0x7f44e80db680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f3200640 1 -- 172.21.15.98:0/353524178 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44e804b990 con 0x7f44e80d5000 2024-05-27T00:15:34.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f1400640 1 --2- 172.21.15.98:0/353524178 >> v2:172.21.15.204:3300/0 conn(0x7f44e80347a0 0x7f44e80dad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f2800640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44e80db270 0x7f44e80db680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f3200640 1 -- 172.21.15.98:0/353524178 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f44e804bc70 con 0x7f44e80347a0 2024-05-27T00:15:34.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f3200640 1 -- 172.21.15.98:0/353524178 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f44e8006a50 con 0x7f44e80db270 2024-05-27T00:15:34.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352522315 0 0) 0x7f44e804b990 con 0x7f44e80d5000 2024-05-27T00:15:34.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44d4003530 con 0x7f44e80d5000 2024-05-27T00:15:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3381575446 0 0) 0x7f44d4003530 con 0x7f44e80d5000 2024-05-27T00:15:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44e80db270 msgr2=0x7f44e80db680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44e80db270 0x7f44e80db680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 >> v2:172.21.15.204:3300/0 conn(0x7f44e80347a0 msgr2=0x7f44e80dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 --2- 172.21.15.98:0/353524178 >> v2:172.21.15.204:3300/0 conn(0x7f44e80347a0 0x7f44e80dad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e8006d30 con 0x7f44e80d5000 2024-05-27T00:15:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f44e0001680 con 0x7f44e80d5000 2024-05-27T00:15:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 >> v1:172.21.15.98:6789/0 conn(0x7f44e80d5000 legacy=0x7f44e804c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40039c0 0x7f44d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d4004810 0x7f44d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d40051d0 0x7f44d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44e804bc70 con 0x7f44d40051d0 2024-05-27T00:15:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f44e8006a50 con 0x7f44d4004810 2024-05-27T00:15:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f44d400bf10 con 0x7f44d40039c0 2024-05-27T00:15:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f1400640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d4004810 0x7f44d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f1e00640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40039c0 0x7f44d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f2800640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d40051d0 0x7f44d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f2800640 1 -- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40039c0 msgr2=0x7f44d4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f2800640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40039c0 0x7f44d4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f2800640 1 -- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d4004810 msgr2=0x7f44d4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f2800640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d4004810 0x7f44d4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f2800640 1 -- 172.21.15.98:0/353524178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d4003530 con 0x7f44d40051d0 2024-05-27T00:15:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f1e00640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40039c0 0x7f44d4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.329+0000 7f44f1400640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d4004810 0x7f44d4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.333+0000 7f44f2800640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d40051d0 0x7f44d400b9d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000b70 tx=0x7f44ec0987d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39950cb2ea96031b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.337+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec050700 con 0x7f44d40051d0 2024-05-27T00:15:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.337+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44ec127540 con 0x7f44d40051d0 2024-05-27T00:15:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.337+0000 7f44f3200640 1 -- 172.21.15.98:0/353524178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d40051d0 msgr2=0x7f44d400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.337+0000 7f44f0a00640 1 -- 172.21.15.98:0/353524178 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec050700 con 0x7f44d40051d0 2024-05-27T00:15:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.337+0000 7f44f3200640 1 --2- 172.21.15.98:0/353524178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44d40051d0 0x7f44d400b9d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000b70 tx=0x7f44ec0987d0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.337+0000 7f44f2800640 1 -- 172.21.15.98:0/353524178 reap_dead start 2024-05-27T00:15:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.337+0000 7f44f3200640 1 -- 172.21.15.98:0/353524178 shutdown_connections 2024-05-27T00:15:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.337+0000 7f44f3200640 1 -- 172.21.15.98:0/353524178 >> 172.21.15.98:0/353524178 conn(0x7f44e8050190 msgr2=0x7f44e80505a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.341+0000 7f44f3200640 1 -- 172.21.15.98:0/353524178 shutdown_connections 2024-05-27T00:15:34.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.341+0000 7f44f3200640 1 -- 172.21.15.98:0/353524178 wait complete. 2024-05-27T00:15:34.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.341+0000 7f44f3200640 1 Processor -- start 2024-05-27T00:15:34.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f3200640 1 -- start start 2024-05-27T00:15:34.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f3200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40051d0 0x7f44e80d9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44e80d5680 0x7f44e81703b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f3200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e80c7dc0 0x7f44e80ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f3200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44e80d53e0 con 0x7f44e80c7dc0 2024-05-27T00:15:34.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f44e8045130 con 0x7f44e80d5680 2024-05-27T00:15:34.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f3200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f44e8034c10 con 0x7f44d40051d0 2024-05-27T00:15:34.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f2800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e80c7dc0 0x7f44e80ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f2800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e80c7dc0 0x7f44e80ccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51044/0 (socket says 172.21.15.98:51044) 2024-05-27T00:15:34.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f2800640 1 -- 172.21.15.98:0/3474471890 learned_addr learned my addr 172.21.15.98:0/3474471890 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.345+0000 7f44f1e00640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40051d0 0x7f44e80d9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44f1400640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44e80d5680 0x7f44e81703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44f1e00640 1 -- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44e80d5680 msgr2=0x7f44e81703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44f1e00640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44e80d5680 0x7f44e81703b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44f1e00640 1 -- 172.21.15.98:0/3474471890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e80c7dc0 msgr2=0x7f44e80ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44f1e00640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e80c7dc0 0x7f44e80ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44f1e00640 1 -- 172.21.15.98:0/3474471890 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e80d8a20 con 0x7f44d40051d0 2024-05-27T00:15:34.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44f1e00640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40051d0 0x7f44e80d9920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44e0000c10 tx=0x7f44e0007750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44e6000640 1 -- 172.21.15.98:0/3474471890 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e000fac0 con 0x7f44d40051d0 2024-05-27T00:15:34.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44e6000640 1 -- 172.21.15.98:0/3474471890 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e0018070 con 0x7f44d40051d0 2024-05-27T00:15:34.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44e6000640 1 -- 172.21.15.98:0/3474471890 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e000fac0 con 0x7f44d40051d0 2024-05-27T00:15:34.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44f3200640 1 -- 172.21.15.98:0/3474471890 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e80da9a0 con 0x7f44d40051d0 2024-05-27T00:15:34.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.349+0000 7f44f3200640 1 -- 172.21.15.98:0/3474471890 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e80daf60 con 0x7f44d40051d0 2024-05-27T00:15:34.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.357+0000 7f44e6000640 1 -- 172.21.15.98:0/3474471890 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f44e000fc60 con 0x7f44d40051d0 2024-05-27T00:15:34.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.357+0000 7f44f3200640 1 -- 172.21.15.98:0/3474471890 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44e80053a0 con 0x7f44d40051d0 2024-05-27T00:15:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.357+0000 7f44e6000640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44d0067cc0 0x7f44d006a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.357+0000 7f44e6000640 1 -- 172.21.15.98:0/3474471890 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f44e0010040 con 0x7f44d40051d0 2024-05-27T00:15:34.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.361+0000 7f44f1400640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44d0067cc0 0x7f44d006a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.361+0000 7f44f1400640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44d0067cc0 0x7f44d006a180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f44e804f5a0 tx=0x7f44dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.373+0000 7f44e6000640 1 -- 172.21.15.98:0/3474471890 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f44e80d8a20 con 0x7f44d40051d0 2024-05-27T00:15:34.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.413+0000 7fd6cf400640 1 Processor -- start 2024-05-27T00:15:34.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.413+0000 7fd6cf400640 1 -- start start 2024-05-27T00:15:34.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.413+0000 7fd6cf400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8070b60 0x7fd6c8159e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.413+0000 7fd6cf400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd6c80c1a60 0x7fd6c80c28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.413+0000 7fd6cf400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c80c7630 con 0x7fd6c8150b30 2024-05-27T00:15:34.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.413+0000 7fd6cca00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd6c8150b30 0x7fd6c80c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47054/0 (socket says 172.21.15.98:47054) 2024-05-27T00:15:34.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.413+0000 7fd6cca00640 1 -- 172.21.15.98:0/2866780562 learned_addr learned my addr 172.21.15.98:0/2866780562 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:34.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.413+0000 7fd6cf400640 1 -- 172.21.15.98:0/2866780562 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd6c80c7100 con 0x7fd6c80c1a60 2024-05-27T00:15:34.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.413+0000 7fd6cf400640 1 -- 172.21.15.98:0/2866780562 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c80c81d0 con 0x7fd6c8070b60 2024-05-27T00:15:34.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.413+0000 7fd6cd400640 1 --2- 172.21.15.98:0/2866780562 >> v2:172.21.15.98:3300/0 conn(0x7fd6c80c1a60 0x7fd6c80c28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.417+0000 7fd6c7e00640 1 --2- 172.21.15.98:0/2866780562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8070b60 0x7fd6c8159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.417+0000 7fd6c7400640 1 -- 172.21.15.98:0/2866780562 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 438101127 0 0) 0x7fd6c80c7630 con 0x7fd6c8150b30 2024-05-27T00:15:34.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.417+0000 7fd6c7400640 1 -- 172.21.15.98:0/2866780562 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6b0003530 con 0x7fd6c8150b30 2024-05-27T00:15:34.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.417+0000 7fd6cd400640 1 -- 172.21.15.98:0/2866780562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8070b60 msgr2=0x7fd6c8159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.417+0000 7fd6cd400640 1 --2- 172.21.15.98:0/2866780562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8070b60 0x7fd6c8159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.417+0000 7fd6cd400640 1 -- 172.21.15.98:0/2866780562 >> v1:172.21.15.98:6789/0 conn(0x7fd6c8150b30 legacy=0x7fd6c80c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.417+0000 7fd6cd400640 1 -- 172.21.15.98:0/2866780562 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c80c7350 con 0x7fd6c80c1a60 2024-05-27T00:15:34.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.421+0000 7fd6cd400640 1 --2- 172.21.15.98:0/2866780562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80c1a60 0x7fd6c80c28a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009d90 tx=0x7fd6c001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746ca52bbfef0263 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.421+0000 7fd6c7400640 1 -- 172.21.15.98:0/2866780562 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00b2070 con 0x7fd6c80c1a60 2024-05-27T00:15:34.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.421+0000 7fd6c7400640 1 -- 172.21.15.98:0/2866780562 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c00a5a30 con 0x7fd6c80c1a60 2024-05-27T00:15:34.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.425+0000 7fd6c7400640 1 -- 172.21.15.98:0/2866780562 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00a5de0 con 0x7fd6c80c1a60 2024-05-27T00:15:34.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.425+0000 7fd6cf400640 1 -- 172.21.15.98:0/2866780562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80c1a60 msgr2=0x7fd6c80c28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.425+0000 7fd6cf400640 1 --2- 172.21.15.98:0/2866780562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80c1a60 0x7fd6c80c28a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009d90 tx=0x7fd6c001c970 comp rx=0 tx=0).stop 2024-05-27T00:15:34.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.425+0000 7fd6cf400640 1 -- 172.21.15.98:0/2866780562 shutdown_connections 2024-05-27T00:15:34.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.425+0000 7fd6cf400640 1 --2- 172.21.15.98:0/2866780562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80c1a60 0x7fd6c80c28a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.425+0000 7fd6cf400640 1 --2- 172.21.15.98:0/2866780562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8070b60 0x7fd6c8159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.425+0000 7fd6cf400640 1 -- 172.21.15.98:0/2866780562 >> 172.21.15.98:0/2866780562 conn(0x7fd6c806e750 msgr2=0x7fd6c806eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.425+0000 7fd6cf400640 1 -- 172.21.15.98:0/2866780562 shutdown_connections 2024-05-27T00:15:34.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.425+0000 7fd6cf400640 1 -- 172.21.15.98:0/2866780562 wait complete. 2024-05-27T00:15:34.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cf400640 1 Processor -- start 2024-05-27T00:15:34.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cf400640 1 -- start start 2024-05-27T00:15:34.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6c8070b60 0x7fd6c81455f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cf400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80c1a60 0x7fd6c8148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cf400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8150b30 0x7fd6c8149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cf400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c80c1740 con 0x7fd6c80c1a60 2024-05-27T00:15:34.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cf400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c80c15c0 con 0x7fd6c8070b60 2024-05-27T00:15:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cf400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c80c7870 con 0x7fd6c8150b30 2024-05-27T00:15:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6c8070b60 0x7fd6c81455f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cd400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8150b30 0x7fd6c8149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cd400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8150b30 0x7fd6c8149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:53258/0 (socket says 172.21.15.98:53258) 2024-05-27T00:15:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.429+0000 7fd6cd400640 1 -- 172.21.15.98:0/4081958155 learned_addr learned my addr 172.21.15.98:0/4081958155 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6cd400640 1 -- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6c8070b60 msgr2=0x7fd6c81455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6cd400640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6c8070b60 0x7fd6c81455f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6cd400640 1 -- 172.21.15.98:0/4081958155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80c1a60 msgr2=0x7fd6c8148d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T00:15:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6c7e00640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80c1a60 0x7fd6c8148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6cd400640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80c1a60 0x7fd6c8148d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6cd400640 1 -- 172.21.15.98:0/4081958155 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c8145fa0 con 0x7fd6c8150b30 2024-05-27T00:15:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6cca00640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6c8070b60 0x7fd6c81455f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6c7e00640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80c1a60 0x7fd6c8148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:34.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6cd400640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8150b30 0x7fd6c8149250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd6c00b0040 tx=0x7fd6c0019710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6c4c00640 1 -- 172.21.15.98:0/4081958155 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00b2050 con 0x7fd6c8150b30 2024-05-27T00:15:34.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6c4c00640 1 -- 172.21.15.98:0/4081958155 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c00b3040 con 0x7fd6c8150b30 2024-05-27T00:15:34.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6c4c00640 1 -- 172.21.15.98:0/4081958155 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0003dd0 con 0x7fd6c8150b30 2024-05-27T00:15:34.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.433+0000 7fd6cf400640 1 -- 172.21.15.98:0/4081958155 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c8200760 con 0x7fd6c8150b30 2024-05-27T00:15:34.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.437+0000 7fd6cf400640 1 -- 172.21.15.98:0/4081958155 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c81576e0 con 0x7fd6c8150b30 2024-05-27T00:15:34.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.437+0000 7fd6ba000640 1 -- 172.21.15.98:0/4081958155 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd69c0052a0 con 0x7fd6c8150b30 2024-05-27T00:15:34.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.441+0000 7fd6c4c00640 1 -- 172.21.15.98:0/4081958155 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd6c00c0020 con 0x7fd6c8150b30 2024-05-27T00:15:34.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.441+0000 7fd6c4c00640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd6a80678c0 0x7fd6a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.441+0000 7fd6c4c00640 1 -- 172.21.15.98:0/4081958155 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd6c0120d70 con 0x7fd6c8150b30 2024-05-27T00:15:34.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.445+0000 7fd6cca00640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd6a80678c0 0x7fd6a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.445+0000 7fd6cca00640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd6a80678c0 0x7fd6a8069d80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0011f0 tx=0x7fd6bc007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.453+0000 7fd6c4c00640 1 -- 172.21.15.98:0/4081958155 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd6c00c7020 con 0x7fd6c8150b30 2024-05-27T00:15:34.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.481+0000 7fd235600640 1 Processor -- start 2024-05-27T00:15:34.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.481+0000 7fd235600640 1 -- start start 2024-05-27T00:15:34.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.485+0000 7fd235600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd2280db470 0x7fd2280e6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.485+0000 7fd235600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22804b9c0 con 0x7fd2280d5030 2024-05-27T00:15:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.485+0000 7fd235600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22804bca0 con 0x7fd2280347d0 2024-05-27T00:15:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.485+0000 7fd235600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd228006a80 con 0x7fd2280db470 2024-05-27T00:15:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.485+0000 7fd22ea00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd2280d5030 0x7fd22804c310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47068/0 (socket says 172.21.15.98:47068) 2024-05-27T00:15:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.485+0000 7fd22ea00640 1 -- 172.21.15.98:0/2024783323 learned_addr learned my addr 172.21.15.98:0/2024783323 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.485+0000 7fd22f400640 1 --2- 172.21.15.98:0/2024783323 >> v2:172.21.15.98:3300/0 conn(0x7fd2280db470 0x7fd2280e6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.485+0000 7fd22f400640 1 -- 172.21.15.98:0/2024783323 >> v1:172.21.15.204:6789/0 conn(0x7fd2280347d0 legacy=0x7fd2280dad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.485+0000 7fd22f400640 1 -- 172.21.15.98:0/2024783323 >> v1:172.21.15.98:6789/0 conn(0x7fd2280d5030 legacy=0x7fd22804c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.485+0000 7fd22f400640 1 -- 172.21.15.98:0/2024783323 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd228006d60 con 0x7fd2280db470 2024-05-27T00:15:34.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd22f400640 1 --2- 172.21.15.98:0/2024783323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2280db470 0x7fd2280e6670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd220009fd0 tx=0x7fd22001e600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbb27e16983a26f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd22d600640 1 -- 172.21.15.98:0/2024783323 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2200b3070 con 0x7fd2280db470 2024-05-27T00:15:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd22d600640 1 -- 172.21.15.98:0/2024783323 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2200ae630 con 0x7fd2280db470 2024-05-27T00:15:34.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd22d600640 1 -- 172.21.15.98:0/2024783323 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2200b3070 con 0x7fd2280db470 2024-05-27T00:15:34.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd235600640 1 -- 172.21.15.98:0/2024783323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2280db470 msgr2=0x7fd2280e6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd235600640 1 --2- 172.21.15.98:0/2024783323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2280db470 0x7fd2280e6670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd220009fd0 tx=0x7fd22001e600 comp rx=0 tx=0).stop 2024-05-27T00:15:34.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fa579c00640 1 Processor -- start 2024-05-27T00:15:34.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fa579c00640 1 -- start start 2024-05-27T00:15:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7f32f4200640 1 Processor -- start 2024-05-27T00:15:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32f4200640 1 -- start start 2024-05-27T00:15:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd235600640 1 -- 172.21.15.98:0/2024783323 shutdown_connections 2024-05-27T00:15:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd235600640 1 --2- 172.21.15.98:0/2024783323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2280db470 0x7fd2280e6670 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd235600640 1 -- 172.21.15.98:0/2024783323 >> 172.21.15.98:0/2024783323 conn(0x7fd2280501c0 msgr2=0x7fd2280505d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd235600640 1 -- 172.21.15.98:0/2024783323 shutdown_connections 2024-05-27T00:15:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.493+0000 7fd235600640 1 -- 172.21.15.98:0/2024783323 wait complete. 2024-05-27T00:15:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa579c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa574158cc0 0x7fa57415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa579c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa574162150 con 0x7fa574071130 2024-05-27T00:15:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa579c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5740c7f70 con 0x7fa574071680 2024-05-27T00:15:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa579c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa5740c8250 con 0x7fa574158cc0 2024-05-27T00:15:34.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fdf4d600640 1 Processor -- start 2024-05-27T00:15:34.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa56b400640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa574071680 0x7fa5740c8650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:41918/0 (socket says 172.21.15.98:41918) 2024-05-27T00:15:34.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa56b400640 1 -- 172.21.15.98:0/4069796721 learned_addr learned my addr 172.21.15.98:0/4069796721 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:34.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1424502334 0 0) 0x7fa5740c7f70 con 0x7fa574071680 2024-05-27T00:15:34.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa558003530 con 0x7fa574071680 2024-05-27T00:15:34.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fd235600640 1 Processor -- start 2024-05-27T00:15:34.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32f4200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f32e404fed0 0x7f32e40502b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32f4200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32e404a880 0x7f32e40dec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32f4200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32e40dfa80 con 0x7f32e404aca0 2024-05-27T00:15:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32f4200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f32e4049870 con 0x7f32e404fed0 2024-05-27T00:15:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32f4200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f32e404a3f0 con 0x7f32e404a880 2024-05-27T00:15:34.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fd235600640 1 -- start start 2024-05-27T00:15:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fdf4d600640 1 -- start start 2024-05-27T00:15:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32f2200640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f32e404aca0 0x7f32e40df190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:41924/0 (socket says 172.21.15.98:41924) 2024-05-27T00:15:34.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32f2200640 1 -- 172.21.15.98:0/1678775487 learned_addr learned my addr 172.21.15.98:0/1678775487 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573e00640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa574158cc0 0x7fa57415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2744270574 0 0) 0x7fa558003530 con 0x7fa574071680 2024-05-27T00:15:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa574158cc0 msgr2=0x7fa57415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa574158cc0 0x7fa57415eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 >> v1:172.21.15.98:6789/0 conn(0x7fa574071130 legacy=0x7fa5741694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa574162410 con 0x7fa574071680 2024-05-27T00:15:34.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fa564001a00 con 0x7fa574071680 2024-05-27T00:15:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 >> v1:172.21.15.204:6789/0 conn(0x7fa574071680 legacy=0x7fa5740c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 0x7fa558003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa558004810 0x7fa558004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa558005220 0x7fa55800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5740c8250 con 0x7fa558004810 2024-05-27T00:15:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa55800c100 con 0x7fa558005220 2024-05-27T00:15:34.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa55800c300 con 0x7fa5580039c0 2024-05-27T00:15:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32f1800640 1 --2- 172.21.15.98:0/1678775487 >> v2:172.21.15.204:3300/0 conn(0x7f32e404fed0 0x7f32e40502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32ebe00640 1 -- 172.21.15.98:0/1678775487 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 5644654 0 0) 0x7f32e40dfa80 con 0x7f32e404aca0 2024-05-27T00:15:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7f32ebe00640 1 -- 172.21.15.98:0/1678775487 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32d4003530 con 0x7f32e404aca0 2024-05-27T00:15:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fd235600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2280347d0 0x7fd22817d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fd235600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2280d5030 0x7fd2280d9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd235600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2280db470 0x7fd228049c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fa573e00640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 0x7fa558003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fa56b400640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa558004810 0x7fa558004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd22e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2280d5030 0x7fd2280d9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7f32f1800640 1 -- 172.21.15.98:0/1678775487 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32e404a880 msgr2=0x7f32e40dec50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T00:15:34.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7f32f1800640 1 --2- 172.21.15.98:0/1678775487 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32e404a880 0x7f32e40dec50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7f32f1800640 1 -- 172.21.15.98:0/1678775487 >> v1:172.21.15.204:6789/0 conn(0x7f32e404aca0 legacy=0x7f32e40df190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7f32f1800640 1 -- 172.21.15.98:0/1678775487 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e4049ac0 con 0x7f32e404fed0 2024-05-27T00:15:34.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd22e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2280d5030 0x7fd2280d9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:39018/0 (socket says 172.21.15.98:39018) 2024-05-27T00:15:34.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd22e000640 1 -- 172.21.15.98:0/1897793723 learned_addr learned my addr 172.21.15.98:0/1897793723 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2280d5410 con 0x7fd2280347d0 2024-05-27T00:15:34.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd228045160 con 0x7fd2280d5030 2024-05-27T00:15:34.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fdf4d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf481590f0 0x7fdf480c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fdf4d600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fdf4806e890 0x7fdf4806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fdf4d600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf4815fb30 con 0x7fdf48162db0 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fdf4d600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fdf480c7700 con 0x7fdf4806e890 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.497+0000 7fdf4d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdf4815edb0 con 0x7fdf481590f0 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fdf46a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf481590f0 0x7fdf480c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fdf46a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf481590f0 0x7fdf480c7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:53284/0 (socket says 172.21.15.98:53284) 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fdf46a00640 1 -- 172.21.15.98:0/1705534729 learned_addr learned my addr 172.21.15.98:0/1705534729 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fdf46a00640 1 -- 172.21.15.98:0/1705534729 >> v2:172.21.15.204:3300/0 conn(0x7fdf4806e890 msgr2=0x7fdf4806ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fdf46a00640 1 --2- 172.21.15.98:0/1705534729 >> v2:172.21.15.204:3300/0 conn(0x7fdf4806e890 0x7fdf4806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fdf46a00640 1 -- 172.21.15.98:0/1705534729 >> v1:172.21.15.204:6789/0 conn(0x7fdf48162db0 legacy=0x7fdf480c1610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T00:15:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fdf46a00640 1 -- 172.21.15.98:0/1705534729 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf480c7420 con 0x7fdf481590f0 2024-05-27T00:15:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd228034c40 con 0x7fd2280db470 2024-05-27T00:15:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7f2b52800640 1 Processor -- start 2024-05-27T00:15:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7f2b52800640 1 -- start start 2024-05-27T00:15:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fa579200640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa558005220 0x7fa55800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fa573e00640 1 -- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa558005220 msgr2=0x7fa55800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fa573e00640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa558005220 0x7fa55800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fa573e00640 1 -- 172.21.15.98:0/4069796721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa558004810 msgr2=0x7fa558004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fa573e00640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa558004810 0x7fa558004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fa573e00640 1 -- 172.21.15.98:0/4069796721 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa574162410 con 0x7fa5580039c0 2024-05-27T00:15:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd22f400640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2280db470 0x7fd228049c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7f8da7200640 1 Processor -- start 2024-05-27T00:15:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7f8da7200640 1 -- start start 2024-05-27T00:15:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd22f400640 1 -- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2280d5030 msgr2=0x7fd2280d9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd22f400640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2280d5030 0x7fd2280d9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd22f400640 1 -- 172.21.15.98:0/1897793723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2280347d0 msgr2=0x7fd22817d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd22f400640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2280347d0 0x7fd22817d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd22f400640 1 -- 172.21.15.98:0/1897793723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd228047e10 con 0x7fd2280db470 2024-05-27T00:15:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd22f400640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2280db470 0x7fd228049c70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd2200ae000 tx=0x7fd22001b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd226a00640 1 -- 172.21.15.98:0/1897793723 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2200bc070 con 0x7fd2280db470 2024-05-27T00:15:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2280480f0 con 0x7fd2280db470 2024-05-27T00:15:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd226a00640 1 -- 172.21.15.98:0/1897793723 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2200aebf0 con 0x7fd2280db470 2024-05-27T00:15:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd226a00640 1 -- 172.21.15.98:0/1897793723 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2200b3070 con 0x7fd2280db470 2024-05-27T00:15:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.501+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd22817c8d0 con 0x7fd2280db470 2024-05-27T00:15:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2080052a0 con 0x7fd2280db470 2024-05-27T00:15:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b52800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f2b4c151920 0x7f2b4c073040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b52800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c0c7a60 0x7f2b4c15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f32f1800640 1 --2- 172.21.15.98:0/1678775487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404fed0 0x7f32e40502b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f32e00097c0 tx=0x7f32e001c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aab816240c8d8eb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b4b400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f2b4c151920 0x7f2b4c073040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b4b400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f2b4c151920 0x7f2b4c073040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:39038/0 (socket says 172.21.15.98:39038) 2024-05-27T00:15:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b4aa00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2b4c0c86a0 0x7f2b4c0c3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47080/0 (socket says 172.21.15.98:47080) 2024-05-27T00:15:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b4be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c0c7a60 0x7f2b4c15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b4be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c0c7a60 0x7f2b4c15ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:53296/0 (socket says 172.21.15.98:53296) 2024-05-27T00:15:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b52800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b4c073580 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b52800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f2b4c151d00 con 0x7f2b4c151920 2024-05-27T00:15:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b52800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2b4c1577c0 con 0x7f2b4c0c7a60 2024-05-27T00:15:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7fdf46a00640 1 --2- 172.21.15.98:0/1705534729 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf481590f0 0x7fdf480c7b10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009e50 tx=0x7fdf40030210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d936d7d5da634f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f8da7200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8da00c7e10 0x7f8da006ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f32ebe00640 1 -- 172.21.15.98:0/1678775487 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e00b0070 con 0x7f32e404fed0 2024-05-27T00:15:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f32ebe00640 1 -- 172.21.15.98:0/1678775487 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e00ab600 con 0x7f32e404fed0 2024-05-27T00:15:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f32ebe00640 1 -- 172.21.15.98:0/1678775487 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e00b0070 con 0x7f32e404fed0 2024-05-27T00:15:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7fd226a00640 1 -- 172.21.15.98:0/1897793723 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd220003620 con 0x7fd2280db470 2024-05-27T00:15:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f8da7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 0x7f8da01578b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b4b400640 1 -- 172.21.15.98:0/748713958 learned_addr learned my addr 172.21.15.98:0/748713958 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f8da7200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8da00c84a0 con 0x7f8da01519d0 2024-05-27T00:15:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f8da7200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f8da00c8260 con 0x7f8da00c7e10 2024-05-27T00:15:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f8da7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8da0070d80 con 0x7f8da006ef40 2024-05-27T00:15:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f8d9f400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8da00c7e10 0x7f8da006ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7fdf45600640 1 -- 172.21.15.98:0/1705534729 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400c5070 con 0x7fdf481590f0 2024-05-27T00:15:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b4be00640 1 -- 172.21.15.98:0/748713958 >> v2:172.21.15.204:3300/0 conn(0x7f2b4c151920 msgr2=0x7f2b4c073040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b4be00640 1 --2- 172.21.15.98:0/748713958 >> v2:172.21.15.204:3300/0 conn(0x7f2b4c151920 0x7f2b4c073040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b4be00640 1 -- 172.21.15.98:0/748713958 >> v1:172.21.15.98:6789/0 conn(0x7f2b4c0c86a0 legacy=0x7f2b4c0c3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f8da5200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 0x7f8da01578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f8da5200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 0x7f8da01578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:53306/0 (socket says 172.21.15.98:53306) 2024-05-27T00:15:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f8da5200640 1 -- 172.21.15.98:0/2667175253 learned_addr learned my addr 172.21.15.98:0/2667175253 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7fdf45600640 1 -- 172.21.15.98:0/1705534729 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf400c0400 con 0x7fdf481590f0 2024-05-27T00:15:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7fdf45600640 1 -- 172.21.15.98:0/1705534729 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400c07d0 con 0x7fdf481590f0 2024-05-27T00:15:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f32f4200640 1 -- 172.21.15.98:0/1678775487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404fed0 msgr2=0x7f32e40502b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f32f4200640 1 --2- 172.21.15.98:0/1678775487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404fed0 0x7f32e40502b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f32e00097c0 tx=0x7f32e001c4e0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.505+0000 7f2b4be00640 1 -- 172.21.15.98:0/748713958 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c157aa0 con 0x7f2b4c0c7a60 2024-05-27T00:15:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f8da5200640 1 -- 172.21.15.98:0/2667175253 >> v2:172.21.15.204:3300/0 conn(0x7f8da00c7e10 msgr2=0x7f8da006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f8da5200640 1 --2- 172.21.15.98:0/2667175253 >> v2:172.21.15.204:3300/0 conn(0x7f8da00c7e10 0x7f8da006ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f8da5200640 1 -- 172.21.15.98:0/2667175253 >> v1:172.21.15.98:6789/0 conn(0x7f8da01519d0 legacy=0x7f8da00c11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f8da5200640 1 -- 172.21.15.98:0/2667175253 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da0070fd0 con 0x7f8da006ef40 2024-05-27T00:15:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fa573e00640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 0x7fa558003dc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa5740cccb0 tx=0x7fa55c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d31f005c6428d949 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fd226a00640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2100678c0 0x7fd210069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fdf4d600640 1 -- 172.21.15.98:0/1705534729 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf481590f0 msgr2=0x7fdf480c7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fdf4d600640 1 --2- 172.21.15.98:0/1705534729 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf481590f0 0x7fdf480c7b10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009e50 tx=0x7fdf40030210 comp rx=0 tx=0).stop 2024-05-27T00:15:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fd226a00640 1 -- 172.21.15.98:0/1897793723 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd22011f580 con 0x7fd2280db470 2024-05-27T00:15:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fd22ea00640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2100678c0 0x7fd210069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fdf4d600640 1 -- 172.21.15.98:0/1705534729 shutdown_connections 2024-05-27T00:15:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fdf4d600640 1 --2- 172.21.15.98:0/1705534729 >> v2:172.21.15.204:3300/0 conn(0x7fdf4806e890 0x7fdf4806ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fdf4d600640 1 --2- 172.21.15.98:0/1705534729 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf481590f0 0x7fdf480c7b10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fdf4d600640 1 -- 172.21.15.98:0/1705534729 >> 172.21.15.98:0/1705534729 conn(0x7fdf480cc200 msgr2=0x7fdf480cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f32f4200640 1 -- 172.21.15.98:0/1678775487 shutdown_connections 2024-05-27T00:15:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f32f4200640 1 --2- 172.21.15.98:0/1678775487 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32e404a880 0x7f32e40dec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f32f4200640 1 --2- 172.21.15.98:0/1678775487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404fed0 0x7f32e40502b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f32f4200640 1 -- 172.21.15.98:0/1678775487 >> 172.21.15.98:0/1678775487 conn(0x7f32e4051000 msgr2=0x7f32e4051410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fdf4d600640 1 -- 172.21.15.98:0/1705534729 shutdown_connections 2024-05-27T00:15:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fdf4d600640 1 -- 172.21.15.98:0/1705534729 wait complete. 2024-05-27T00:15:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f32f4200640 1 -- 172.21.15.98:0/1678775487 shutdown_connections 2024-05-27T00:15:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7f32f4200640 1 -- 172.21.15.98:0/1678775487 wait complete. 2024-05-27T00:15:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.509+0000 7fd22ea00640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2100678c0 0x7fd210069d80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd21c001600 tx=0x7fd21c007a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf4d600640 1 Processor -- start 2024-05-27T00:15:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f8da5200640 1 --2- 172.21.15.98:0/2667175253 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 0x7f8da01578b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000bb10 tx=0x7f8d9001ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93da7bf40a9813bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf4d600640 1 -- start start 2024-05-27T00:15:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f4200640 1 Processor -- start 2024-05-27T00:15:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f4200640 1 -- start start 2024-05-27T00:15:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f2b4be00640 1 --2- 172.21.15.98:0/748713958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c0c7a60 0x7f2b4c15ac40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400f370 tx=0x7f2b44020d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=426f61ea334e5662 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf4d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf4806e890 0x7fdf48159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf4d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf481590f0 0x7fdf481ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf4d600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf48162db0 0x7fdf4815d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf4d600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4815f610 con 0x7fdf48162db0 2024-05-27T00:15:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf4d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4815f330 con 0x7fdf481590f0 2024-05-27T00:15:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf4d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdf480c1d20 con 0x7fdf4806e890 2024-05-27T00:15:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf46a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf4806e890 0x7fdf48159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf46a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf4806e890 0x7fdf48159580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:53308/0 (socket says 172.21.15.98:53308) 2024-05-27T00:15:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf46a00640 1 -- 172.21.15.98:0/1905315670 learned_addr learned my addr 172.21.15.98:0/1905315670 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f4200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404a880 0x7f32e40d5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f4200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32e404aca0 0x7f32e40dd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf47400640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf48162db0 0x7fdf4815d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf46000640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf481590f0 0x7fdf481ef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f4200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e404fed0 0x7f32e40c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f4200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f32e4050c30 con 0x7f32e404fed0 2024-05-27T00:15:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f1800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404a880 0x7f32e40d5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f1800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404a880 0x7f32e40d5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:39080/0 (socket says 172.21.15.98:39080) 2024-05-27T00:15:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f1800640 1 -- 172.21.15.98:0/888368864 learned_addr learned my addr 172.21.15.98:0/888368864 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f32e4006250 con 0x7f32e404a880 2024-05-27T00:15:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f32e40ebdf0 con 0x7f32e404aca0 2024-05-27T00:15:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f0e00640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32e404aca0 0x7f32e40dd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf46a00640 1 -- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf481590f0 msgr2=0x7fdf481ef8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf46a00640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf481590f0 0x7fdf481ef8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf46a00640 1 -- 172.21.15.98:0/1905315670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf48162db0 msgr2=0x7fdf4815d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf46a00640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf48162db0 0x7fdf4815d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf46a00640 1 -- 172.21.15.98:0/1905315670 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf48159c30 con 0x7fdf4806e890 2024-05-27T00:15:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f2200640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e404fed0 0x7f32e40c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f0e00640 1 -- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404a880 msgr2=0x7f32e40d5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f0e00640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404a880 0x7f32e40d5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f0e00640 1 -- 172.21.15.98:0/888368864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e404fed0 msgr2=0x7f32e40c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f0e00640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e404fed0 0x7f32e40c88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f0e00640 1 -- 172.21.15.98:0/888368864 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e417be60 con 0x7f32e404aca0 2024-05-27T00:15:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f1800640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404a880 0x7f32e40d5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7f32f2200640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e404fed0 0x7f32e40c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.513+0000 7fdf46a00640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf4806e890 0x7fdf48159580 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdf400c3040 tx=0x7fdf4002cfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.517+0000 7f32f0e00640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32e404aca0 0x7f32e40dd7d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f32d800b410 tx=0x7f32d800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f8d9ea00640 1 -- 172.21.15.98:0/2667175253 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d900b4070 con 0x7f8da006ef40 2024-05-27T00:15:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f8d9ea00640 1 -- 172.21.15.98:0/2667175253 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d900a7b10 con 0x7f8da006ef40 2024-05-27T00:15:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7fdf3ea00640 1 -- 172.21.15.98:0/1905315670 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400c5040 con 0x7fdf4806e890 2024-05-27T00:15:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c0b1070 con 0x7fa5580039c0 2024-05-27T00:15:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f32e9600640 1 -- 172.21.15.98:0/888368864 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8015020 con 0x7f32e404aca0 2024-05-27T00:15:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e417b2e0 con 0x7f32e404aca0 2024-05-27T00:15:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7fd226a00640 1 -- 172.21.15.98:0/1897793723 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd2200181f0 con 0x7fd2280db470 2024-05-27T00:15:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55c0ac620 con 0x7fa5580039c0 2024-05-27T00:15:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7fa573400640 1 -- 172.21.15.98:0/4069796721 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c00a4d0 con 0x7fa5580039c0 2024-05-27T00:15:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7fdf4d600640 1 -- 172.21.15.98:0/1905315670 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf48159f10 con 0x7fdf4806e890 2024-05-27T00:15:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7fdf4d600640 1 -- 172.21.15.98:0/1905315670 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf4815bc30 con 0x7fdf4806e890 2024-05-27T00:15:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f2b4a000640 1 -- 172.21.15.98:0/748713958 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b440a9c50 con 0x7f2b4c0c7a60 2024-05-27T00:15:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f2b4a000640 1 -- 172.21.15.98:0/748713958 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b440a9df0 con 0x7f2b4c0c7a60 2024-05-27T00:15:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f2b4a000640 1 -- 172.21.15.98:0/748713958 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4400d2f0 con 0x7f2b4c0c7a60 2024-05-27T00:15:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f2b52800640 1 -- 172.21.15.98:0/748713958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c0c7a60 msgr2=0x7f2b4c15ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f2b52800640 1 --2- 172.21.15.98:0/748713958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c0c7a60 0x7f2b4c15ac40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400f370 tx=0x7f2b44020d70 comp rx=0 tx=0).stop 2024-05-27T00:15:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f8d9ea00640 1 -- 172.21.15.98:0/2667175253 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d900b4070 con 0x7f8da006ef40 2024-05-27T00:15:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f8da7200640 1 -- 172.21.15.98:0/2667175253 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 msgr2=0x7f8da01578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f8da7200640 1 --2- 172.21.15.98:0/2667175253 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 0x7f8da01578b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000bb10 tx=0x7f8d9001ec30 comp rx=0 tx=0).stop 2024-05-27T00:15:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7fa579c00640 1 -- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 msgr2=0x7fa558003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7fa579c00640 1 --2- 172.21.15.98:0/4069796721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 0x7fa558003dc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa5740cccb0 tx=0x7fa55c01c470 comp rx=0 tx=0).stop 2024-05-27T00:15:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f2b52800640 1 -- 172.21.15.98:0/748713958 shutdown_connections 2024-05-27T00:15:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e417b8a0 con 0x7f32e404aca0 2024-05-27T00:15:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f32e9600640 1 -- 172.21.15.98:0/888368864 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d8011830 con 0x7f32e404aca0 2024-05-27T00:15:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.521+0000 7f32e9600640 1 -- 172.21.15.98:0/888368864 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8011c20 con 0x7f32e404aca0 2024-05-27T00:15:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f8da7200640 1 -- 172.21.15.98:0/2667175253 shutdown_connections 2024-05-27T00:15:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f8da7200640 1 --2- 172.21.15.98:0/2667175253 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 0x7f8da01578b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f8da7200640 1 --2- 172.21.15.98:0/2667175253 >> v2:172.21.15.204:3300/0 conn(0x7f8da00c7e10 0x7f8da006ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f8da7200640 1 -- 172.21.15.98:0/2667175253 >> 172.21.15.98:0/2667175253 conn(0x7f8da00cc200 msgr2=0x7f8da00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f2b52800640 1 --2- 172.21.15.98:0/748713958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c0c7a60 0x7f2b4c15ac40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f2b52800640 1 --2- 172.21.15.98:0/748713958 >> v2:172.21.15.204:3300/0 conn(0x7f2b4c151920 0x7f2b4c073040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f2b52800640 1 -- 172.21.15.98:0/748713958 >> 172.21.15.98:0/748713958 conn(0x7f2b4c0cd060 msgr2=0x7f2b4c0cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7fa579200640 1 -- 172.21.15.98:0/4069796721 reap_dead start 2024-05-27T00:15:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7fdf3ea00640 1 -- 172.21.15.98:0/1905315670 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf400ca460 con 0x7fdf4806e890 2024-05-27T00:15:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7fdf3ea00640 1 -- 172.21.15.98:0/1905315670 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400ca850 con 0x7fdf4806e890 2024-05-27T00:15:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f2b52800640 1 -- 172.21.15.98:0/748713958 shutdown_connections 2024-05-27T00:15:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f2b52800640 1 -- 172.21.15.98:0/748713958 wait complete. 2024-05-27T00:15:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32c80052a0 con 0x7f32e404aca0 2024-05-27T00:15:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f8da7200640 1 -- 172.21.15.98:0/2667175253 shutdown_connections 2024-05-27T00:15:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7fa579c00640 1 -- 172.21.15.98:0/4069796721 shutdown_connections 2024-05-27T00:15:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7fa579c00640 1 -- 172.21.15.98:0/4069796721 >> 172.21.15.98:0/4069796721 conn(0x7fa57406e700 msgr2=0x7fa57406eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f8da7200640 1 -- 172.21.15.98:0/2667175253 wait complete. 2024-05-27T00:15:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7fdf4d600640 1 -- 172.21.15.98:0/1905315670 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf200052a0 con 0x7fdf4806e890 2024-05-27T00:15:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7fa579c00640 1 -- 172.21.15.98:0/4069796721 shutdown_connections 2024-05-27T00:15:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7fa579c00640 1 -- 172.21.15.98:0/4069796721 wait complete. 2024-05-27T00:15:34.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.525+0000 7f2b52800640 1 Processor -- start 2024-05-27T00:15:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8da7200640 1 Processor -- start 2024-05-27T00:15:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7fdf3ea00640 1 -- 172.21.15.98:0/1905315670 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf400d4030 con 0x7fdf4806e890 2024-05-27T00:15:34.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f32e9600640 1 -- 172.21.15.98:0/888368864 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f32d8018020 con 0x7f32e404aca0 2024-05-27T00:15:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7fa579c00640 1 Processor -- start 2024-05-27T00:15:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f32e9600640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32cc0678f0 0x7f32cc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b52800640 1 -- start start 2024-05-27T00:15:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b52800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b4c0c7a60 0x7f2b4c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b52800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b4c0c86a0 0x7f2b4c143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7fa579c00640 1 -- start start 2024-05-27T00:15:34.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b52800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c151920 0x7f2b4c1435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8da7200640 1 -- start start 2024-05-27T00:15:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8da7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 0x7f8da00cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8da7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da00c7e10 0x7f8da014e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8da7200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8da01519d0 0x7f8da0153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8da7200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8da006d180 con 0x7f8da01519d0 2024-05-27T00:15:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8da7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8da00c18b0 con 0x7f8da00c7e10 2024-05-27T00:15:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8da7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8da006d300 con 0x7f8da006ef40 2024-05-27T00:15:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b4aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b4c0c86a0 0x7f2b4c143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b4aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b4c0c86a0 0x7f2b4c143060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:39098/0 (socket says 172.21.15.98:39098) 2024-05-27T00:15:34.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b4b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b4c0c7a60 0x7f2b4c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b4b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b4c0c7a60 0x7f2b4c156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51106/0 (socket says 172.21.15.98:51106) 2024-05-27T00:15:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8d9f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da00c7e10 0x7f8da014e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8d9f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da00c7e10 0x7f8da014e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:39084/0 (socket says 172.21.15.98:39084) 2024-05-27T00:15:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8d9f400640 1 -- 172.21.15.98:0/3233128649 learned_addr learned my addr 172.21.15.98:0/3233128649 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8da5200640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8da01519d0 0x7f8da0153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7fdf3ea00640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf280678f0 0x7fdf28069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7fdf3ea00640 1 -- 172.21.15.98:0/1905315670 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fdf40002020 con 0x7fdf4806e890 2024-05-27T00:15:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f32e9600640 1 -- 172.21.15.98:0/888368864 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f32d80859c0 con 0x7f32e404aca0 2024-05-27T00:15:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b4aa00640 1 -- 172.21.15.98:0/2051486691 learned_addr learned my addr 172.21.15.98:0/2051486691 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7fdf46000640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf280678f0 0x7fdf28069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c0c3f90 con 0x7f2b4c0c7a60 2024-05-27T00:15:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f32f1800640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32cc0678f0 0x7f32cc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b4be00640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c151920 0x7f2b4c1435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c0c3e10 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2b4c067940 con 0x7f2b4c151920 2024-05-27T00:15:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8d9f400640 1 -- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 msgr2=0x7f8da00cb6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T00:15:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8d9f400640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 0x7f8da00cb6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8d9f400640 1 -- 172.21.15.98:0/3233128649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8da01519d0 msgr2=0x7f8da0153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8d9f400640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8da01519d0 0x7f8da0153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.529+0000 7f8d9f400640 1 -- 172.21.15.98:0/3233128649 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da0154990 con 0x7f8da00c7e10 2024-05-27T00:15:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 0x7fa5741ea300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5741f50a0 0x7fa5740722a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5741e90e0 0x7fa5741e9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa574070bd0 con 0x7fa5741e90e0 2024-05-27T00:15:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741629d0 con 0x7fa5741f50a0 2024-05-27T00:15:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa5740c7b50 con 0x7fa5580039c0 2024-05-27T00:15:34.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fdf46000640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf280678f0 0x7fdf28069db0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdf48071040 tx=0x7fdf38008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f8d9f400640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da00c7e10 0x7f8da014e3a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00e9e0 tx=0x7f8d8c00eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5741e90e0 0x7fa5741e9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa56b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5741f50a0 0x7fa5740722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5741e90e0 0x7fa5741e9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51130/0 (socket says 172.21.15.98:51130) 2024-05-27T00:15:34.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579200640 1 -- 172.21.15.98:0/3472638821 learned_addr learned my addr 172.21.15.98:0/3472638821 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b4aa00640 1 -- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c151920 msgr2=0x7f2b4c1435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b4aa00640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c151920 0x7f2b4c1435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b4aa00640 1 -- 172.21.15.98:0/2051486691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b4c0c7a60 msgr2=0x7f2b4c156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa573e00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 0x7fa5741ea300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f32f1800640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32cc0678f0 0x7f32cc069db0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f32e001ab90 tx=0x7f32e00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579200640 1 -- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 msgr2=0x7fa5741ea300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579200640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 0x7fa5741ea300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579200640 1 -- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5741f50a0 msgr2=0x7fa5740722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579200640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5741f50a0 0x7fa5740722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579200640 1 -- 172.21.15.98:0/3472638821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5741f5a90 con 0x7fa5741e90e0 2024-05-27T00:15:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa573e00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 0x7fa5741ea300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f8d97e00640 1 -- 172.21.15.98:0/3233128649 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c017070 con 0x7f8da00c7e10 2024-05-27T00:15:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f8d97e00640 1 -- 172.21.15.98:0/3233128649 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d8c00c9d0 con 0x7f8da00c7e10 2024-05-27T00:15:34.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f8da7200640 1 -- 172.21.15.98:0/3233128649 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da01f29d0 con 0x7f8da00c7e10 2024-05-27T00:15:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b4aa00640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b4c0c7a60 0x7f2b4c156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b4aa00640 1 -- 172.21.15.98:0/2051486691 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c15ae80 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f8d97e00640 1 -- 172.21.15.98:0/3233128649 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00cda0 con 0x7f8da00c7e10 2024-05-27T00:15:34.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f8da7200640 1 -- 172.21.15.98:0/3233128649 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da01f2f90 con 0x7f8da00c7e10 2024-05-27T00:15:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b4b400640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b4c0c7a60 0x7f2b4c156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579200640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5741e90e0 0x7fa5741e9560 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa5740715e0 tx=0x7fa56c00f860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa570c00640 1 -- 172.21.15.98:0/3472638821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0052e0 con 0x7fa5741e90e0 2024-05-27T00:15:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa570c00640 1 -- 172.21.15.98:0/3472638821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa56c018070 con 0x7fa5741e90e0 2024-05-27T00:15:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa570c00640 1 -- 172.21.15.98:0/3472638821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0054a0 con 0x7fa5741e90e0 2024-05-27T00:15:34.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579c00640 1 -- 172.21.15.98:0/3472638821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5741e7d10 con 0x7fa5741e90e0 2024-05-27T00:15:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b4aa00640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b4c0c86a0 0x7f2b4c143060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800c490 tx=0x7f2b3800c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f8da7200640 1 -- 172.21.15.98:0/3233128649 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da0075cb0 con 0x7f8da00c7e10 2024-05-27T00:15:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579c00640 1 -- 172.21.15.98:0/3472638821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5741e82d0 con 0x7fa5741e90e0 2024-05-27T00:15:34.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b43400640 1 -- 172.21.15.98:0/2051486691 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38018070 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b43400640 1 -- 172.21.15.98:0/2051486691 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b38007480 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b43400640 1 -- 172.21.15.98:0/2051486691 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b380047f0 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4c144e30 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7fa579c00640 1 -- 172.21.15.98:0/3472638821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa54c0052a0 con 0x7fa5741e90e0 2024-05-27T00:15:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.533+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4c150a10 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.537+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b4c075cb0 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.537+0000 7fa570c00640 1 -- 172.21.15.98:0/3472638821 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa56c003840 con 0x7fa5741e90e0 2024-05-27T00:15:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.537+0000 7f8d97e00640 1 -- 172.21.15.98:0/3233128649 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8d8c01a070 con 0x7f8da00c7e10 2024-05-27T00:15:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.537+0000 7fa570c00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa550067910 0x7fa550069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.537+0000 7f8d97e00640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d800678c0 0x7f8d80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.537+0000 7f8d97e00640 1 -- 172.21.15.98:0/3233128649 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f8d8c086610 con 0x7f8da00c7e10 2024-05-27T00:15:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.537+0000 7fa570c00640 1 -- 172.21.15.98:0/3472638821 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa56c054760 con 0x7fa5741e90e0 2024-05-27T00:15:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.537+0000 7f8d9fe00640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d800678c0 0x7f8d80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.537+0000 7fa573e00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa550067910 0x7fa550069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.541+0000 7f8d9fe00640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d800678c0 0x7f8d80069d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8d98007fd0 tx=0x7f8d98007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.541+0000 7fa573e00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa550067910 0x7fa550069dd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5740c6b10 tx=0x7fa55c0046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.541+0000 7fa570c00640 1 -- 172.21.15.98:0/3472638821 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa5741f5a90 con 0x7fa5741e90e0 2024-05-27T00:15:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.541+0000 7f32e9600640 1 -- 172.21.15.98:0/888368864 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f32d801f020 con 0x7f32e404aca0 2024-05-27T00:15:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.549+0000 7f8d97e00640 1 -- 172.21.15.98:0/3233128649 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8d8c0570a0 con 0x7f8da00c7e10 2024-05-27T00:15:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.553+0000 7f2b43400640 1 -- 172.21.15.98:0/2051486691 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2b3801b070 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.553+0000 7f2b43400640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2b2c067990 0x7f2b2c069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.553+0000 7f2b43400640 1 -- 172.21.15.98:0/2051486691 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2b38085460 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.557+0000 7fdf3ea00640 1 -- 172.21.15.98:0/1905315670 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fdf400db020 con 0x7fdf4806e890 2024-05-27T00:15:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.557+0000 7f2b4b400640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2b2c067990 0x7f2b2c069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.561+0000 7f2b4b400640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2b2c067990 0x7f2b2c069e50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c00bf50 tx=0x7f2b3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.569+0000 7f2b43400640 1 -- 172.21.15.98:0/2051486691 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f2b38022020 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.705+0000 7f44f3200640 1 -- 172.21.15.98:0/3474471890 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f44e804c0c0 con 0x7f44d40051d0 2024-05-27T00:15:34.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.709+0000 7f44e6000640 1 -- 172.21.15.98:0/3474471890 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f44e80d8a20 con 0x7f44d40051d0 2024-05-27T00:15:34.711 INFO:teuthology.orchestra.run.smithi098.stdout:60129542152 2024-05-27T00:15:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 -- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44d0067cc0 msgr2=0x7f44d006a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44d0067cc0 0x7f44d006a180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f44e804f5a0 tx=0x7f44dc008040 comp rx=0 tx=0).stop 2024-05-27T00:15:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 -- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40051d0 msgr2=0x7f44e80d9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40051d0 0x7f44e80d9920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f44e0000c10 tx=0x7f44e0007750 comp rx=0 tx=0).stop 2024-05-27T00:15:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 -- 172.21.15.98:0/3474471890 shutdown_connections 2024-05-27T00:15:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44e80c7dc0 0x7f44e80ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44d0067cc0 0x7f44d006a180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44e80d5680 0x7f44e81703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 --2- 172.21.15.98:0/3474471890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f44d40051d0 0x7f44e80d9920 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 -- 172.21.15.98:0/3474471890 >> 172.21.15.98:0/3474471890 conn(0x7f44e8050190 msgr2=0x7f44e80d2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 -- 172.21.15.98:0/3474471890 shutdown_connections 2024-05-27T00:15:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.713+0000 7f44f3200640 1 -- 172.21.15.98:0/3474471890 wait complete. 2024-05-27T00:15:34.748 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2024-05-27T00:15:34.749 DEBUG:teuthology.parallel:result is None 2024-05-27T00:15:34.790 INFO:teuthology.orchestra.run.smithi098.stdout:60129542151 2024-05-27T00:15:34.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.785+0000 7fd6ba000640 1 -- 172.21.15.98:0/4081958155 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd69c005860 con 0x7fd6c8150b30 2024-05-27T00:15:34.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.789+0000 7fd6c4c00640 1 -- 172.21.15.98:0/4081958155 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd6c8145fa0 con 0x7fd6c8150b30 2024-05-27T00:15:34.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 -- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd6a80678c0 msgr2=0x7fd6a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd6a80678c0 0x7fd6a8069d80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0011f0 tx=0x7fd6bc007d40 comp rx=0 tx=0).stop 2024-05-27T00:15:34.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 -- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8150b30 msgr2=0x7fd6c8149250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8150b30 0x7fd6c8149250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd6c00b0040 tx=0x7fd6c0019710 comp rx=0 tx=0).stop 2024-05-27T00:15:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 -- 172.21.15.98:0/4081958155 shutdown_connections 2024-05-27T00:15:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6c8150b30 0x7fd6c8149250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6c80c1a60 0x7fd6c8148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd6a80678c0 0x7fd6a8069d80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 --2- 172.21.15.98:0/4081958155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6c8070b60 0x7fd6c81455f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 -- 172.21.15.98:0/4081958155 >> 172.21.15.98:0/4081958155 conn(0x7fd6c806e750 msgr2=0x7fd6c80c0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 -- 172.21.15.98:0/4081958155 shutdown_connections 2024-05-27T00:15:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.793+0000 7fd6ba000640 1 -- 172.21.15.98:0/4081958155 wait complete. 2024-05-27T00:15:34.833 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.0 2024-05-27T00:15:34.833 DEBUG:teuthology.parallel:result is None 2024-05-27T00:15:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.849+0000 7f8da7200640 1 -- 172.21.15.98:0/3233128649 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f8da0157bb0 con 0x7f8da00c7e10 2024-05-27T00:15:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.849+0000 7f8d97e00640 1 -- 172.21.15.98:0/3233128649 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8d8c0569a0 con 0x7f8da00c7e10 2024-05-27T00:15:34.854 INFO:teuthology.orchestra.run.smithi098.stdout:60129542152 2024-05-27T00:15:34.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 -- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d800678c0 msgr2=0x7f8d80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d800678c0 0x7f8d80069d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8d98007fd0 tx=0x7f8d98007d60 comp rx=0 tx=0).stop 2024-05-27T00:15:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 -- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da00c7e10 msgr2=0x7f8da014e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da00c7e10 0x7f8da014e3a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00e9e0 tx=0x7f8d8c00eea0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 -- 172.21.15.98:0/3233128649 shutdown_connections 2024-05-27T00:15:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8da01519d0 0x7f8da0153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8d800678c0 0x7f8d80069d80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da00c7e10 0x7f8da014e3a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 --2- 172.21.15.98:0/3233128649 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da006ef40 0x7f8da00cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 -- 172.21.15.98:0/3233128649 >> 172.21.15.98:0/3233128649 conn(0x7f8da00cc200 msgr2=0x7f8da00cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 -- 172.21.15.98:0/3233128649 shutdown_connections 2024-05-27T00:15:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.853+0000 7f8da7200640 1 -- 172.21.15.98:0/3233128649 wait complete. 2024-05-27T00:15:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.857+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd208005590 con 0x7fd2280db470 2024-05-27T00:15:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.857+0000 7fa579c00640 1 -- 172.21.15.98:0/3472638821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa54c005860 con 0x7fa5741e90e0 2024-05-27T00:15:34.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.857+0000 7fd226a00640 1 -- 172.21.15.98:0/1897793723 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd2200ef9d0 con 0x7fd2280db470 2024-05-27T00:15:34.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.857+0000 7fa570c00640 1 -- 172.21.15.98:0/3472638821 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa56c056dc0 con 0x7fa5741e90e0 2024-05-27T00:15:34.862 INFO:teuthology.orchestra.run.smithi098.stdout:60129542151 2024-05-27T00:15:34.862 INFO:teuthology.orchestra.run.smithi098.stdout:60129542152 2024-05-27T00:15:34.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fdf4d600640 1 -- 172.21.15.98:0/1905315670 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fdf20005590 con 0x7fdf4806e890 2024-05-27T00:15:34.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2100678c0 msgr2=0x7fd210069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fdf3ea00640 1 -- 172.21.15.98:0/1905315670 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdf40029d70 con 0x7fdf4806e890 2024-05-27T00:15:34.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2100678c0 0x7fd210069d80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd21c001600 tx=0x7fd21c007a50 comp rx=0 tx=0).stop 2024-05-27T00:15:34.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2280db470 msgr2=0x7fd228049c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2280db470 0x7fd228049c70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd2200ae000 tx=0x7fd22001b3a0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.865 INFO:teuthology.orchestra.run.smithi098.stdout:60129542153 2024-05-27T00:15:34.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 shutdown_connections 2024-05-27T00:15:34.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2280db470 0x7fd228049c70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2100678c0 0x7fd210069d80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2280d5030 0x7fd2280d9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 --2- 172.21.15.98:0/1897793723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2280347d0 0x7fd22817d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 >> 172.21.15.98:0/1897793723 conn(0x7fd2280501c0 msgr2=0x7fd2280d3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 shutdown_connections 2024-05-27T00:15:34.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.861+0000 7fd235600640 1 -- 172.21.15.98:0/1897793723 wait complete. 2024-05-27T00:15:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 -- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa550067910 msgr2=0x7fa550069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa550067910 0x7fa550069dd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5740c6b10 tx=0x7fa55c0046d0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 -- 172.21.15.98:0/3472638821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5741e90e0 msgr2=0x7fa5741e9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5741e90e0 0x7fa5741e9560 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa5740715e0 tx=0x7fa56c00f860 comp rx=0 tx=0).stop 2024-05-27T00:15:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 -- 172.21.15.98:0/3472638821 shutdown_connections 2024-05-27T00:15:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5741e90e0 0x7fa5741e9560 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa550067910 0x7fa550069dd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5741f50a0 0x7fa5740722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 --2- 172.21.15.98:0/3472638821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5580039c0 0x7fa5741ea300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 -- 172.21.15.98:0/3472638821 >> 172.21.15.98:0/3472638821 conn(0x7fa57406e700 msgr2=0x7fa5740c47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 -- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf280678f0 msgr2=0x7fdf28069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf280678f0 0x7fdf28069db0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdf48071040 tx=0x7fdf38008040 comp rx=0 tx=0).stop 2024-05-27T00:15:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 -- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf4806e890 msgr2=0x7fdf48159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf4806e890 0x7fdf48159580 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdf400c3040 tx=0x7fdf4002cfb0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 -- 172.21.15.98:0/1905315670 shutdown_connections 2024-05-27T00:15:34.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf48162db0 0x7fdf4815d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf280678f0 0x7fdf28069db0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf481590f0 0x7fdf481ef8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 --2- 172.21.15.98:0/1905315670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf4806e890 0x7fdf48159580 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 -- 172.21.15.98:0/1905315670 >> 172.21.15.98:0/1905315670 conn(0x7fdf480cc200 msgr2=0x7fdf480cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 -- 172.21.15.98:0/3472638821 shutdown_connections 2024-05-27T00:15:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fa579c00640 1 -- 172.21.15.98:0/3472638821 wait complete. 2024-05-27T00:15:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 -- 172.21.15.98:0/1905315670 shutdown_connections 2024-05-27T00:15:34.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.865+0000 7fdf4d600640 1 -- 172.21.15.98:0/1905315670 wait complete. 2024-05-27T00:15:34.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.873+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f32c8005590 con 0x7f32e404aca0 2024-05-27T00:15:34.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.877+0000 7f32e9600640 1 -- 172.21.15.98:0/888368864 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f32d8055e10 con 0x7f32e404aca0 2024-05-27T00:15:34.879 INFO:teuthology.orchestra.run.smithi098.stdout:60129542151 2024-05-27T00:15:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.877+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32cc0678f0 msgr2=0x7f32cc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.877+0000 7f32f4200640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32cc0678f0 0x7f32cc069db0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f32e001ab90 tx=0x7f32e00a6000 comp rx=0 tx=0).stop 2024-05-27T00:15:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.881+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32e404aca0 msgr2=0x7f32e40dd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.881+0000 7f32f4200640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32e404aca0 0x7f32e40dd7d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f32d800b410 tx=0x7f32d800b8d0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.881+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 shutdown_connections 2024-05-27T00:15:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.881+0000 7f32f4200640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32e404fed0 0x7f32e40c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.881+0000 7f32f4200640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32e404aca0 0x7f32e40dd7d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.881+0000 7f32f4200640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32cc0678f0 0x7f32cc069db0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.881+0000 7f32f4200640 1 --2- 172.21.15.98:0/888368864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32e404a880 0x7f32e40d5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.881+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 >> 172.21.15.98:0/888368864 conn(0x7f32e4051000 msgr2=0x7f32e40d6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.881+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 shutdown_connections 2024-05-27T00:15:34.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.881+0000 7f32f4200640 1 -- 172.21.15.98:0/888368864 wait complete. 2024-05-27T00:15:34.885 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.3 2024-05-27T00:15:34.886 DEBUG:teuthology.parallel:result is None 2024-05-27T00:15:34.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.2 2024-05-27T00:15:34.900 DEBUG:teuthology.parallel:result is None 2024-05-27T00:15:34.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.6 2024-05-27T00:15:34.902 DEBUG:teuthology.parallel:result is None 2024-05-27T00:15:34.906 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.5 2024-05-27T00:15:34.906 DEBUG:teuthology.parallel:result is None 2024-05-27T00:15:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.905+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2b4c0c8270 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.905+0000 7f2b43400640 1 -- 172.21.15.98:0/2051486691 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2b380558a0 con 0x7f2b4c0c86a0 2024-05-27T00:15:34.908 INFO:teuthology.orchestra.run.smithi098.stdout:60129542151 2024-05-27T00:15:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2b2c067990 msgr2=0x7f2b2c069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2b2c067990 0x7f2b2c069e50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c00bf50 tx=0x7f2b3c008040 comp rx=0 tx=0).stop 2024-05-27T00:15:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b4c0c86a0 msgr2=0x7f2b4c143060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b4c0c86a0 0x7f2b4c143060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800c490 tx=0x7f2b3800c950 comp rx=0 tx=0).stop 2024-05-27T00:15:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 shutdown_connections 2024-05-27T00:15:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2b4c151920 0x7f2b4c1435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2b2c067990 0x7f2b2c069e50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2b4c0c86a0 0x7f2b4c143060 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 --2- 172.21.15.98:0/2051486691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b4c0c7a60 0x7f2b4c156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 >> 172.21.15.98:0/2051486691 conn(0x7f2b4c0cd060 msgr2=0x7f2b4c0cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 shutdown_connections 2024-05-27T00:15:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:34.909+0000 7f2b52800640 1 -- 172.21.15.98:0/2051486691 wait complete. 2024-05-27T00:15:34.913 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.7 2024-05-27T00:15:34.914 DEBUG:teuthology.parallel:result is None 2024-05-27T00:15:34.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.4 2024-05-27T00:15:34.931 DEBUG:teuthology.parallel:result is None 2024-05-27T00:15:34.931 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-27T00:15:34.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:15:35.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.105+0000 7f941ec00640 1 Processor -- start 2024-05-27T00:15:35.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.105+0000 7f941ec00640 1 -- start start 2024-05-27T00:15:35.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.105+0000 7f941ec00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cbe10 0x7f94181592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.105+0000 7f941ec00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9418073570 0x7f94181559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.105+0000 7f941ec00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94180c1f60 con 0x7f94180cc1f0 2024-05-27T00:15:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.105+0000 7f941ec00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9418159810 con 0x7f9418073570 2024-05-27T00:15:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.105+0000 7f941ec00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f94180c6dd0 con 0x7f94180cbe10 2024-05-27T00:15:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.105+0000 7f9417400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f94180cc1f0 0x7f94180c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47098/0 (socket says 172.21.15.98:47098) 2024-05-27T00:15:35.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.105+0000 7f9417400640 1 -- 172.21.15.98:0/834551745 learned_addr learned my addr 172.21.15.98:0/834551745 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.105+0000 7f9417e00640 1 --2- 172.21.15.98:0/834551745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cbe10 0x7f94181592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f941cc00640 1 --2- 172.21.15.98:0/834551745 >> v2:172.21.15.204:3300/0 conn(0x7f9418073570 0x7f94181559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f9416a00640 1 -- 172.21.15.98:0/834551745 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375305629 0 0) 0x7f94180c1f60 con 0x7f94180cc1f0 2024-05-27T00:15:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f9416a00640 1 -- 172.21.15.98:0/834551745 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93fc003530 con 0x7f94180cc1f0 2024-05-27T00:15:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f9417e00640 1 -- 172.21.15.98:0/834551745 >> v2:172.21.15.204:3300/0 conn(0x7f9418073570 msgr2=0x7f94181559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f9417e00640 1 --2- 172.21.15.98:0/834551745 >> v2:172.21.15.204:3300/0 conn(0x7f9418073570 0x7f94181559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f9417e00640 1 -- 172.21.15.98:0/834551745 >> v1:172.21.15.98:6789/0 conn(0x7f94180cc1f0 legacy=0x7f94180c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f9417e00640 1 -- 172.21.15.98:0/834551745 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94180c70b0 con 0x7f94180cbe10 2024-05-27T00:15:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f9417e00640 1 --2- 172.21.15.98:0/834551745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cbe10 0x7f94181592d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9408009d90 tx=0x7f940801f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af3df7cbddb9c5ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:35.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f9416a00640 1 -- 172.21.15.98:0/834551745 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94080b4070 con 0x7f94180cbe10 2024-05-27T00:15:35.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f9416a00640 1 -- 172.21.15.98:0/834551745 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94080af600 con 0x7f94180cbe10 2024-05-27T00:15:35.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.109+0000 7f9416a00640 1 -- 172.21.15.98:0/834551745 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94080af9b0 con 0x7f94180cbe10 2024-05-27T00:15:35.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 -- 172.21.15.98:0/834551745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cbe10 msgr2=0x7f94181592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 --2- 172.21.15.98:0/834551745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cbe10 0x7f94181592d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9408009d90 tx=0x7f940801f150 comp rx=0 tx=0).stop 2024-05-27T00:15:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 -- 172.21.15.98:0/834551745 shutdown_connections 2024-05-27T00:15:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 --2- 172.21.15.98:0/834551745 >> v2:172.21.15.204:3300/0 conn(0x7f9418073570 0x7f94181559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 --2- 172.21.15.98:0/834551745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cbe10 0x7f94181592d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 -- 172.21.15.98:0/834551745 >> 172.21.15.98:0/834551745 conn(0x7f9418067530 msgr2=0x7f941806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 -- 172.21.15.98:0/834551745 shutdown_connections 2024-05-27T00:15:35.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 -- 172.21.15.98:0/834551745 wait complete. 2024-05-27T00:15:35.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 Processor -- start 2024-05-27T00:15:35.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 -- start start 2024-05-27T00:15:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9418073570 0x7f94180720c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94180cbe10 0x7f941815a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cc1f0 0x7f941815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94180c1330 con 0x7f94180cbe10 2024-05-27T00:15:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f94180c1050 con 0x7f9418073570 2024-05-27T00:15:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941ec00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f94180c79d0 con 0x7f94180cc1f0 2024-05-27T00:15:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f9417400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94180cbe10 0x7f941815a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f9417400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94180cbe10 0x7f941815a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51144/0 (socket says 172.21.15.98:51144) 2024-05-27T00:15:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f9417400640 1 -- 172.21.15.98:0/3464103706 learned_addr learned my addr 172.21.15.98:0/3464103706 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f941cc00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cc1f0 0x7f941815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f9417e00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9418073570 0x7f94180720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f9417400640 1 -- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cc1f0 msgr2=0x7f941815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f9417400640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cc1f0 0x7f941815ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f9417400640 1 -- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9418073570 msgr2=0x7f94180720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f9417400640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9418073570 0x7f94180720c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.113+0000 7f9417400640 1 -- 172.21.15.98:0/3464103706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94181f36f0 con 0x7f94180cbe10 2024-05-27T00:15:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f9417400640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94180cbe10 0x7f941815a820 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f940400c3c0 tx=0x7f940400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f940fe00640 1 -- 172.21.15.98:0/3464103706 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9404017070 con 0x7f94180cbe10 2024-05-27T00:15:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f940fe00640 1 -- 172.21.15.98:0/3464103706 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9404005d10 con 0x7f94180cbe10 2024-05-27T00:15:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f941ec00640 1 -- 172.21.15.98:0/3464103706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9418072a50 con 0x7f94180cbe10 2024-05-27T00:15:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f940fe00640 1 -- 172.21.15.98:0/3464103706 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9404007310 con 0x7f94180cbe10 2024-05-27T00:15:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f941ec00640 1 -- 172.21.15.98:0/3464103706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94180c3a70 con 0x7f94180cbe10 2024-05-27T00:15:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f941ec00640 1 -- 172.21.15.98:0/3464103706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9418075cb0 con 0x7f94180cbe10 2024-05-27T00:15:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f940fe00640 1 -- 172.21.15.98:0/3464103706 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f940401a070 con 0x7f94180cbe10 2024-05-27T00:15:35.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f940fe00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93f80678c0 0x7f93f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f940fe00640 1 -- 172.21.15.98:0/3464103706 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f94040850e0 con 0x7f94180cbe10 2024-05-27T00:15:35.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.117+0000 7f9417e00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93f80678c0 0x7f93f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.121+0000 7f9417e00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93f80678c0 0x7f93f8069d80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f94080b1000 tx=0x7f94080aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:35.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.121+0000 7f940fe00640 1 -- 172.21.15.98:0/3464103706 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9404055560 con 0x7f94180cbe10 2024-05-27T00:15:35.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.277+0000 7f941ec00640 1 -- 172.21.15.98:0/3464103706 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f941806f2e0 con 0x7f93f80678c0 2024-05-27T00:15:35.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.297+0000 7f940fe00640 1 -- 172.21.15.98:0/3464103706 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251835 (secure 0 0 0) 0x7f941806f2e0 con 0x7f93f80678c0 2024-05-27T00:15:35.302 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:15:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 -- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93f80678c0 msgr2=0x7f93f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93f80678c0 0x7f93f8069d80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f94080b1000 tx=0x7f94080aa000 comp rx=0 tx=0).stop 2024-05-27T00:15:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 -- 172.21.15.98:0/3464103706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94180cbe10 msgr2=0x7f941815a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94180cbe10 0x7f941815a820 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f940400c3c0 tx=0x7f940400c880 comp rx=0 tx=0).stop 2024-05-27T00:15:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 -- 172.21.15.98:0/3464103706 shutdown_connections 2024-05-27T00:15:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94180cc1f0 0x7f941815ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94180cbe10 0x7f941815a820 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93f80678c0 0x7f93f8069d80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 --2- 172.21.15.98:0/3464103706 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9418073570 0x7f94180720c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 -- 172.21.15.98:0/3464103706 >> 172.21.15.98:0/3464103706 conn(0x7f9418067530 msgr2=0x7f94180c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 -- 172.21.15.98:0/3464103706 shutdown_connections 2024-05-27T00:15:35.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.301+0000 7f941ec00640 1 -- 172.21.15.98:0/3464103706 wait complete. 2024-05-27T00:15:35.321 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-05-27T00:15:35.136220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219112,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755608,"statfs":{"total":773094113280,"available":772869742592,"internally_reserved":0,"allocated":3055616,"data_stored":1816820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.556163"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748978+0000","last_change":"2024-05-27T00:15:25.640698+0000","last_active":"2024-05-27T00:15:26.748978+0000","last_peered":"2024-05-27T00:15:26.748978+0000","last_clean":"2024-05-27T00:15:26.748978+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:15:26.748978+0000","last_undegraded":"2024-05-27T00:15:26.748978+0000","last_fullsized":"2024-05-27T00:15:26.748978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:47:27.030637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583615+0000","last_change":"2024-05-27T00:15:25.598672+0000","last_active":"2024-05-27T00:15:26.583615+0000","last_peered":"2024-05-27T00:15:26.583615+0000","last_clean":"2024-05-27T00:15:26.583615+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:15:26.583615+0000","last_undegraded":"2024-05-27T00:15:26.583615+0000","last_fullsized":"2024-05-27T00:15:26.583615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745467+0000","last_change":"2024-05-27T00:15:25.593316+0000","last_active":"2024-05-27T00:15:26.745467+0000","last_peered":"2024-05-27T00:15:26.745467+0000","last_clean":"2024-05-27T00:15:26.745467+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:15:26.745467+0000","last_undegraded":"2024-05-27T00:15:26.745467+0000","last_fullsized":"2024-05-27T00:15:26.745467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587216+0000","last_change":"2024-05-27T00:15:25.620237+0000","last_active":"2024-05-27T00:15:26.587216+0000","last_peered":"2024-05-27T00:15:26.587216+0000","last_clean":"2024-05-27T00:15:26.587216+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:15:26.587216+0000","last_undegraded":"2024-05-27T00:15:26.587216+0000","last_fullsized":"2024-05-27T00:15:26.587216+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:17:53.053461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748475+0000","last_change":"2024-05-27T00:15:25.640146+0000","last_active":"2024-05-27T00:15:26.748475+0000","last_peered":"2024-05-27T00:15:26.748475+0000","last_clean":"2024-05-27T00:15:26.748475+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:15:26.748475+0000","last_undegraded":"2024-05-27T00:15:26.748475+0000","last_fullsized":"2024-05-27T00:15:26.748475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:41:46.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589129+0000","last_change":"2024-05-27T00:15:25.627128+0000","last_active":"2024-05-27T00:15:26.589129+0000","last_peered":"2024-05-27T00:15:26.589129+0000","last_clean":"2024-05-27T00:15:26.589129+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:15:26.589129+0000","last_undegraded":"2024-05-27T00:15:26.589129+0000","last_fullsized":"2024-05-27T00:15:26.589129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:47:27.030637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746743+0000","last_change":"2024-05-27T00:15:25.630446+0000","last_active":"2024-05-27T00:15:26.746743+0000","last_peered":"2024-05-27T00:15:26.746743+0000","last_clean":"2024-05-27T00:15:26.746743+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:15:26.746743+0000","last_undegraded":"2024-05-27T00:15:26.746743+0000","last_fullsized":"2024-05-27T00:15:26.746743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:59.804368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744980+0000","last_change":"2024-05-27T00:15:25.612157+0000","last_active":"2024-05-27T00:15:26.744980+0000","last_peered":"2024-05-27T00:15:26.744980+0000","last_clean":"2024-05-27T00:15:26.744980+0000","last_became_active":"2024-05-27T00:15:25.611884+0000","last_became_peered":"2024-05-27T00:15:25.611884+0000","last_unstale":"2024-05-27T00:15:26.744980+0000","last_undegraded":"2024-05-27T00:15:26.744980+0000","last_fullsized":"2024-05-27T00:15:26.744980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:12:28.505840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749289+0000","last_change":"2024-05-27T00:15:25.643312+0000","last_active":"2024-05-27T00:15:26.749289+0000","last_peered":"2024-05-27T00:15:26.749289+0000","last_clean":"2024-05-27T00:15:26.749289+0000","last_became_active":"2024-05-27T00:15:25.642663+0000","last_became_peered":"2024-05-27T00:15:25.642663+0000","last_unstale":"2024-05-27T00:15:26.749289+0000","last_undegraded":"2024-05-27T00:15:26.749289+0000","last_fullsized":"2024-05-27T00:15:26.749289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:06:27.232620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747471+0000","last_change":"2024-05-27T00:15:25.629892+0000","last_active":"2024-05-27T00:15:26.747471+0000","last_peered":"2024-05-27T00:15:26.747471+0000","last_clean":"2024-05-27T00:15:26.747471+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:15:26.747471+0000","last_undegraded":"2024-05-27T00:15:26.747471+0000","last_fullsized":"2024-05-27T00:15:26.747471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747909+0000","last_change":"2024-05-27T00:15:25.614887+0000","last_active":"2024-05-27T00:15:26.747909+0000","last_peered":"2024-05-27T00:15:26.747909+0000","last_clean":"2024-05-27T00:15:26.747909+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:15:26.747909+0000","last_undegraded":"2024-05-27T00:15:26.747909+0000","last_fullsized":"2024-05-27T00:15:26.747909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588218+0000","last_change":"2024-05-27T00:15:25.623864+0000","last_active":"2024-05-27T00:15:26.588218+0000","last_peered":"2024-05-27T00:15:26.588218+0000","last_clean":"2024-05-27T00:15:26.588218+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:15:26.588218+0000","last_undegraded":"2024-05-27T00:15:26.588218+0000","last_fullsized":"2024-05-27T00:15:26.588218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746025+0000","last_change":"2024-05-27T00:15:25.615495+0000","last_active":"2024-05-27T00:15:26.746025+0000","last_peered":"2024-05-27T00:15:26.746025+0000","last_clean":"2024-05-27T00:15:26.746025+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:15:26.746025+0000","last_undegraded":"2024-05-27T00:15:26.746025+0000","last_fullsized":"2024-05-27T00:15:26.746025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:42:53.882749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745273+0000","last_change":"2024-05-27T00:15:25.594508+0000","last_active":"2024-05-27T00:15:26.745273+0000","last_peered":"2024-05-27T00:15:26.745273+0000","last_clean":"2024-05-27T00:15:26.745273+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:15:26.745273+0000","last_undegraded":"2024-05-27T00:15:26.745273+0000","last_fullsized":"2024-05-27T00:15:26.745273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583449+0000","last_change":"2024-05-27T00:15:25.611818+0000","last_active":"2024-05-27T00:15:26.583449+0000","last_peered":"2024-05-27T00:15:26.583449+0000","last_clean":"2024-05-27T00:15:26.583449+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:15:26.583449+0000","last_undegraded":"2024-05-27T00:15:26.583449+0000","last_fullsized":"2024-05-27T00:15:26.583449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586675+0000","last_change":"2024-05-27T00:15:25.611365+0000","last_active":"2024-05-27T00:15:26.586675+0000","last_peered":"2024-05-27T00:15:26.586675+0000","last_clean":"2024-05-27T00:15:26.586675+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:15:26.586675+0000","last_undegraded":"2024-05-27T00:15:26.586675+0000","last_fullsized":"2024-05-27T00:15:26.586675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.590092+0000","last_change":"2024-05-27T00:15:25.632120+0000","last_active":"2024-05-27T00:15:26.590092+0000","last_peered":"2024-05-27T00:15:26.590092+0000","last_clean":"2024-05-27T00:15:26.590092+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:15:26.590092+0000","last_undegraded":"2024-05-27T00:15:26.590092+0000","last_fullsized":"2024-05-27T00:15:26.590092+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:06:27.232620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745618+0000","last_change":"2024-05-27T00:15:25.615868+0000","last_active":"2024-05-27T00:15:26.745618+0000","last_peered":"2024-05-27T00:15:26.745618+0000","last_clean":"2024-05-27T00:15:26.745618+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:15:26.745618+0000","last_undegraded":"2024-05-27T00:15:26.745618+0000","last_fullsized":"2024-05-27T00:15:26.745618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749314+0000","last_change":"2024-05-27T00:15:25.629101+0000","last_active":"2024-05-27T00:15:26.749314+0000","last_peered":"2024-05-27T00:15:26.749314+0000","last_clean":"2024-05-27T00:15:26.749314+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:15:26.749314+0000","last_undegraded":"2024-05-27T00:15:26.749314+0000","last_fullsized":"2024-05-27T00:15:26.749314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:26.893574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747537+0000","last_change":"2024-05-27T00:15:25.619351+0000","last_active":"2024-05-27T00:15:26.747537+0000","last_peered":"2024-05-27T00:15:26.747537+0000","last_clean":"2024-05-27T00:15:26.747537+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:15:26.747537+0000","last_undegraded":"2024-05-27T00:15:26.747537+0000","last_fullsized":"2024-05-27T00:15:26.747537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746054+0000","last_change":"2024-05-27T00:15:25.616436+0000","last_active":"2024-05-27T00:15:26.746054+0000","last_peered":"2024-05-27T00:15:26.746054+0000","last_clean":"2024-05-27T00:15:26.746054+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:15:26.746054+0000","last_undegraded":"2024-05-27T00:15:26.746054+0000","last_fullsized":"2024-05-27T00:15:26.746054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586830+0000","last_change":"2024-05-27T00:15:25.629373+0000","last_active":"2024-05-27T00:15:26.586830+0000","last_peered":"2024-05-27T00:15:26.586830+0000","last_clean":"2024-05-27T00:15:26.586830+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:15:26.586830+0000","last_undegraded":"2024-05-27T00:15:26.586830+0000","last_fullsized":"2024-05-27T00:15:26.586830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:02:15.012827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745308+0000","last_change":"2024-05-27T00:15:25.610707+0000","last_active":"2024-05-27T00:15:26.745308+0000","last_peered":"2024-05-27T00:15:26.745308+0000","last_clean":"2024-05-27T00:15:26.745308+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:15:26.745308+0000","last_undegraded":"2024-05-27T00:15:26.745308+0000","last_fullsized":"2024-05-27T00:15:26.745308+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:09:16.698319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587101+0000","last_change":"2024-05-27T00:15:25.613013+0000","last_active":"2024-05-27T00:15:26.587101+0000","last_peered":"2024-05-27T00:15:26.587101+0000","last_clean":"2024-05-27T00:15:26.587101+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:15:26.587101+0000","last_undegraded":"2024-05-27T00:15:26.587101+0000","last_fullsized":"2024-05-27T00:15:26.587101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745910+0000","last_change":"2024-05-27T00:15:25.610070+0000","last_active":"2024-05-27T00:15:26.745910+0000","last_peered":"2024-05-27T00:15:26.745910+0000","last_clean":"2024-05-27T00:15:26.745910+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:15:26.745910+0000","last_undegraded":"2024-05-27T00:15:26.745910+0000","last_fullsized":"2024-05-27T00:15:26.745910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:02:17.514654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747817+0000","last_change":"2024-05-27T00:15:25.633470+0000","last_active":"2024-05-27T00:15:26.747817+0000","last_peered":"2024-05-27T00:15:26.747817+0000","last_clean":"2024-05-27T00:15:26.747817+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:15:26.747817+0000","last_undegraded":"2024-05-27T00:15:26.747817+0000","last_fullsized":"2024-05-27T00:15:26.747817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583597+0000","last_change":"2024-05-27T00:15:25.597615+0000","last_active":"2024-05-27T00:15:26.583597+0000","last_peered":"2024-05-27T00:15:26.583597+0000","last_clean":"2024-05-27T00:15:26.583597+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:15:26.583597+0000","last_undegraded":"2024-05-27T00:15:26.583597+0000","last_fullsized":"2024-05-27T00:15:26.583597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746038+0000","last_change":"2024-05-27T00:15:25.610933+0000","last_active":"2024-05-27T00:15:26.746038+0000","last_peered":"2024-05-27T00:15:26.746038+0000","last_clean":"2024-05-27T00:15:26.746038+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:15:26.746038+0000","last_undegraded":"2024-05-27T00:15:26.746038+0000","last_fullsized":"2024-05-27T00:15:26.746038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:45:25.888432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746185+0000","last_change":"2024-05-27T00:15:25.598227+0000","last_active":"2024-05-27T00:15:26.746185+0000","last_peered":"2024-05-27T00:15:26.746185+0000","last_clean":"2024-05-27T00:15:26.746185+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:15:26.746185+0000","last_undegraded":"2024-05-27T00:15:26.746185+0000","last_fullsized":"2024-05-27T00:15:26.746185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588189+0000","last_change":"2024-05-27T00:15:25.620621+0000","last_active":"2024-05-27T00:15:26.588189+0000","last_peered":"2024-05-27T00:15:26.588189+0000","last_clean":"2024-05-27T00:15:26.588189+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:15:26.588189+0000","last_undegraded":"2024-05-27T00:15:26.588189+0000","last_fullsized":"2024-05-27T00:15:26.588189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:22:30.152810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744910+0000","last_change":"2024-05-27T00:15:25.605839+0000","last_active":"2024-05-27T00:15:26.744910+0000","last_peered":"2024-05-27T00:15:26.744910+0000","last_clean":"2024-05-27T00:15:26.744910+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:15:26.744910+0000","last_undegraded":"2024-05-27T00:15:26.744910+0000","last_fullsized":"2024-05-27T00:15:26.744910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746666+0000","last_change":"2024-05-27T00:15:25.607188+0000","last_active":"2024-05-27T00:15:26.746666+0000","last_peered":"2024-05-27T00:15:26.746666+0000","last_clean":"2024-05-27T00:15:26.746666+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:15:26.746666+0000","last_undegraded":"2024-05-27T00:15:26.746666+0000","last_fullsized":"2024-05-27T00:15:26.746666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587251+0000","last_change":"2024-05-27T00:15:25.615579+0000","last_active":"2024-05-27T00:15:26.587251+0000","last_peered":"2024-05-27T00:15:26.587251+0000","last_clean":"2024-05-27T00:15:26.587251+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:15:26.587251+0000","last_undegraded":"2024-05-27T00:15:26.587251+0000","last_fullsized":"2024-05-27T00:15:26.587251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:53.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746392+0000","last_change":"2024-05-27T00:15:24.611108+0000","last_active":"2024-05-27T00:15:26.746392+0000","last_peered":"2024-05-27T00:15:26.746392+0000","last_clean":"2024-05-27T00:15:26.746392+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:15:26.746392+0000","last_undegraded":"2024-05-27T00:15:26.746392+0000","last_fullsized":"2024-05-27T00:15:26.746392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:11.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746332+0000","last_change":"2024-05-27T00:15:25.614251+0000","last_active":"2024-05-27T00:15:26.746332+0000","last_peered":"2024-05-27T00:15:26.746332+0000","last_clean":"2024-05-27T00:15:26.746332+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:15:26.746332+0000","last_undegraded":"2024-05-27T00:15:26.746332+0000","last_fullsized":"2024-05-27T00:15:26.746332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747555+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:15:26.747555+0000","last_peered":"2024-05-27T00:15:26.747555+0000","last_clean":"2024-05-27T00:15:26.747555+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:15:26.747555+0000","last_undegraded":"2024-05-27T00:15:26.747555+0000","last_fullsized":"2024-05-27T00:15:26.747555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745607+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:15:26.745607+0000","last_peered":"2024-05-27T00:15:26.745607+0000","last_clean":"2024-05-27T00:15:26.745607+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:15:26.745607+0000","last_undegraded":"2024-05-27T00:15:26.745607+0000","last_fullsized":"2024-05-27T00:15:26.745607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:46:08.645927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745132+0000","last_change":"2024-05-27T00:15:25.596844+0000","last_active":"2024-05-27T00:15:26.745132+0000","last_peered":"2024-05-27T00:15:26.745132+0000","last_clean":"2024-05-27T00:15:26.745132+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:15:26.745132+0000","last_undegraded":"2024-05-27T00:15:26.745132+0000","last_fullsized":"2024-05-27T00:15:26.745132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:17:40.503615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587344+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:15:26.587344+0000","last_peered":"2024-05-27T00:15:26.587344+0000","last_clean":"2024-05-27T00:15:26.587344+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:15:26.587344+0000","last_undegraded":"2024-05-27T00:15:26.587344+0000","last_fullsized":"2024-05-27T00:15:26.587344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583235+0000","last_change":"2024-05-27T00:15:25.593149+0000","last_active":"2024-05-27T00:15:26.583235+0000","last_peered":"2024-05-27T00:15:26.583235+0000","last_clean":"2024-05-27T00:15:26.583235+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:15:26.583235+0000","last_undegraded":"2024-05-27T00:15:26.583235+0000","last_fullsized":"2024-05-27T00:15:26.583235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746974+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:15:26.746974+0000","last_peered":"2024-05-27T00:15:26.746974+0000","last_clean":"2024-05-27T00:15:26.746974+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:15:26.746974+0000","last_undegraded":"2024-05-27T00:15:26.746974+0000","last_fullsized":"2024-05-27T00:15:26.746974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589140+0000","last_change":"2024-05-27T00:15:25.618650+0000","last_active":"2024-05-27T00:15:26.589140+0000","last_peered":"2024-05-27T00:15:26.589140+0000","last_clean":"2024-05-27T00:15:26.589140+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:15:26.589140+0000","last_undegraded":"2024-05-27T00:15:26.589140+0000","last_fullsized":"2024-05-27T00:15:26.589140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:21.935526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589356+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:15:26.589356+0000","last_peered":"2024-05-27T00:15:26.589356+0000","last_clean":"2024-05-27T00:15:26.589356+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:15:26.589356+0000","last_undegraded":"2024-05-27T00:15:26.589356+0000","last_fullsized":"2024-05-27T00:15:26.589356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746448+0000","last_change":"2024-05-27T00:15:25.613142+0000","last_active":"2024-05-27T00:15:26.746448+0000","last_peered":"2024-05-27T00:15:26.746448+0000","last_clean":"2024-05-27T00:15:26.746448+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:15:26.746448+0000","last_undegraded":"2024-05-27T00:15:26.746448+0000","last_fullsized":"2024-05-27T00:15:26.746448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.620068+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:15:30.620068+0000","last_peered":"2024-05-27T00:15:30.620068+0000","last_clean":"2024-05-27T00:15:30.620068+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:15:30.620068+0000","last_undegraded":"2024-05-27T00:15:30.620068+0000","last_fullsized":"2024-05-27T00:15:30.620068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588245+0000","last_change":"2024-05-27T00:15:25.622383+0000","last_active":"2024-05-27T00:15:26.588245+0000","last_peered":"2024-05-27T00:15:26.588245+0000","last_clean":"2024-05-27T00:15:26.588245+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:15:26.588245+0000","last_undegraded":"2024-05-27T00:15:26.588245+0000","last_fullsized":"2024-05-27T00:15:26.588245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.624481+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:15:30.624481+0000","last_peered":"2024-05-27T00:15:30.624481+0000","last_clean":"2024-05-27T00:15:30.624481+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:15:30.624481+0000","last_undegraded":"2024-05-27T00:15:30.624481+0000","last_fullsized":"2024-05-27T00:15:30.624481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748511+0000","last_change":"2024-05-27T00:15:25.628646+0000","last_active":"2024-05-27T00:15:26.748511+0000","last_peered":"2024-05-27T00:15:26.748511+0000","last_clean":"2024-05-27T00:15:26.748511+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:15:26.748511+0000","last_undegraded":"2024-05-27T00:15:26.748511+0000","last_fullsized":"2024-05-27T00:15:26.748511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:22:30.152810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619790+0000","last_change":"2024-05-27T00:15:24.605986+0000","last_active":"2024-05-27T00:15:30.619790+0000","last_peered":"2024-05-27T00:15:30.619790+0000","last_clean":"2024-05-27T00:15:30.619790+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:15:30.619790+0000","last_undegraded":"2024-05-27T00:15:30.619790+0000","last_fullsized":"2024-05-27T00:15:30.619790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745414+0000","last_change":"2024-05-27T00:15:25.624136+0000","last_active":"2024-05-27T00:15:26.745414+0000","last_peered":"2024-05-27T00:15:26.745414+0000","last_clean":"2024-05-27T00:15:26.745414+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:15:26.745414+0000","last_undegraded":"2024-05-27T00:15:26.745414+0000","last_fullsized":"2024-05-27T00:15:26.745414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588435+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:15:26.588435+0000","last_peered":"2024-05-27T00:15:26.588435+0000","last_clean":"2024-05-27T00:15:26.588435+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:15:26.588435+0000","last_undegraded":"2024-05-27T00:15:26.588435+0000","last_fullsized":"2024-05-27T00:15:26.588435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748720+0000","last_change":"2024-05-27T00:15:25.641965+0000","last_active":"2024-05-27T00:15:26.748720+0000","last_peered":"2024-05-27T00:15:26.748720+0000","last_clean":"2024-05-27T00:15:26.748720+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:15:26.748720+0000","last_undegraded":"2024-05-27T00:15:26.748720+0000","last_fullsized":"2024-05-27T00:15:26.748720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:17.513529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746779+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:15:26.746779+0000","last_peered":"2024-05-27T00:15:26.746779+0000","last_clean":"2024-05-27T00:15:26.746779+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:15:26.746779+0000","last_undegraded":"2024-05-27T00:15:26.746779+0000","last_fullsized":"2024-05-27T00:15:26.746779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746732+0000","last_change":"2024-05-27T00:15:25.608549+0000","last_active":"2024-05-27T00:15:26.746732+0000","last_peered":"2024-05-27T00:15:26.746732+0000","last_clean":"2024-05-27T00:15:26.746732+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:15:26.746732+0000","last_undegraded":"2024-05-27T00:15:26.746732+0000","last_fullsized":"2024-05-27T00:15:26.746732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619615+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:15:30.619615+0000","last_peered":"2024-05-27T00:15:30.619615+0000","last_clean":"2024-05-27T00:15:30.619615+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:15:30.619615+0000","last_undegraded":"2024-05-27T00:15:30.619615+0000","last_fullsized":"2024-05-27T00:15:30.619615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747206+0000","last_change":"2024-05-27T00:15:25.638558+0000","last_active":"2024-05-27T00:15:26.747206+0000","last_peered":"2024-05-27T00:15:26.747206+0000","last_clean":"2024-05-27T00:15:26.747206+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:15:26.747206+0000","last_undegraded":"2024-05-27T00:15:26.747206+0000","last_fullsized":"2024-05-27T00:15:26.747206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:02:15.012827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587664+0000","last_change":"2024-05-27T00:15:24.601924+0000","last_active":"2024-05-27T00:15:26.587664+0000","last_peered":"2024-05-27T00:15:26.587664+0000","last_clean":"2024-05-27T00:15:26.587664+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:15:26.587664+0000","last_undegraded":"2024-05-27T00:15:26.587664+0000","last_fullsized":"2024-05-27T00:15:26.587664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:32.185839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749041+0000","last_change":"2024-05-27T00:15:25.639140+0000","last_active":"2024-05-27T00:15:26.749041+0000","last_peered":"2024-05-27T00:15:26.749041+0000","last_clean":"2024-05-27T00:15:26.749041+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:15:26.749041+0000","last_undegraded":"2024-05-27T00:15:26.749041+0000","last_fullsized":"2024-05-27T00:15:26.749041+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:21.935526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619020+0000","last_change":"2024-05-27T00:15:24.600878+0000","last_active":"2024-05-27T00:15:30.619020+0000","last_peered":"2024-05-27T00:15:30.619020+0000","last_clean":"2024-05-27T00:15:30.619020+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:15:30.619020+0000","last_undegraded":"2024-05-27T00:15:30.619020+0000","last_fullsized":"2024-05-27T00:15:30.619020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749435+0000","last_change":"2024-05-27T00:15:25.644249+0000","last_active":"2024-05-27T00:15:26.749435+0000","last_peered":"2024-05-27T00:15:26.749435+0000","last_clean":"2024-05-27T00:15:26.749435+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:15:26.749435+0000","last_undegraded":"2024-05-27T00:15:26.749435+0000","last_fullsized":"2024-05-27T00:15:26.749435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:59:25.042662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583715+0000","last_change":"2024-05-27T00:15:24.603452+0000","last_active":"2024-05-27T00:15:26.583715+0000","last_peered":"2024-05-27T00:15:26.583715+0000","last_clean":"2024-05-27T00:15:26.583715+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:15:26.583715+0000","last_undegraded":"2024-05-27T00:15:26.583715+0000","last_fullsized":"2024-05-27T00:15:26.583715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587865+0000","last_change":"2024-05-27T00:15:25.619341+0000","last_active":"2024-05-27T00:15:26.587865+0000","last_peered":"2024-05-27T00:15:26.587865+0000","last_clean":"2024-05-27T00:15:26.587865+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:15:26.587865+0000","last_undegraded":"2024-05-27T00:15:26.587865+0000","last_fullsized":"2024-05-27T00:15:26.587865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:41:46.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745117+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:15:26.745117+0000","last_peered":"2024-05-27T00:15:26.745117+0000","last_clean":"2024-05-27T00:15:26.745117+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:15:26.745117+0000","last_undegraded":"2024-05-27T00:15:26.745117+0000","last_fullsized":"2024-05-27T00:15:26.745117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748893+0000","last_change":"2024-05-27T00:15:25.639462+0000","last_active":"2024-05-27T00:15:26.748893+0000","last_peered":"2024-05-27T00:15:26.748893+0000","last_clean":"2024-05-27T00:15:26.748893+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:15:26.748893+0000","last_undegraded":"2024-05-27T00:15:26.748893+0000","last_fullsized":"2024-05-27T00:15:26.748893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:52:55.175945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746823+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:15:26.746823+0000","last_peered":"2024-05-27T00:15:26.746823+0000","last_clean":"2024-05-27T00:15:26.746823+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:15:26.746823+0000","last_undegraded":"2024-05-27T00:15:26.746823+0000","last_fullsized":"2024-05-27T00:15:26.746823+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586683+0000","last_change":"2024-05-27T00:15:24.603781+0000","last_active":"2024-05-27T00:15:26.586683+0000","last_peered":"2024-05-27T00:15:26.586683+0000","last_clean":"2024-05-27T00:15:26.586683+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:15:26.586683+0000","last_undegraded":"2024-05-27T00:15:26.586683+0000","last_fullsized":"2024-05-27T00:15:26.586683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746108+0000","last_change":"2024-05-27T00:15:25.631978+0000","last_active":"2024-05-27T00:15:26.746108+0000","last_peered":"2024-05-27T00:15:26.746108+0000","last_clean":"2024-05-27T00:15:26.746108+0000","last_became_active":"2024-05-27T00:15:25.631697+0000","last_became_peered":"2024-05-27T00:15:25.631697+0000","last_unstale":"2024-05-27T00:15:26.746108+0000","last_undegraded":"2024-05-27T00:15:26.746108+0000","last_fullsized":"2024-05-27T00:15:26.746108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:05.141408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746536+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:15:26.746536+0000","last_peered":"2024-05-27T00:15:26.746536+0000","last_clean":"2024-05-27T00:15:26.746536+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:15:26.746536+0000","last_undegraded":"2024-05-27T00:15:26.746536+0000","last_fullsized":"2024-05-27T00:15:26.746536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746253+0000","last_change":"2024-05-27T00:15:25.610627+0000","last_active":"2024-05-27T00:15:26.746253+0000","last_peered":"2024-05-27T00:15:26.746253+0000","last_clean":"2024-05-27T00:15:26.746253+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:15:26.746253+0000","last_undegraded":"2024-05-27T00:15:26.746253+0000","last_fullsized":"2024-05-27T00:15:26.746253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744898+0000","last_change":"2024-05-27T00:15:24.598182+0000","last_active":"2024-05-27T00:15:26.744898+0000","last_peered":"2024-05-27T00:15:26.744898+0000","last_clean":"2024-05-27T00:15:26.744898+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:15:26.744898+0000","last_undegraded":"2024-05-27T00:15:26.744898+0000","last_fullsized":"2024-05-27T00:15:26.744898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583243+0000","last_change":"2024-05-27T00:15:25.591449+0000","last_active":"2024-05-27T00:15:26.583243+0000","last_peered":"2024-05-27T00:15:26.583243+0000","last_clean":"2024-05-27T00:15:26.583243+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:15:26.583243+0000","last_undegraded":"2024-05-27T00:15:26.583243+0000","last_fullsized":"2024-05-27T00:15:26.583243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588693+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:15:26.588693+0000","last_peered":"2024-05-27T00:15:26.588693+0000","last_clean":"2024-05-27T00:15:26.588693+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:15:26.588693+0000","last_undegraded":"2024-05-27T00:15:26.588693+0000","last_fullsized":"2024-05-27T00:15:26.588693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587675+0000","last_change":"2024-05-27T00:15:25.614073+0000","last_active":"2024-05-27T00:15:26.587675+0000","last_peered":"2024-05-27T00:15:26.587675+0000","last_clean":"2024-05-27T00:15:26.587675+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:15:26.587675+0000","last_undegraded":"2024-05-27T00:15:26.587675+0000","last_fullsized":"2024-05-27T00:15:26.587675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583569+0000","last_change":"2024-05-27T00:15:24.603518+0000","last_active":"2024-05-27T00:15:26.583569+0000","last_peered":"2024-05-27T00:15:26.583569+0000","last_clean":"2024-05-27T00:15:26.583569+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:15:26.583569+0000","last_undegraded":"2024-05-27T00:15:26.583569+0000","last_fullsized":"2024-05-27T00:15:26.583569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588810+0000","last_change":"2024-05-27T00:15:25.621519+0000","last_active":"2024-05-27T00:15:26.588810+0000","last_peered":"2024-05-27T00:15:26.588810+0000","last_clean":"2024-05-27T00:15:26.588810+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:15:26.588810+0000","last_undegraded":"2024-05-27T00:15:26.588810+0000","last_fullsized":"2024-05-27T00:15:26.588810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:52:55.175945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749299+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:15:26.749299+0000","last_peered":"2024-05-27T00:15:26.749299+0000","last_clean":"2024-05-27T00:15:26.749299+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:15:26.749299+0000","last_undegraded":"2024-05-27T00:15:26.749299+0000","last_fullsized":"2024-05-27T00:15:26.749299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583460+0000","last_change":"2024-05-27T00:15:25.593180+0000","last_active":"2024-05-27T00:15:26.583460+0000","last_peered":"2024-05-27T00:15:26.583460+0000","last_clean":"2024-05-27T00:15:26.583460+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:15:26.583460+0000","last_undegraded":"2024-05-27T00:15:26.583460+0000","last_fullsized":"2024-05-27T00:15:26.583460+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583660+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:15:26.583660+0000","last_peered":"2024-05-27T00:15:26.583660+0000","last_clean":"2024-05-27T00:15:26.583660+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:15:26.583660+0000","last_undegraded":"2024-05-27T00:15:26.583660+0000","last_fullsized":"2024-05-27T00:15:26.583660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745915+0000","last_change":"2024-05-27T00:15:25.631919+0000","last_active":"2024-05-27T00:15:26.745915+0000","last_peered":"2024-05-27T00:15:26.745915+0000","last_clean":"2024-05-27T00:15:26.745915+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:15:26.745915+0000","last_undegraded":"2024-05-27T00:15:26.745915+0000","last_fullsized":"2024-05-27T00:15:26.745915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:24:31.995603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.620892+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:15:30.620892+0000","last_peered":"2024-05-27T00:15:30.620892+0000","last_clean":"2024-05-27T00:15:30.620892+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:15:30.620892+0000","last_undegraded":"2024-05-27T00:15:30.620892+0000","last_fullsized":"2024-05-27T00:15:30.620892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745629+0000","last_change":"2024-05-27T00:15:25.596792+0000","last_active":"2024-05-27T00:15:26.745629+0000","last_peered":"2024-05-27T00:15:26.745629+0000","last_clean":"2024-05-27T00:15:26.745629+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:15:26.745629+0000","last_undegraded":"2024-05-27T00:15:26.745629+0000","last_fullsized":"2024-05-27T00:15:26.745629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747720+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:15:26.747720+0000","last_peered":"2024-05-27T00:15:26.747720+0000","last_clean":"2024-05-27T00:15:26.747720+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:15:26.747720+0000","last_undegraded":"2024-05-27T00:15:26.747720+0000","last_fullsized":"2024-05-27T00:15:26.747720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587779+0000","last_change":"2024-05-27T00:15:25.610697+0000","last_active":"2024-05-27T00:15:26.587779+0000","last_peered":"2024-05-27T00:15:26.587779+0000","last_clean":"2024-05-27T00:15:26.587779+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:15:26.587779+0000","last_undegraded":"2024-05-27T00:15:26.587779+0000","last_fullsized":"2024-05-27T00:15:26.587779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:14.969827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583375+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:15:26.583375+0000","last_peered":"2024-05-27T00:15:26.583375+0000","last_clean":"2024-05-27T00:15:26.583375+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:15:26.583375+0000","last_undegraded":"2024-05-27T00:15:26.583375+0000","last_fullsized":"2024-05-27T00:15:26.583375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745282+0000","last_change":"2024-05-27T00:15:25.597164+0000","last_active":"2024-05-27T00:15:26.745282+0000","last_peered":"2024-05-27T00:15:26.745282+0000","last_clean":"2024-05-27T00:15:26.745282+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:15:26.745282+0000","last_undegraded":"2024-05-27T00:15:26.745282+0000","last_fullsized":"2024-05-27T00:15:26.745282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:04.655872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583422+0000","last_change":"2024-05-27T00:15:24.598238+0000","last_active":"2024-05-27T00:15:26.583422+0000","last_peered":"2024-05-27T00:15:26.583422+0000","last_clean":"2024-05-27T00:15:26.583422+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:15:26.583422+0000","last_undegraded":"2024-05-27T00:15:26.583422+0000","last_fullsized":"2024-05-27T00:15:26.583422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746235+0000","last_change":"2024-05-27T00:15:25.601140+0000","last_active":"2024-05-27T00:15:26.746235+0000","last_peered":"2024-05-27T00:15:26.746235+0000","last_clean":"2024-05-27T00:15:26.746235+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:15:26.746235+0000","last_undegraded":"2024-05-27T00:15:26.746235+0000","last_fullsized":"2024-05-27T00:15:26.746235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:42:05.334730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.621275+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:15:30.621275+0000","last_peered":"2024-05-27T00:15:30.621275+0000","last_clean":"2024-05-27T00:15:30.621275+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:15:30.621275+0000","last_undegraded":"2024-05-27T00:15:30.621275+0000","last_fullsized":"2024-05-27T00:15:30.621275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747292+0000","last_change":"2024-05-27T00:15:25.619694+0000","last_active":"2024-05-27T00:15:26.747292+0000","last_peered":"2024-05-27T00:15:26.747292+0000","last_clean":"2024-05-27T00:15:26.747292+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:15:26.747292+0000","last_undegraded":"2024-05-27T00:15:26.747292+0000","last_fullsized":"2024-05-27T00:15:26.747292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588751+0000","last_change":"2024-05-27T00:15:25.627578+0000","last_active":"2024-05-27T00:15:26.588751+0000","last_peered":"2024-05-27T00:15:26.588751+0000","last_clean":"2024-05-27T00:15:26.588751+0000","last_became_active":"2024-05-27T00:15:25.622421+0000","last_became_peered":"2024-05-27T00:15:25.622421+0000","last_unstale":"2024-05-27T00:15:26.588751+0000","last_undegraded":"2024-05-27T00:15:26.588751+0000","last_fullsized":"2024-05-27T00:15:26.588751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:17.513529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619639+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:15:30.619639+0000","last_peered":"2024-05-27T00:15:30.619639+0000","last_clean":"2024-05-27T00:15:30.619639+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:15:30.619639+0000","last_undegraded":"2024-05-27T00:15:30.619639+0000","last_fullsized":"2024-05-27T00:15:30.619639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587486+0000","last_change":"2024-05-27T00:15:25.623207+0000","last_active":"2024-05-27T00:15:26.587486+0000","last_peered":"2024-05-27T00:15:26.587486+0000","last_clean":"2024-05-27T00:15:26.587486+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:15:26.587486+0000","last_undegraded":"2024-05-27T00:15:26.587486+0000","last_fullsized":"2024-05-27T00:15:26.587486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:59.804368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.618376+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:15:30.618376+0000","last_peered":"2024-05-27T00:15:30.618376+0000","last_clean":"2024-05-27T00:15:30.618376+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:15:30.618376+0000","last_undegraded":"2024-05-27T00:15:30.618376+0000","last_fullsized":"2024-05-27T00:15:30.618376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746305+0000","last_change":"2024-05-27T00:15:25.613018+0000","last_active":"2024-05-27T00:15:26.746305+0000","last_peered":"2024-05-27T00:15:26.746305+0000","last_clean":"2024-05-27T00:15:26.746305+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:15:26.746305+0000","last_undegraded":"2024-05-27T00:15:26.746305+0000","last_fullsized":"2024-05-27T00:15:26.746305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745698+0000","last_change":"2024-05-27T00:15:24.600571+0000","last_active":"2024-05-27T00:15:26.745698+0000","last_peered":"2024-05-27T00:15:26.745698+0000","last_clean":"2024-05-27T00:15:26.745698+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:15:26.745698+0000","last_undegraded":"2024-05-27T00:15:26.745698+0000","last_fullsized":"2024-05-27T00:15:26.745698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:44:21.645223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745724+0000","last_change":"2024-05-27T00:15:25.615649+0000","last_active":"2024-05-27T00:15:26.745724+0000","last_peered":"2024-05-27T00:15:26.745724+0000","last_clean":"2024-05-27T00:15:26.745724+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:15:26.745724+0000","last_undegraded":"2024-05-27T00:15:26.745724+0000","last_fullsized":"2024-05-27T00:15:26.745724+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:16:16.309939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.621025+0000","last_change":"2024-05-27T00:15:24.615066+0000","last_active":"2024-05-27T00:15:30.621025+0000","last_peered":"2024-05-27T00:15:30.621025+0000","last_clean":"2024-05-27T00:15:30.621025+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:15:30.621025+0000","last_undegraded":"2024-05-27T00:15:30.621025+0000","last_fullsized":"2024-05-27T00:15:30.621025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":669858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":79034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":78908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":79068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":669914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:15:35.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:15:35.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f2305000640 1 Processor -- start 2024-05-27T00:15:35.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f2305000640 1 -- start start 2024-05-27T00:15:35.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f2305000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f23000cdf90 0x7f230006f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f2305000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23000c4fd0 0x7f23000ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f2305000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23000c8fd0 0x7f23000c9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f2305000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f230006d500 con 0x7f23000c8fd0 2024-05-27T00:15:35.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f2305000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f230006ef50 con 0x7f23000cdf90 2024-05-27T00:15:35.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f2305000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f230006cfb0 con 0x7f23000c4fd0 2024-05-27T00:15:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f22fe800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23000c8fd0 0x7f23000c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f22fe800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f23000c8fd0 0x7f23000c9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51148/0 (socket says 172.21.15.98:51148) 2024-05-27T00:15:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f22fe800640 1 -- 172.21.15.98:0/1146767199 learned_addr learned my addr 172.21.15.98:0/1146767199 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f22fde00640 1 --2- 172.21.15.98:0/1146767199 >> v2:172.21.15.204:3300/0 conn(0x7f23000cdf90 0x7f230006f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f22f7e00640 1 --2- 172.21.15.98:0/1146767199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23000c4fd0 0x7f23000ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f22fe800640 1 -- 172.21.15.98:0/1146767199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23000c4fd0 msgr2=0x7f23000ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f22fe800640 1 --2- 172.21.15.98:0/1146767199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23000c4fd0 0x7f23000ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f22fe800640 1 -- 172.21.15.98:0/1146767199 >> v2:172.21.15.204:3300/0 conn(0x7f23000cdf90 msgr2=0x7f230006f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f22fe800640 1 --2- 172.21.15.98:0/1146767199 >> v2:172.21.15.204:3300/0 conn(0x7f23000cdf90 0x7f230006f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.497+0000 7f22fe800640 1 -- 172.21.15.98:0/1146767199 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23000ca840 con 0x7f23000c8fd0 2024-05-27T00:15:35.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f22fe800640 1 --2- 172.21.15.98:0/1146767199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c8fd0 0x7f23000c9430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f23000cbb70 tx=0x7f22f801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18c7e85575df877 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:35.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f22fd400640 1 -- 172.21.15.98:0/1146767199 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80b4070 con 0x7f23000c8fd0 2024-05-27T00:15:35.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f22fd400640 1 -- 172.21.15.98:0/1146767199 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f80af400 con 0x7f23000c8fd0 2024-05-27T00:15:35.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f22fd400640 1 -- 172.21.15.98:0/1146767199 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80af7d0 con 0x7f23000c8fd0 2024-05-27T00:15:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f2305000640 1 -- 172.21.15.98:0/1146767199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c8fd0 msgr2=0x7f23000c9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f2305000640 1 --2- 172.21.15.98:0/1146767199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c8fd0 0x7f23000c9430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f23000cbb70 tx=0x7f22f801f1e0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f2305000640 1 -- 172.21.15.98:0/1146767199 shutdown_connections 2024-05-27T00:15:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f2305000640 1 --2- 172.21.15.98:0/1146767199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c8fd0 0x7f23000c9430 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f2305000640 1 --2- 172.21.15.98:0/1146767199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23000c4fd0 0x7f23000ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f2305000640 1 --2- 172.21.15.98:0/1146767199 >> v2:172.21.15.204:3300/0 conn(0x7f23000cdf90 0x7f230006f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f2305000640 1 -- 172.21.15.98:0/1146767199 >> 172.21.15.98:0/1146767199 conn(0x7f230006d680 msgr2=0x7f230006da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f2305000640 1 -- 172.21.15.98:0/1146767199 shutdown_connections 2024-05-27T00:15:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.501+0000 7f2305000640 1 -- 172.21.15.98:0/1146767199 wait complete. 2024-05-27T00:15:35.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f2305000640 1 Processor -- start 2024-05-27T00:15:35.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f2305000640 1 -- start start 2024-05-27T00:15:35.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f2305000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c4fd0 0x7f23000cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f2305000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23000c8fd0 0x7f2300156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f2305000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23000cdf90 0x7f2300148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f2305000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f230006d220 con 0x7f23000c4fd0 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f2305000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f230006b610 con 0x7f23000cdf90 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f2305000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f230006b8b0 con 0x7f23000c8fd0 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fde00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c4fd0 0x7f23000cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fde00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c4fd0 0x7f23000cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51158/0 (socket says 172.21.15.98:51158) 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fde00640 1 -- 172.21.15.98:0/2897940825 learned_addr learned my addr 172.21.15.98:0/2897940825 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22f7e00640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23000c8fd0 0x7f2300156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fe800640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23000cdf90 0x7f2300148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fde00640 1 -- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23000c8fd0 msgr2=0x7f2300156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fde00640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23000c8fd0 0x7f2300156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fde00640 1 -- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23000cdf90 msgr2=0x7f2300148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fde00640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23000cdf90 0x7f2300148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fde00640 1 -- 172.21.15.98:0/2897940825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23001570a0 con 0x7f23000c4fd0 2024-05-27T00:15:35.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fe800640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23000cdf90 0x7f2300148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:35.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22fde00640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c4fd0 0x7f23000cbe00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f22f000d5f0 tx=0x7f22f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22f6000640 1 -- 172.21.15.98:0/2897940825 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f00125f0 con 0x7f23000c4fd0 2024-05-27T00:15:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22f6000640 1 -- 172.21.15.98:0/2897940825 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f0019070 con 0x7f23000c4fd0 2024-05-27T00:15:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f22f6000640 1 -- 172.21.15.98:0/2897940825 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f00129b0 con 0x7f23000c4fd0 2024-05-27T00:15:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f2305000640 1 -- 172.21.15.98:0/2897940825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2300157770 con 0x7f23000c4fd0 2024-05-27T00:15:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.505+0000 7f2305000640 1 -- 172.21.15.98:0/2897940825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2300151380 con 0x7f23000c4fd0 2024-05-27T00:15:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.509+0000 7f2305000640 1 -- 172.21.15.98:0/2897940825 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2300075cb0 con 0x7f23000c4fd0 2024-05-27T00:15:35.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.509+0000 7f22f6000640 1 -- 172.21.15.98:0/2897940825 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f22f0012b50 con 0x7f23000c4fd0 2024-05-27T00:15:35.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.509+0000 7f22f6000640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f22dc0678c0 0x7f22dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.509+0000 7f22f6000640 1 -- 172.21.15.98:0/2897940825 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f22f00861c0 con 0x7f23000c4fd0 2024-05-27T00:15:35.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.509+0000 7f22f7e00640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f22dc0678c0 0x7f22dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.513+0000 7f22f7e00640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f22dc0678c0 0x7f22dc069d80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f230006fb40 tx=0x7f22e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:35.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.513+0000 7f22f6000640 1 -- 172.21.15.98:0/2897940825 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f22f0056bd0 con 0x7f23000c4fd0 2024-05-27T00:15:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.669+0000 7f2305000640 1 -- 172.21.15.98:0/2897940825 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2300153000 con 0x7f22dc0678c0 2024-05-27T00:15:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.685+0000 7f22f6000640 1 -- 172.21.15.98:0/2897940825 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251835 (secure 0 0 0) 0x7f2300153000 con 0x7f22dc0678c0 2024-05-27T00:15:35.694 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:35.696 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:15:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 -- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f22dc0678c0 msgr2=0x7f22dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f22dc0678c0 0x7f22dc069d80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f230006fb40 tx=0x7f22e8008040 comp rx=0 tx=0).stop 2024-05-27T00:15:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 -- 172.21.15.98:0/2897940825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c4fd0 msgr2=0x7f23000cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c4fd0 0x7f23000cbe00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f22f000d5f0 tx=0x7f22f000dab0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 -- 172.21.15.98:0/2897940825 shutdown_connections 2024-05-27T00:15:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f22dc0678c0 0x7f22dc069d80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23000cdf90 0x7f2300148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23000c8fd0 0x7f2300156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 --2- 172.21.15.98:0/2897940825 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23000c4fd0 0x7f23000cbe00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 -- 172.21.15.98:0/2897940825 >> 172.21.15.98:0/2897940825 conn(0x7f230006d680 msgr2=0x7f23000c3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 -- 172.21.15.98:0/2897940825 shutdown_connections 2024-05-27T00:15:35.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.689+0000 7f2305000640 1 -- 172.21.15.98:0/2897940825 wait complete. 2024-05-27T00:15:35.714 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-05-27T00:15:35.136220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219112,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755608,"statfs":{"total":773094113280,"available":772869742592,"internally_reserved":0,"allocated":3055616,"data_stored":1816820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.556163"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748978+0000","last_change":"2024-05-27T00:15:25.640698+0000","last_active":"2024-05-27T00:15:26.748978+0000","last_peered":"2024-05-27T00:15:26.748978+0000","last_clean":"2024-05-27T00:15:26.748978+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:15:26.748978+0000","last_undegraded":"2024-05-27T00:15:26.748978+0000","last_fullsized":"2024-05-27T00:15:26.748978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:47:27.030637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583615+0000","last_change":"2024-05-27T00:15:25.598672+0000","last_active":"2024-05-27T00:15:26.583615+0000","last_peered":"2024-05-27T00:15:26.583615+0000","last_clean":"2024-05-27T00:15:26.583615+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:15:26.583615+0000","last_undegraded":"2024-05-27T00:15:26.583615+0000","last_fullsized":"2024-05-27T00:15:26.583615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745467+0000","last_change":"2024-05-27T00:15:25.593316+0000","last_active":"2024-05-27T00:15:26.745467+0000","last_peered":"2024-05-27T00:15:26.745467+0000","last_clean":"2024-05-27T00:15:26.745467+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:15:26.745467+0000","last_undegraded":"2024-05-27T00:15:26.745467+0000","last_fullsized":"2024-05-27T00:15:26.745467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587216+0000","last_change":"2024-05-27T00:15:25.620237+0000","last_active":"2024-05-27T00:15:26.587216+0000","last_peered":"2024-05-27T00:15:26.587216+0000","last_clean":"2024-05-27T00:15:26.587216+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:15:26.587216+0000","last_undegraded":"2024-05-27T00:15:26.587216+0000","last_fullsized":"2024-05-27T00:15:26.587216+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:17:53.053461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748475+0000","last_change":"2024-05-27T00:15:25.640146+0000","last_active":"2024-05-27T00:15:26.748475+0000","last_peered":"2024-05-27T00:15:26.748475+0000","last_clean":"2024-05-27T00:15:26.748475+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:15:26.748475+0000","last_undegraded":"2024-05-27T00:15:26.748475+0000","last_fullsized":"2024-05-27T00:15:26.748475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:41:46.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589129+0000","last_change":"2024-05-27T00:15:25.627128+0000","last_active":"2024-05-27T00:15:26.589129+0000","last_peered":"2024-05-27T00:15:26.589129+0000","last_clean":"2024-05-27T00:15:26.589129+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:15:26.589129+0000","last_undegraded":"2024-05-27T00:15:26.589129+0000","last_fullsized":"2024-05-27T00:15:26.589129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:47:27.030637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746743+0000","last_change":"2024-05-27T00:15:25.630446+0000","last_active":"2024-05-27T00:15:26.746743+0000","last_peered":"2024-05-27T00:15:26.746743+0000","last_clean":"2024-05-27T00:15:26.746743+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:15:26.746743+0000","last_undegraded":"2024-05-27T00:15:26.746743+0000","last_fullsized":"2024-05-27T00:15:26.746743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:59.804368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744980+0000","last_change":"2024-05-27T00:15:25.612157+0000","last_active":"2024-05-27T00:15:26.744980+0000","last_peered":"2024-05-27T00:15:26.744980+0000","last_clean":"2024-05-27T00:15:26.744980+0000","last_became_active":"2024-05-27T00:15:25.611884+0000","last_became_peered":"2024-05-27T00:15:25.611884+0000","last_unstale":"2024-05-27T00:15:26.744980+0000","last_undegraded":"2024-05-27T00:15:26.744980+0000","last_fullsized":"2024-05-27T00:15:26.744980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:12:28.505840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749289+0000","last_change":"2024-05-27T00:15:25.643312+0000","last_active":"2024-05-27T00:15:26.749289+0000","last_peered":"2024-05-27T00:15:26.749289+0000","last_clean":"2024-05-27T00:15:26.749289+0000","last_became_active":"2024-05-27T00:15:25.642663+0000","last_became_peered":"2024-05-27T00:15:25.642663+0000","last_unstale":"2024-05-27T00:15:26.749289+0000","last_undegraded":"2024-05-27T00:15:26.749289+0000","last_fullsized":"2024-05-27T00:15:26.749289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:06:27.232620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747471+0000","last_change":"2024-05-27T00:15:25.629892+0000","last_active":"2024-05-27T00:15:26.747471+0000","last_peered":"2024-05-27T00:15:26.747471+0000","last_clean":"2024-05-27T00:15:26.747471+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:15:26.747471+0000","last_undegraded":"2024-05-27T00:15:26.747471+0000","last_fullsized":"2024-05-27T00:15:26.747471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747909+0000","last_change":"2024-05-27T00:15:25.614887+0000","last_active":"2024-05-27T00:15:26.747909+0000","last_peered":"2024-05-27T00:15:26.747909+0000","last_clean":"2024-05-27T00:15:26.747909+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:15:26.747909+0000","last_undegraded":"2024-05-27T00:15:26.747909+0000","last_fullsized":"2024-05-27T00:15:26.747909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588218+0000","last_change":"2024-05-27T00:15:25.623864+0000","last_active":"2024-05-27T00:15:26.588218+0000","last_peered":"2024-05-27T00:15:26.588218+0000","last_clean":"2024-05-27T00:15:26.588218+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:15:26.588218+0000","last_undegraded":"2024-05-27T00:15:26.588218+0000","last_fullsized":"2024-05-27T00:15:26.588218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746025+0000","last_change":"2024-05-27T00:15:25.615495+0000","last_active":"2024-05-27T00:15:26.746025+0000","last_peered":"2024-05-27T00:15:26.746025+0000","last_clean":"2024-05-27T00:15:26.746025+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:15:26.746025+0000","last_undegraded":"2024-05-27T00:15:26.746025+0000","last_fullsized":"2024-05-27T00:15:26.746025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:42:53.882749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745273+0000","last_change":"2024-05-27T00:15:25.594508+0000","last_active":"2024-05-27T00:15:26.745273+0000","last_peered":"2024-05-27T00:15:26.745273+0000","last_clean":"2024-05-27T00:15:26.745273+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:15:26.745273+0000","last_undegraded":"2024-05-27T00:15:26.745273+0000","last_fullsized":"2024-05-27T00:15:26.745273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583449+0000","last_change":"2024-05-27T00:15:25.611818+0000","last_active":"2024-05-27T00:15:26.583449+0000","last_peered":"2024-05-27T00:15:26.583449+0000","last_clean":"2024-05-27T00:15:26.583449+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:15:26.583449+0000","last_undegraded":"2024-05-27T00:15:26.583449+0000","last_fullsized":"2024-05-27T00:15:26.583449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586675+0000","last_change":"2024-05-27T00:15:25.611365+0000","last_active":"2024-05-27T00:15:26.586675+0000","last_peered":"2024-05-27T00:15:26.586675+0000","last_clean":"2024-05-27T00:15:26.586675+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:15:26.586675+0000","last_undegraded":"2024-05-27T00:15:26.586675+0000","last_fullsized":"2024-05-27T00:15:26.586675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.590092+0000","last_change":"2024-05-27T00:15:25.632120+0000","last_active":"2024-05-27T00:15:26.590092+0000","last_peered":"2024-05-27T00:15:26.590092+0000","last_clean":"2024-05-27T00:15:26.590092+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:15:26.590092+0000","last_undegraded":"2024-05-27T00:15:26.590092+0000","last_fullsized":"2024-05-27T00:15:26.590092+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:06:27.232620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745618+0000","last_change":"2024-05-27T00:15:25.615868+0000","last_active":"2024-05-27T00:15:26.745618+0000","last_peered":"2024-05-27T00:15:26.745618+0000","last_clean":"2024-05-27T00:15:26.745618+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:15:26.745618+0000","last_undegraded":"2024-05-27T00:15:26.745618+0000","last_fullsized":"2024-05-27T00:15:26.745618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749314+0000","last_change":"2024-05-27T00:15:25.629101+0000","last_active":"2024-05-27T00:15:26.749314+0000","last_peered":"2024-05-27T00:15:26.749314+0000","last_clean":"2024-05-27T00:15:26.749314+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:15:26.749314+0000","last_undegraded":"2024-05-27T00:15:26.749314+0000","last_fullsized":"2024-05-27T00:15:26.749314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:26.893574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747537+0000","last_change":"2024-05-27T00:15:25.619351+0000","last_active":"2024-05-27T00:15:26.747537+0000","last_peered":"2024-05-27T00:15:26.747537+0000","last_clean":"2024-05-27T00:15:26.747537+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:15:26.747537+0000","last_undegraded":"2024-05-27T00:15:26.747537+0000","last_fullsized":"2024-05-27T00:15:26.747537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746054+0000","last_change":"2024-05-27T00:15:25.616436+0000","last_active":"2024-05-27T00:15:26.746054+0000","last_peered":"2024-05-27T00:15:26.746054+0000","last_clean":"2024-05-27T00:15:26.746054+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:15:26.746054+0000","last_undegraded":"2024-05-27T00:15:26.746054+0000","last_fullsized":"2024-05-27T00:15:26.746054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586830+0000","last_change":"2024-05-27T00:15:25.629373+0000","last_active":"2024-05-27T00:15:26.586830+0000","last_peered":"2024-05-27T00:15:26.586830+0000","last_clean":"2024-05-27T00:15:26.586830+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:15:26.586830+0000","last_undegraded":"2024-05-27T00:15:26.586830+0000","last_fullsized":"2024-05-27T00:15:26.586830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:02:15.012827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745308+0000","last_change":"2024-05-27T00:15:25.610707+0000","last_active":"2024-05-27T00:15:26.745308+0000","last_peered":"2024-05-27T00:15:26.745308+0000","last_clean":"2024-05-27T00:15:26.745308+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:15:26.745308+0000","last_undegraded":"2024-05-27T00:15:26.745308+0000","last_fullsized":"2024-05-27T00:15:26.745308+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:09:16.698319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587101+0000","last_change":"2024-05-27T00:15:25.613013+0000","last_active":"2024-05-27T00:15:26.587101+0000","last_peered":"2024-05-27T00:15:26.587101+0000","last_clean":"2024-05-27T00:15:26.587101+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:15:26.587101+0000","last_undegraded":"2024-05-27T00:15:26.587101+0000","last_fullsized":"2024-05-27T00:15:26.587101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745910+0000","last_change":"2024-05-27T00:15:25.610070+0000","last_active":"2024-05-27T00:15:26.745910+0000","last_peered":"2024-05-27T00:15:26.745910+0000","last_clean":"2024-05-27T00:15:26.745910+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:15:26.745910+0000","last_undegraded":"2024-05-27T00:15:26.745910+0000","last_fullsized":"2024-05-27T00:15:26.745910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:02:17.514654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747817+0000","last_change":"2024-05-27T00:15:25.633470+0000","last_active":"2024-05-27T00:15:26.747817+0000","last_peered":"2024-05-27T00:15:26.747817+0000","last_clean":"2024-05-27T00:15:26.747817+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:15:26.747817+0000","last_undegraded":"2024-05-27T00:15:26.747817+0000","last_fullsized":"2024-05-27T00:15:26.747817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583597+0000","last_change":"2024-05-27T00:15:25.597615+0000","last_active":"2024-05-27T00:15:26.583597+0000","last_peered":"2024-05-27T00:15:26.583597+0000","last_clean":"2024-05-27T00:15:26.583597+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:15:26.583597+0000","last_undegraded":"2024-05-27T00:15:26.583597+0000","last_fullsized":"2024-05-27T00:15:26.583597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746038+0000","last_change":"2024-05-27T00:15:25.610933+0000","last_active":"2024-05-27T00:15:26.746038+0000","last_peered":"2024-05-27T00:15:26.746038+0000","last_clean":"2024-05-27T00:15:26.746038+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:15:26.746038+0000","last_undegraded":"2024-05-27T00:15:26.746038+0000","last_fullsized":"2024-05-27T00:15:26.746038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:45:25.888432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746185+0000","last_change":"2024-05-27T00:15:25.598227+0000","last_active":"2024-05-27T00:15:26.746185+0000","last_peered":"2024-05-27T00:15:26.746185+0000","last_clean":"2024-05-27T00:15:26.746185+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:15:26.746185+0000","last_undegraded":"2024-05-27T00:15:26.746185+0000","last_fullsized":"2024-05-27T00:15:26.746185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588189+0000","last_change":"2024-05-27T00:15:25.620621+0000","last_active":"2024-05-27T00:15:26.588189+0000","last_peered":"2024-05-27T00:15:26.588189+0000","last_clean":"2024-05-27T00:15:26.588189+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:15:26.588189+0000","last_undegraded":"2024-05-27T00:15:26.588189+0000","last_fullsized":"2024-05-27T00:15:26.588189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:22:30.152810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744910+0000","last_change":"2024-05-27T00:15:25.605839+0000","last_active":"2024-05-27T00:15:26.744910+0000","last_peered":"2024-05-27T00:15:26.744910+0000","last_clean":"2024-05-27T00:15:26.744910+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:15:26.744910+0000","last_undegraded":"2024-05-27T00:15:26.744910+0000","last_fullsized":"2024-05-27T00:15:26.744910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746666+0000","last_change":"2024-05-27T00:15:25.607188+0000","last_active":"2024-05-27T00:15:26.746666+0000","last_peered":"2024-05-27T00:15:26.746666+0000","last_clean":"2024-05-27T00:15:26.746666+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:15:26.746666+0000","last_undegraded":"2024-05-27T00:15:26.746666+0000","last_fullsized":"2024-05-27T00:15:26.746666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587251+0000","last_change":"2024-05-27T00:15:25.615579+0000","last_active":"2024-05-27T00:15:26.587251+0000","last_peered":"2024-05-27T00:15:26.587251+0000","last_clean":"2024-05-27T00:15:26.587251+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:15:26.587251+0000","last_undegraded":"2024-05-27T00:15:26.587251+0000","last_fullsized":"2024-05-27T00:15:26.587251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:53.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746392+0000","last_change":"2024-05-27T00:15:24.611108+0000","last_active":"2024-05-27T00:15:26.746392+0000","last_peered":"2024-05-27T00:15:26.746392+0000","last_clean":"2024-05-27T00:15:26.746392+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:15:26.746392+0000","last_undegraded":"2024-05-27T00:15:26.746392+0000","last_fullsized":"2024-05-27T00:15:26.746392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:11.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746332+0000","last_change":"2024-05-27T00:15:25.614251+0000","last_active":"2024-05-27T00:15:26.746332+0000","last_peered":"2024-05-27T00:15:26.746332+0000","last_clean":"2024-05-27T00:15:26.746332+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:15:26.746332+0000","last_undegraded":"2024-05-27T00:15:26.746332+0000","last_fullsized":"2024-05-27T00:15:26.746332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747555+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:15:26.747555+0000","last_peered":"2024-05-27T00:15:26.747555+0000","last_clean":"2024-05-27T00:15:26.747555+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:15:26.747555+0000","last_undegraded":"2024-05-27T00:15:26.747555+0000","last_fullsized":"2024-05-27T00:15:26.747555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745607+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:15:26.745607+0000","last_peered":"2024-05-27T00:15:26.745607+0000","last_clean":"2024-05-27T00:15:26.745607+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:15:26.745607+0000","last_undegraded":"2024-05-27T00:15:26.745607+0000","last_fullsized":"2024-05-27T00:15:26.745607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:46:08.645927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745132+0000","last_change":"2024-05-27T00:15:25.596844+0000","last_active":"2024-05-27T00:15:26.745132+0000","last_peered":"2024-05-27T00:15:26.745132+0000","last_clean":"2024-05-27T00:15:26.745132+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:15:26.745132+0000","last_undegraded":"2024-05-27T00:15:26.745132+0000","last_fullsized":"2024-05-27T00:15:26.745132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:17:40.503615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587344+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:15:26.587344+0000","last_peered":"2024-05-27T00:15:26.587344+0000","last_clean":"2024-05-27T00:15:26.587344+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:15:26.587344+0000","last_undegraded":"2024-05-27T00:15:26.587344+0000","last_fullsized":"2024-05-27T00:15:26.587344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583235+0000","last_change":"2024-05-27T00:15:25.593149+0000","last_active":"2024-05-27T00:15:26.583235+0000","last_peered":"2024-05-27T00:15:26.583235+0000","last_clean":"2024-05-27T00:15:26.583235+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:15:26.583235+0000","last_undegraded":"2024-05-27T00:15:26.583235+0000","last_fullsized":"2024-05-27T00:15:26.583235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746974+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:15:26.746974+0000","last_peered":"2024-05-27T00:15:26.746974+0000","last_clean":"2024-05-27T00:15:26.746974+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:15:26.746974+0000","last_undegraded":"2024-05-27T00:15:26.746974+0000","last_fullsized":"2024-05-27T00:15:26.746974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589140+0000","last_change":"2024-05-27T00:15:25.618650+0000","last_active":"2024-05-27T00:15:26.589140+0000","last_peered":"2024-05-27T00:15:26.589140+0000","last_clean":"2024-05-27T00:15:26.589140+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:15:26.589140+0000","last_undegraded":"2024-05-27T00:15:26.589140+0000","last_fullsized":"2024-05-27T00:15:26.589140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:21.935526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589356+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:15:26.589356+0000","last_peered":"2024-05-27T00:15:26.589356+0000","last_clean":"2024-05-27T00:15:26.589356+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:15:26.589356+0000","last_undegraded":"2024-05-27T00:15:26.589356+0000","last_fullsized":"2024-05-27T00:15:26.589356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746448+0000","last_change":"2024-05-27T00:15:25.613142+0000","last_active":"2024-05-27T00:15:26.746448+0000","last_peered":"2024-05-27T00:15:26.746448+0000","last_clean":"2024-05-27T00:15:26.746448+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:15:26.746448+0000","last_undegraded":"2024-05-27T00:15:26.746448+0000","last_fullsized":"2024-05-27T00:15:26.746448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.620068+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:15:30.620068+0000","last_peered":"2024-05-27T00:15:30.620068+0000","last_clean":"2024-05-27T00:15:30.620068+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:15:30.620068+0000","last_undegraded":"2024-05-27T00:15:30.620068+0000","last_fullsized":"2024-05-27T00:15:30.620068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588245+0000","last_change":"2024-05-27T00:15:25.622383+0000","last_active":"2024-05-27T00:15:26.588245+0000","last_peered":"2024-05-27T00:15:26.588245+0000","last_clean":"2024-05-27T00:15:26.588245+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:15:26.588245+0000","last_undegraded":"2024-05-27T00:15:26.588245+0000","last_fullsized":"2024-05-27T00:15:26.588245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.624481+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:15:30.624481+0000","last_peered":"2024-05-27T00:15:30.624481+0000","last_clean":"2024-05-27T00:15:30.624481+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:15:30.624481+0000","last_undegraded":"2024-05-27T00:15:30.624481+0000","last_fullsized":"2024-05-27T00:15:30.624481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748511+0000","last_change":"2024-05-27T00:15:25.628646+0000","last_active":"2024-05-27T00:15:26.748511+0000","last_peered":"2024-05-27T00:15:26.748511+0000","last_clean":"2024-05-27T00:15:26.748511+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:15:26.748511+0000","last_undegraded":"2024-05-27T00:15:26.748511+0000","last_fullsized":"2024-05-27T00:15:26.748511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:22:30.152810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619790+0000","last_change":"2024-05-27T00:15:24.605986+0000","last_active":"2024-05-27T00:15:30.619790+0000","last_peered":"2024-05-27T00:15:30.619790+0000","last_clean":"2024-05-27T00:15:30.619790+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:15:30.619790+0000","last_undegraded":"2024-05-27T00:15:30.619790+0000","last_fullsized":"2024-05-27T00:15:30.619790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745414+0000","last_change":"2024-05-27T00:15:25.624136+0000","last_active":"2024-05-27T00:15:26.745414+0000","last_peered":"2024-05-27T00:15:26.745414+0000","last_clean":"2024-05-27T00:15:26.745414+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:15:26.745414+0000","last_undegraded":"2024-05-27T00:15:26.745414+0000","last_fullsized":"2024-05-27T00:15:26.745414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588435+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:15:26.588435+0000","last_peered":"2024-05-27T00:15:26.588435+0000","last_clean":"2024-05-27T00:15:26.588435+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:15:26.588435+0000","last_undegraded":"2024-05-27T00:15:26.588435+0000","last_fullsized":"2024-05-27T00:15:26.588435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748720+0000","last_change":"2024-05-27T00:15:25.641965+0000","last_active":"2024-05-27T00:15:26.748720+0000","last_peered":"2024-05-27T00:15:26.748720+0000","last_clean":"2024-05-27T00:15:26.748720+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:15:26.748720+0000","last_undegraded":"2024-05-27T00:15:26.748720+0000","last_fullsized":"2024-05-27T00:15:26.748720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:17.513529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746779+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:15:26.746779+0000","last_peered":"2024-05-27T00:15:26.746779+0000","last_clean":"2024-05-27T00:15:26.746779+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:15:26.746779+0000","last_undegraded":"2024-05-27T00:15:26.746779+0000","last_fullsized":"2024-05-27T00:15:26.746779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746732+0000","last_change":"2024-05-27T00:15:25.608549+0000","last_active":"2024-05-27T00:15:26.746732+0000","last_peered":"2024-05-27T00:15:26.746732+0000","last_clean":"2024-05-27T00:15:26.746732+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:15:26.746732+0000","last_undegraded":"2024-05-27T00:15:26.746732+0000","last_fullsized":"2024-05-27T00:15:26.746732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619615+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:15:30.619615+0000","last_peered":"2024-05-27T00:15:30.619615+0000","last_clean":"2024-05-27T00:15:30.619615+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:15:30.619615+0000","last_undegraded":"2024-05-27T00:15:30.619615+0000","last_fullsized":"2024-05-27T00:15:30.619615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747206+0000","last_change":"2024-05-27T00:15:25.638558+0000","last_active":"2024-05-27T00:15:26.747206+0000","last_peered":"2024-05-27T00:15:26.747206+0000","last_clean":"2024-05-27T00:15:26.747206+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:15:26.747206+0000","last_undegraded":"2024-05-27T00:15:26.747206+0000","last_fullsized":"2024-05-27T00:15:26.747206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:02:15.012827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587664+0000","last_change":"2024-05-27T00:15:24.601924+0000","last_active":"2024-05-27T00:15:26.587664+0000","last_peered":"2024-05-27T00:15:26.587664+0000","last_clean":"2024-05-27T00:15:26.587664+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:15:26.587664+0000","last_undegraded":"2024-05-27T00:15:26.587664+0000","last_fullsized":"2024-05-27T00:15:26.587664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:32.185839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749041+0000","last_change":"2024-05-27T00:15:25.639140+0000","last_active":"2024-05-27T00:15:26.749041+0000","last_peered":"2024-05-27T00:15:26.749041+0000","last_clean":"2024-05-27T00:15:26.749041+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:15:26.749041+0000","last_undegraded":"2024-05-27T00:15:26.749041+0000","last_fullsized":"2024-05-27T00:15:26.749041+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:21.935526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619020+0000","last_change":"2024-05-27T00:15:24.600878+0000","last_active":"2024-05-27T00:15:30.619020+0000","last_peered":"2024-05-27T00:15:30.619020+0000","last_clean":"2024-05-27T00:15:30.619020+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:15:30.619020+0000","last_undegraded":"2024-05-27T00:15:30.619020+0000","last_fullsized":"2024-05-27T00:15:30.619020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749435+0000","last_change":"2024-05-27T00:15:25.644249+0000","last_active":"2024-05-27T00:15:26.749435+0000","last_peered":"2024-05-27T00:15:26.749435+0000","last_clean":"2024-05-27T00:15:26.749435+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:15:26.749435+0000","last_undegraded":"2024-05-27T00:15:26.749435+0000","last_fullsized":"2024-05-27T00:15:26.749435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:59:25.042662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583715+0000","last_change":"2024-05-27T00:15:24.603452+0000","last_active":"2024-05-27T00:15:26.583715+0000","last_peered":"2024-05-27T00:15:26.583715+0000","last_clean":"2024-05-27T00:15:26.583715+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:15:26.583715+0000","last_undegraded":"2024-05-27T00:15:26.583715+0000","last_fullsized":"2024-05-27T00:15:26.583715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587865+0000","last_change":"2024-05-27T00:15:25.619341+0000","last_active":"2024-05-27T00:15:26.587865+0000","last_peered":"2024-05-27T00:15:26.587865+0000","last_clean":"2024-05-27T00:15:26.587865+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:15:26.587865+0000","last_undegraded":"2024-05-27T00:15:26.587865+0000","last_fullsized":"2024-05-27T00:15:26.587865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:41:46.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745117+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:15:26.745117+0000","last_peered":"2024-05-27T00:15:26.745117+0000","last_clean":"2024-05-27T00:15:26.745117+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:15:26.745117+0000","last_undegraded":"2024-05-27T00:15:26.745117+0000","last_fullsized":"2024-05-27T00:15:26.745117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748893+0000","last_change":"2024-05-27T00:15:25.639462+0000","last_active":"2024-05-27T00:15:26.748893+0000","last_peered":"2024-05-27T00:15:26.748893+0000","last_clean":"2024-05-27T00:15:26.748893+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:15:26.748893+0000","last_undegraded":"2024-05-27T00:15:26.748893+0000","last_fullsized":"2024-05-27T00:15:26.748893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:52:55.175945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746823+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:15:26.746823+0000","last_peered":"2024-05-27T00:15:26.746823+0000","last_clean":"2024-05-27T00:15:26.746823+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:15:26.746823+0000","last_undegraded":"2024-05-27T00:15:26.746823+0000","last_fullsized":"2024-05-27T00:15:26.746823+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586683+0000","last_change":"2024-05-27T00:15:24.603781+0000","last_active":"2024-05-27T00:15:26.586683+0000","last_peered":"2024-05-27T00:15:26.586683+0000","last_clean":"2024-05-27T00:15:26.586683+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:15:26.586683+0000","last_undegraded":"2024-05-27T00:15:26.586683+0000","last_fullsized":"2024-05-27T00:15:26.586683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746108+0000","last_change":"2024-05-27T00:15:25.631978+0000","last_active":"2024-05-27T00:15:26.746108+0000","last_peered":"2024-05-27T00:15:26.746108+0000","last_clean":"2024-05-27T00:15:26.746108+0000","last_became_active":"2024-05-27T00:15:25.631697+0000","last_became_peered":"2024-05-27T00:15:25.631697+0000","last_unstale":"2024-05-27T00:15:26.746108+0000","last_undegraded":"2024-05-27T00:15:26.746108+0000","last_fullsized":"2024-05-27T00:15:26.746108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:05.141408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746536+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:15:26.746536+0000","last_peered":"2024-05-27T00:15:26.746536+0000","last_clean":"2024-05-27T00:15:26.746536+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:15:26.746536+0000","last_undegraded":"2024-05-27T00:15:26.746536+0000","last_fullsized":"2024-05-27T00:15:26.746536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746253+0000","last_change":"2024-05-27T00:15:25.610627+0000","last_active":"2024-05-27T00:15:26.746253+0000","last_peered":"2024-05-27T00:15:26.746253+0000","last_clean":"2024-05-27T00:15:26.746253+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:15:26.746253+0000","last_undegraded":"2024-05-27T00:15:26.746253+0000","last_fullsized":"2024-05-27T00:15:26.746253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744898+0000","last_change":"2024-05-27T00:15:24.598182+0000","last_active":"2024-05-27T00:15:26.744898+0000","last_peered":"2024-05-27T00:15:26.744898+0000","last_clean":"2024-05-27T00:15:26.744898+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:15:26.744898+0000","last_undegraded":"2024-05-27T00:15:26.744898+0000","last_fullsized":"2024-05-27T00:15:26.744898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583243+0000","last_change":"2024-05-27T00:15:25.591449+0000","last_active":"2024-05-27T00:15:26.583243+0000","last_peered":"2024-05-27T00:15:26.583243+0000","last_clean":"2024-05-27T00:15:26.583243+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:15:26.583243+0000","last_undegraded":"2024-05-27T00:15:26.583243+0000","last_fullsized":"2024-05-27T00:15:26.583243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588693+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:15:26.588693+0000","last_peered":"2024-05-27T00:15:26.588693+0000","last_clean":"2024-05-27T00:15:26.588693+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:15:26.588693+0000","last_undegraded":"2024-05-27T00:15:26.588693+0000","last_fullsized":"2024-05-27T00:15:26.588693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587675+0000","last_change":"2024-05-27T00:15:25.614073+0000","last_active":"2024-05-27T00:15:26.587675+0000","last_peered":"2024-05-27T00:15:26.587675+0000","last_clean":"2024-05-27T00:15:26.587675+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:15:26.587675+0000","last_undegraded":"2024-05-27T00:15:26.587675+0000","last_fullsized":"2024-05-27T00:15:26.587675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583569+0000","last_change":"2024-05-27T00:15:24.603518+0000","last_active":"2024-05-27T00:15:26.583569+0000","last_peered":"2024-05-27T00:15:26.583569+0000","last_clean":"2024-05-27T00:15:26.583569+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:15:26.583569+0000","last_undegraded":"2024-05-27T00:15:26.583569+0000","last_fullsized":"2024-05-27T00:15:26.583569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588810+0000","last_change":"2024-05-27T00:15:25.621519+0000","last_active":"2024-05-27T00:15:26.588810+0000","last_peered":"2024-05-27T00:15:26.588810+0000","last_clean":"2024-05-27T00:15:26.588810+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:15:26.588810+0000","last_undegraded":"2024-05-27T00:15:26.588810+0000","last_fullsized":"2024-05-27T00:15:26.588810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:52:55.175945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749299+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:15:26.749299+0000","last_peered":"2024-05-27T00:15:26.749299+0000","last_clean":"2024-05-27T00:15:26.749299+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:15:26.749299+0000","last_undegraded":"2024-05-27T00:15:26.749299+0000","last_fullsized":"2024-05-27T00:15:26.749299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583460+0000","last_change":"2024-05-27T00:15:25.593180+0000","last_active":"2024-05-27T00:15:26.583460+0000","last_peered":"2024-05-27T00:15:26.583460+0000","last_clean":"2024-05-27T00:15:26.583460+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:15:26.583460+0000","last_undegraded":"2024-05-27T00:15:26.583460+0000","last_fullsized":"2024-05-27T00:15:26.583460+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583660+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:15:26.583660+0000","last_peered":"2024-05-27T00:15:26.583660+0000","last_clean":"2024-05-27T00:15:26.583660+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:15:26.583660+0000","last_undegraded":"2024-05-27T00:15:26.583660+0000","last_fullsized":"2024-05-27T00:15:26.583660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745915+0000","last_change":"2024-05-27T00:15:25.631919+0000","last_active":"2024-05-27T00:15:26.745915+0000","last_peered":"2024-05-27T00:15:26.745915+0000","last_clean":"2024-05-27T00:15:26.745915+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:15:26.745915+0000","last_undegraded":"2024-05-27T00:15:26.745915+0000","last_fullsized":"2024-05-27T00:15:26.745915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:24:31.995603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.620892+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:15:30.620892+0000","last_peered":"2024-05-27T00:15:30.620892+0000","last_clean":"2024-05-27T00:15:30.620892+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:15:30.620892+0000","last_undegraded":"2024-05-27T00:15:30.620892+0000","last_fullsized":"2024-05-27T00:15:30.620892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745629+0000","last_change":"2024-05-27T00:15:25.596792+0000","last_active":"2024-05-27T00:15:26.745629+0000","last_peered":"2024-05-27T00:15:26.745629+0000","last_clean":"2024-05-27T00:15:26.745629+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:15:26.745629+0000","last_undegraded":"2024-05-27T00:15:26.745629+0000","last_fullsized":"2024-05-27T00:15:26.745629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747720+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:15:26.747720+0000","last_peered":"2024-05-27T00:15:26.747720+0000","last_clean":"2024-05-27T00:15:26.747720+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:15:26.747720+0000","last_undegraded":"2024-05-27T00:15:26.747720+0000","last_fullsized":"2024-05-27T00:15:26.747720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587779+0000","last_change":"2024-05-27T00:15:25.610697+0000","last_active":"2024-05-27T00:15:26.587779+0000","last_peered":"2024-05-27T00:15:26.587779+0000","last_clean":"2024-05-27T00:15:26.587779+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:15:26.587779+0000","last_undegraded":"2024-05-27T00:15:26.587779+0000","last_fullsized":"2024-05-27T00:15:26.587779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:14.969827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583375+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:15:26.583375+0000","last_peered":"2024-05-27T00:15:26.583375+0000","last_clean":"2024-05-27T00:15:26.583375+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:15:26.583375+0000","last_undegraded":"2024-05-27T00:15:26.583375+0000","last_fullsized":"2024-05-27T00:15:26.583375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745282+0000","last_change":"2024-05-27T00:15:25.597164+0000","last_active":"2024-05-27T00:15:26.745282+0000","last_peered":"2024-05-27T00:15:26.745282+0000","last_clean":"2024-05-27T00:15:26.745282+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:15:26.745282+0000","last_undegraded":"2024-05-27T00:15:26.745282+0000","last_fullsized":"2024-05-27T00:15:26.745282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:04.655872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583422+0000","last_change":"2024-05-27T00:15:24.598238+0000","last_active":"2024-05-27T00:15:26.583422+0000","last_peered":"2024-05-27T00:15:26.583422+0000","last_clean":"2024-05-27T00:15:26.583422+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:15:26.583422+0000","last_undegraded":"2024-05-27T00:15:26.583422+0000","last_fullsized":"2024-05-27T00:15:26.583422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746235+0000","last_change":"2024-05-27T00:15:25.601140+0000","last_active":"2024-05-27T00:15:26.746235+0000","last_peered":"2024-05-27T00:15:26.746235+0000","last_clean":"2024-05-27T00:15:26.746235+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:15:26.746235+0000","last_undegraded":"2024-05-27T00:15:26.746235+0000","last_fullsized":"2024-05-27T00:15:26.746235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:42:05.334730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.621275+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:15:30.621275+0000","last_peered":"2024-05-27T00:15:30.621275+0000","last_clean":"2024-05-27T00:15:30.621275+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:15:30.621275+0000","last_undegraded":"2024-05-27T00:15:30.621275+0000","last_fullsized":"2024-05-27T00:15:30.621275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747292+0000","last_change":"2024-05-27T00:15:25.619694+0000","last_active":"2024-05-27T00:15:26.747292+0000","last_peered":"2024-05-27T00:15:26.747292+0000","last_clean":"2024-05-27T00:15:26.747292+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:15:26.747292+0000","last_undegraded":"2024-05-27T00:15:26.747292+0000","last_fullsized":"2024-05-27T00:15:26.747292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588751+0000","last_change":"2024-05-27T00:15:25.627578+0000","last_active":"2024-05-27T00:15:26.588751+0000","last_peered":"2024-05-27T00:15:26.588751+0000","last_clean":"2024-05-27T00:15:26.588751+0000","last_became_active":"2024-05-27T00:15:25.622421+0000","last_became_peered":"2024-05-27T00:15:25.622421+0000","last_unstale":"2024-05-27T00:15:26.588751+0000","last_undegraded":"2024-05-27T00:15:26.588751+0000","last_fullsized":"2024-05-27T00:15:26.588751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:17.513529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619639+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:15:30.619639+0000","last_peered":"2024-05-27T00:15:30.619639+0000","last_clean":"2024-05-27T00:15:30.619639+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:15:30.619639+0000","last_undegraded":"2024-05-27T00:15:30.619639+0000","last_fullsized":"2024-05-27T00:15:30.619639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587486+0000","last_change":"2024-05-27T00:15:25.623207+0000","last_active":"2024-05-27T00:15:26.587486+0000","last_peered":"2024-05-27T00:15:26.587486+0000","last_clean":"2024-05-27T00:15:26.587486+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:15:26.587486+0000","last_undegraded":"2024-05-27T00:15:26.587486+0000","last_fullsized":"2024-05-27T00:15:26.587486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:59.804368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.618376+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:15:30.618376+0000","last_peered":"2024-05-27T00:15:30.618376+0000","last_clean":"2024-05-27T00:15:30.618376+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:15:30.618376+0000","last_undegraded":"2024-05-27T00:15:30.618376+0000","last_fullsized":"2024-05-27T00:15:30.618376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746305+0000","last_change":"2024-05-27T00:15:25.613018+0000","last_active":"2024-05-27T00:15:26.746305+0000","last_peered":"2024-05-27T00:15:26.746305+0000","last_clean":"2024-05-27T00:15:26.746305+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:15:26.746305+0000","last_undegraded":"2024-05-27T00:15:26.746305+0000","last_fullsized":"2024-05-27T00:15:26.746305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745698+0000","last_change":"2024-05-27T00:15:24.600571+0000","last_active":"2024-05-27T00:15:26.745698+0000","last_peered":"2024-05-27T00:15:26.745698+0000","last_clean":"2024-05-27T00:15:26.745698+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:15:26.745698+0000","last_undegraded":"2024-05-27T00:15:26.745698+0000","last_fullsized":"2024-05-27T00:15:26.745698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:44:21.645223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745724+0000","last_change":"2024-05-27T00:15:25.615649+0000","last_active":"2024-05-27T00:15:26.745724+0000","last_peered":"2024-05-27T00:15:26.745724+0000","last_clean":"2024-05-27T00:15:26.745724+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:15:26.745724+0000","last_undegraded":"2024-05-27T00:15:26.745724+0000","last_fullsized":"2024-05-27T00:15:26.745724+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:16:16.309939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.621025+0000","last_change":"2024-05-27T00:15:24.615066+0000","last_active":"2024-05-27T00:15:30.621025+0000","last_peered":"2024-05-27T00:15:30.621025+0000","last_clean":"2024-05-27T00:15:30.621025+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:15:30.621025+0000","last_undegraded":"2024-05-27T00:15:30.621025+0000","last_fullsized":"2024-05-27T00:15:30.621025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":669858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":79034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":78908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":79068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":669914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:15:35.719 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-27T00:15:35.719 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-27T00:15:35.719 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-27T00:15:35.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-27T00:15:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10fa00640 1 Processor -- start 2024-05-27T00:15:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10fa00640 1 -- start start 2024-05-27T00:15:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10fa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc108152a50 0x7fc1080709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10fa00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc10815be40 con 0x7fc1080cb7e0 2024-05-27T00:15:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10fa00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1080c50e0 con 0x7fc108158e60 2024-05-27T00:15:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10fa00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc1080c8a00 con 0x7fc108152a50 2024-05-27T00:15:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10d000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc108152a50 0x7fc1080709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10d000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc108152a50 0x7fc1080709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51162/0 (socket says 172.21.15.98:51162) 2024-05-27T00:15:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10d000640 1 -- 172.21.15.98:0/3674416219 learned_addr learned my addr 172.21.15.98:0/3674416219 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc0ffe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc1080cb7e0 0x7fc108158750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47114/0 (socket says 172.21.15.98:47114) 2024-05-27T00:15:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10d000640 1 -- 172.21.15.98:0/3674416219 >> v1:172.21.15.204:6789/0 conn(0x7fc108158e60 legacy=0x7fc108162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10d000640 1 -- 172.21.15.98:0/3674416219 >> v1:172.21.15.98:6789/0 conn(0x7fc1080cb7e0 legacy=0x7fc108158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc10d000640 1 -- 172.21.15.98:0/3674416219 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1080c8c50 con 0x7fc108152a50 2024-05-27T00:15:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.897+0000 7fc107e00640 1 -- 172.21.15.98:0/3674416219 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3035631128 0 0) 0x7fc10815be40 con 0x7fc1080cb7e0 2024-05-27T00:15:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc10d000640 1 --2- 172.21.15.98:0/3674416219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc108152a50 0x7fc1080709a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009870 tx=0x7fc0f801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899c82c2de29ee11 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc107e00640 1 -- 172.21.15.98:0/3674416219 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f80ab660 con 0x7fc108152a50 2024-05-27T00:15:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc107e00640 1 -- 172.21.15.98:0/3674416219 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0f80b0070 con 0x7fc108152a50 2024-05-27T00:15:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc107e00640 1 -- 172.21.15.98:0/3674416219 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f80aba20 con 0x7fc108152a50 2024-05-27T00:15:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc10fa00640 1 -- 172.21.15.98:0/3674416219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc108152a50 msgr2=0x7fc1080709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc10fa00640 1 --2- 172.21.15.98:0/3674416219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc108152a50 0x7fc1080709a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009870 tx=0x7fc0f801c460 comp rx=0 tx=0).stop 2024-05-27T00:15:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc10fa00640 1 -- 172.21.15.98:0/3674416219 shutdown_connections 2024-05-27T00:15:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc10fa00640 1 --2- 172.21.15.98:0/3674416219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc108152a50 0x7fc1080709a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc10fa00640 1 -- 172.21.15.98:0/3674416219 >> 172.21.15.98:0/3674416219 conn(0x7fc108072f90 msgr2=0x7fc1080733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc10fa00640 1 -- 172.21.15.98:0/3674416219 shutdown_connections 2024-05-27T00:15:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc10fa00640 1 -- 172.21.15.98:0/3674416219 wait complete. 2024-05-27T00:15:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.901+0000 7fc10fa00640 1 Processor -- start 2024-05-27T00:15:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10fa00640 1 -- start start 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10fa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1080cb7e0 0x7fc10806ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10fa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc108152a50 0x7fc10814a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10fa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc108158e60 0x7fc1081f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10fa00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc108152e30 con 0x7fc1080cb7e0 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10fa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc1080cb570 con 0x7fc108152a50 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10fa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc1080c9c10 con 0x7fc108158e60 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1080cb7e0 0x7fc10806ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1080cb7e0 0x7fc10806ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51170/0 (socket says 172.21.15.98:51170) 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10d000640 1 -- 172.21.15.98:0/3703318861 learned_addr learned my addr 172.21.15.98:0/3703318861 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc0ffe00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc108152a50 0x7fc10814a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10da00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc108158e60 0x7fc1081f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10d000640 1 -- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc108158e60 msgr2=0x7fc1081f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10d000640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc108158e60 0x7fc1081f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10d000640 1 -- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc108152a50 msgr2=0x7fc10814a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10d000640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc108152a50 0x7fc10814a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10d000640 1 -- 172.21.15.98:0/3703318861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10806d390 con 0x7fc1080cb7e0 2024-05-27T00:15:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc0ffe00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc108152a50 0x7fc10814a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10d000640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1080cb7e0 0x7fc10806ce50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80ad040 tx=0x7fc0f801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc105600640 1 -- 172.21.15.98:0/3703318861 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f80b4800 con 0x7fc1080cb7e0 2024-05-27T00:15:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc105600640 1 -- 172.21.15.98:0/3703318861 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0f80b49c0 con 0x7fc1080cb7e0 2024-05-27T00:15:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10fa00640 1 -- 172.21.15.98:0/3703318861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1081f7580 con 0x7fc1080cb7e0 2024-05-27T00:15:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc105600640 1 -- 172.21.15.98:0/3703318861 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f80b4db0 con 0x7fc1080cb7e0 2024-05-27T00:15:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10fa00640 1 -- 172.21.15.98:0/3703318861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1081f7b40 con 0x7fc1080cb7e0 2024-05-27T00:15:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.905+0000 7fc10fa00640 1 -- 172.21.15.98:0/3703318861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1081f3dc0 con 0x7fc1080cb7e0 2024-05-27T00:15:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.909+0000 7fc105600640 1 -- 172.21.15.98:0/3703318861 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc0f80c7440 con 0x7fc1080cb7e0 2024-05-27T00:15:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.909+0000 7fc105600640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0e40678f0 0x7fc0e4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.909+0000 7fc105600640 1 -- 172.21.15.98:0/3703318861 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc0f8120710 con 0x7fc1080cb7e0 2024-05-27T00:15:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.909+0000 7fc0ffe00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0e40678f0 0x7fc0e4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.909+0000 7fc0ffe00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0e40678f0 0x7fc0e4069db0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc108148de0 tx=0x7fc0f0007c60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:35.913+0000 7fc105600640 1 -- 172.21.15.98:0/3703318861 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc1081f3dc0 con 0x7fc1080cb7e0 2024-05-27T00:15:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 -- 172.21.15.98:0/3703318861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc108075cb0 con 0x7fc1080cb7e0 2024-05-27T00:15:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc105600640 1 -- 172.21.15.98:0/3703318861 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fc0f8016110 con 0x7fc1080cb7e0 2024-05-27T00:15:36.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:36.104 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-27T00:15:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 -- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0e40678f0 msgr2=0x7fc0e4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0e40678f0 0x7fc0e4069db0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc108148de0 tx=0x7fc0f0007c60 comp rx=0 tx=0).stop 2024-05-27T00:15:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 -- 172.21.15.98:0/3703318861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1080cb7e0 msgr2=0x7fc10806ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1080cb7e0 0x7fc10806ce50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80ad040 tx=0x7fc0f801aca0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 -- 172.21.15.98:0/3703318861 shutdown_connections 2024-05-27T00:15:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc108158e60 0x7fc1081f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0e40678f0 0x7fc0e4069db0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc108152a50 0x7fc10814a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 --2- 172.21.15.98:0/3703318861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1080cb7e0 0x7fc10806ce50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 -- 172.21.15.98:0/3703318861 >> 172.21.15.98:0/3703318861 conn(0x7fc108072f90 msgr2=0x7fc10815b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 -- 172.21.15.98:0/3703318861 shutdown_connections 2024-05-27T00:15:36.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.101+0000 7fc10fa00640 1 -- 172.21.15.98:0/3703318861 wait complete. 2024-05-27T00:15:36.126 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-27T00:15:36.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.257+0000 7f9b41800640 1 Processor -- start 2024-05-27T00:15:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.257+0000 7f9b41800640 1 -- start start 2024-05-27T00:15:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b41800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c154830 0x7f9b3c15b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b41800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9b3c0cbbb0 0x7f9b3c0cc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b41800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b3c0c1a90 con 0x7f9b3c0c68c0 2024-05-27T00:15:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b41800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9b3c0c2b30 con 0x7f9b3c0cbbb0 2024-05-27T00:15:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b41800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3c0c26a0 con 0x7f9b3c154830 2024-05-27T00:15:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b3b400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9b3c0c68c0 0x7f9b3c071e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47120/0 (socket says 172.21.15.98:47120) 2024-05-27T00:15:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b3b400640 1 -- 172.21.15.98:0/449541486 learned_addr learned my addr 172.21.15.98:0/449541486 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b3aa00640 1 --2- 172.21.15.98:0/449541486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c154830 0x7f9b3c15b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b3a000640 1 --2- 172.21.15.98:0/449541486 >> v2:172.21.15.204:3300/0 conn(0x7f9b3c0cbbb0 0x7f9b3c0cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b39600640 1 -- 172.21.15.98:0/449541486 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659964966 0 0) 0x7f9b3c0c1a90 con 0x7f9b3c0c68c0 2024-05-27T00:15:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b39600640 1 -- 172.21.15.98:0/449541486 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b20003530 con 0x7f9b3c0c68c0 2024-05-27T00:15:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b3aa00640 1 -- 172.21.15.98:0/449541486 >> v2:172.21.15.204:3300/0 conn(0x7f9b3c0cbbb0 msgr2=0x7f9b3c0cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b3aa00640 1 --2- 172.21.15.98:0/449541486 >> v2:172.21.15.204:3300/0 conn(0x7f9b3c0cbbb0 0x7f9b3c0cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b3aa00640 1 -- 172.21.15.98:0/449541486 >> v1:172.21.15.98:6789/0 conn(0x7f9b3c0c68c0 legacy=0x7f9b3c071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.261+0000 7f9b3aa00640 1 -- 172.21.15.98:0/449541486 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3c0c1d70 con 0x7f9b3c154830 2024-05-27T00:15:36.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b3aa00640 1 --2- 172.21.15.98:0/449541486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c154830 0x7f9b3c15b470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009d90 tx=0x7f9b2c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b08f28c04b3f0b71 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b39600640 1 -- 172.21.15.98:0/449541486 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c0b2070 con 0x7f9b3c154830 2024-05-27T00:15:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b39600640 1 -- 172.21.15.98:0/449541486 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b2c0a5920 con 0x7f9b3c154830 2024-05-27T00:15:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b39600640 1 -- 172.21.15.98:0/449541486 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c0a5cd0 con 0x7f9b3c154830 2024-05-27T00:15:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b41800640 1 -- 172.21.15.98:0/449541486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c154830 msgr2=0x7f9b3c15b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b41800640 1 --2- 172.21.15.98:0/449541486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c154830 0x7f9b3c15b470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009d90 tx=0x7f9b2c01c970 comp rx=0 tx=0).stop 2024-05-27T00:15:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b41800640 1 -- 172.21.15.98:0/449541486 shutdown_connections 2024-05-27T00:15:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b41800640 1 --2- 172.21.15.98:0/449541486 >> v2:172.21.15.204:3300/0 conn(0x7f9b3c0cbbb0 0x7f9b3c0cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b41800640 1 --2- 172.21.15.98:0/449541486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c154830 0x7f9b3c15b470 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b41800640 1 -- 172.21.15.98:0/449541486 >> 172.21.15.98:0/449541486 conn(0x7f9b3c067530 msgr2=0x7f9b3c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b41800640 1 -- 172.21.15.98:0/449541486 shutdown_connections 2024-05-27T00:15:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b41800640 1 -- 172.21.15.98:0/449541486 wait complete. 2024-05-27T00:15:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b41800640 1 Processor -- start 2024-05-27T00:15:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.265+0000 7f9b41800640 1 -- start start 2024-05-27T00:15:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b41800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b3c0c68c0 0x7f9b3c15a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b41800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c0cbbb0 0x7f9b3c158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b41800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c154830 0x7f9b3c15ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b41800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c154d90 con 0x7f9b3c154830 2024-05-27T00:15:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b41800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c154c10 con 0x7f9b3c0c68c0 2024-05-27T00:15:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b41800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3c0725a0 con 0x7f9b3c0cbbb0 2024-05-27T00:15:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c0cbbb0 0x7f9b3c158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c154830 0x7f9b3c15ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c0cbbb0 0x7f9b3c158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:53398/0 (socket says 172.21.15.98:53398) 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3a000640 1 -- 172.21.15.98:0/2367059677 learned_addr learned my addr 172.21.15.98:0/2367059677 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3aa00640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b3c0c68c0 0x7f9b3c15a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3b400640 1 -- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c0cbbb0 msgr2=0x7f9b3c158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3b400640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c0cbbb0 0x7f9b3c158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3b400640 1 -- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b3c0c68c0 msgr2=0x7f9b3c15a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3b400640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b3c0c68c0 0x7f9b3c15a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3b400640 1 -- 172.21.15.98:0/2367059677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3c159870 con 0x7f9b3c154830 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b3b400640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c154830 0x7f9b3c15ba90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400c3f0 tx=0x7f9b3400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b32a00640 1 -- 172.21.15.98:0/2367059677 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34017070 con 0x7f9b3c154830 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b32a00640 1 -- 172.21.15.98:0/2367059677 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b34005cf0 con 0x7f9b3c154830 2024-05-27T00:15:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b32a00640 1 -- 172.21.15.98:0/2367059677 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b340072d0 con 0x7f9b3c154830 2024-05-27T00:15:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b41800640 1 -- 172.21.15.98:0/2367059677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b3c0ca6a0 con 0x7f9b3c154830 2024-05-27T00:15:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b41800640 1 -- 172.21.15.98:0/2367059677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b3c0cac60 con 0x7f9b3c154830 2024-05-27T00:15:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b41800640 1 -- 172.21.15.98:0/2367059677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b3c075cb0 con 0x7f9b3c154830 2024-05-27T00:15:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.269+0000 7f9b32a00640 1 -- 172.21.15.98:0/2367059677 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9b3401a070 con 0x7f9b3c154830 2024-05-27T00:15:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.273+0000 7f9b32a00640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9b1c0678c0 0x7f9b1c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.273+0000 7f9b32a00640 1 -- 172.21.15.98:0/2367059677 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f9b34084f80 con 0x7f9b3c154830 2024-05-27T00:15:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.273+0000 7f9b3aa00640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9b1c0678c0 0x7f9b1c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.273+0000 7f9b3aa00640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9b1c0678c0 0x7f9b1c069d80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c0af000 tx=0x7f9b2c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.277+0000 7f9b32a00640 1 -- 172.21.15.98:0/2367059677 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9b34055340 con 0x7f9b3c154830 2024-05-27T00:15:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.433+0000 7f9b41800640 1 -- 172.21.15.98:0/2367059677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b3c0c1300 con 0x7f9b3c154830 2024-05-27T00:15:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.433+0000 7f9b32a00640 1 -- 172.21.15.98:0/2367059677 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16581 (secure 0 0 0) 0x7f9b34059640 con 0x7f9b3c154830 2024-05-27T00:15:36.438 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:36.438 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:26.579229+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:36.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 -- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9b1c0678c0 msgr2=0x7f9b1c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9b1c0678c0 0x7f9b1c069d80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c0af000 tx=0x7f9b2c0a7070 comp rx=0 tx=0).stop 2024-05-27T00:15:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 -- 172.21.15.98:0/2367059677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c154830 msgr2=0x7f9b3c15ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c154830 0x7f9b3c15ba90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400c3f0 tx=0x7f9b3400c8b0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 -- 172.21.15.98:0/2367059677 shutdown_connections 2024-05-27T00:15:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b3c154830 0x7f9b3c15ba90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b3c0cbbb0 0x7f9b3c158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9b1c0678c0 0x7f9b1c069d80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 --2- 172.21.15.98:0/2367059677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b3c0c68c0 0x7f9b3c15a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 -- 172.21.15.98:0/2367059677 >> 172.21.15.98:0/2367059677 conn(0x7f9b3c067530 msgr2=0x7f9b3c15a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 -- 172.21.15.98:0/2367059677 shutdown_connections 2024-05-27T00:15:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.437+0000 7f9b41800640 1 -- 172.21.15.98:0/2367059677 wait complete. 2024-05-27T00:15:36.459 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T00:15:13.211703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T00:15:23.224464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T00:15:23.950896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T00:15:36.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T00:15:36.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd622200640 1 Processor -- start 2024-05-27T00:15:36.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd622200640 1 -- start start 2024-05-27T00:15:36.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd622200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd61c1526e0 0x7fd61c0c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd622200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd61c0c3090 0x7fd61c06f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd622200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd61c0c4d20 con 0x7fd61c159a80 2024-05-27T00:15:36.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd622200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd61c152ac0 con 0x7fd61c0c3090 2024-05-27T00:15:36.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd622200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd61c0c3640 con 0x7fd61c1526e0 2024-05-27T00:15:36.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd61be00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd61c0c3090 0x7fd61c06f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd61aa00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd61c159a80 0x7fd61c158fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47124/0 (socket says 172.21.15.98:47124) 2024-05-27T00:15:36.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd61aa00640 1 -- 172.21.15.98:0/1695061609 learned_addr learned my addr 172.21.15.98:0/1695061609 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:36.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd61b400640 1 --2- 172.21.15.98:0/1695061609 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd61c1526e0 0x7fd61c0c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd61be00640 1 -- 172.21.15.98:0/1695061609 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd61c1526e0 msgr2=0x7fd61c0c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:36.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd61be00640 1 --2- 172.21.15.98:0/1695061609 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd61c1526e0 0x7fd61c0c8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd61be00640 1 -- 172.21.15.98:0/1695061609 >> v1:172.21.15.98:6789/0 conn(0x7fd61c159a80 legacy=0x7fd61c158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd61be00640 1 -- 172.21.15.98:0/1695061609 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61c0c5000 con 0x7fd61c0c3090 2024-05-27T00:15:36.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.597+0000 7fd61a000640 1 -- 172.21.15.98:0/1695061609 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1186104003 0 0) 0x7fd61c0c4d20 con 0x7fd61c159a80 2024-05-27T00:15:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd61be00640 1 --2- 172.21.15.98:0/1695061609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c0c3090 0x7fd61c06f2d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd6140097f0 tx=0x7fd61401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d312a1258be62ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd61a000640 1 -- 172.21.15.98:0/1695061609 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140b5070 con 0x7fd61c0c3090 2024-05-27T00:15:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd61a000640 1 -- 172.21.15.98:0/1695061609 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6140a8990 con 0x7fd61c0c3090 2024-05-27T00:15:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd61a000640 1 -- 172.21.15.98:0/1695061609 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140a8d60 con 0x7fd61c0c3090 2024-05-27T00:15:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd622200640 1 -- 172.21.15.98:0/1695061609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c0c3090 msgr2=0x7fd61c06f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd622200640 1 --2- 172.21.15.98:0/1695061609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c0c3090 0x7fd61c06f2d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd6140097f0 tx=0x7fd61401ec90 comp rx=0 tx=0).stop 2024-05-27T00:15:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd622200640 1 -- 172.21.15.98:0/1695061609 shutdown_connections 2024-05-27T00:15:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd622200640 1 --2- 172.21.15.98:0/1695061609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c0c3090 0x7fd61c06f2d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd622200640 1 --2- 172.21.15.98:0/1695061609 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd61c1526e0 0x7fd61c0c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd622200640 1 -- 172.21.15.98:0/1695061609 >> 172.21.15.98:0/1695061609 conn(0x7fd61c06d680 msgr2=0x7fd61c06da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd622200640 1 -- 172.21.15.98:0/1695061609 shutdown_connections 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.601+0000 7fd622200640 1 -- 172.21.15.98:0/1695061609 wait complete. 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 Processor -- start 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 -- start start 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd61c0c3090 0x7fd61c14e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd61c1526e0 0x7fd61c1e2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c159a80 0x7fd61c1f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c06fff0 con 0x7fd61c159a80 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c158a80 con 0x7fd61c0c3090 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd61c070170 con 0x7fd61c1526e0 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c159a80 0x7fd61c1f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c159a80 0x7fd61c1f16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51190/0 (socket says 172.21.15.98:51190) 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61be00640 1 -- 172.21.15.98:0/454440258 learned_addr learned my addr 172.21.15.98:0/454440258 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:36.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61aa00640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd61c1526e0 0x7fd61c1e2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61b400640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd61c0c3090 0x7fd61c14e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61be00640 1 -- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd61c1526e0 msgr2=0x7fd61c1e2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61be00640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd61c1526e0 0x7fd61c1e2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61be00640 1 -- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd61c0c3090 msgr2=0x7fd61c14e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61be00640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd61c0c3090 0x7fd61c14e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61be00640 1 -- 172.21.15.98:0/454440258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61c0c4a30 con 0x7fd61c159a80 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd61be00640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c159a80 0x7fd61c1f16f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd6140b3040 tx=0x7fd61401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd613400640 1 -- 172.21.15.98:0/454440258 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140b5040 con 0x7fd61c159a80 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd613400640 1 -- 172.21.15.98:0/454440258 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6140bf070 con 0x7fd61c159a80 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 -- 172.21.15.98:0/454440258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd61c1ef050 con 0x7fd61c159a80 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd613400640 1 -- 172.21.15.98:0/454440258 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140b0940 con 0x7fd61c159a80 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 -- 172.21.15.98:0/454440258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd61c1ee290 con 0x7fd61c159a80 2024-05-27T00:15:36.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.605+0000 7fd622200640 1 -- 172.21.15.98:0/454440258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd61c075cb0 con 0x7fd61c159a80 2024-05-27T00:15:36.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.609+0000 7fd613400640 1 -- 172.21.15.98:0/454440258 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd6140b0ae0 con 0x7fd61c159a80 2024-05-27T00:15:36.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.609+0000 7fd613400640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd5fc0678c0 0x7fd5fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.609+0000 7fd613400640 1 -- 172.21.15.98:0/454440258 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd6141231d0 con 0x7fd61c159a80 2024-05-27T00:15:36.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.609+0000 7fd61b400640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd5fc0678c0 0x7fd5fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.609+0000 7fd61b400640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd5fc0678c0 0x7fd5fc069d80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0074a0 tx=0x7fd60c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:36.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.613+0000 7fd613400640 1 -- 172.21.15.98:0/454440258 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd6140187f0 con 0x7fd61c159a80 2024-05-27T00:15:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.769+0000 7fd622200640 1 -- 172.21.15.98:0/454440258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd61c068950 con 0x7fd61c159a80 2024-05-27T00:15:36.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.769+0000 7fd613400640 1 -- 172.21.15.98:0/454440258 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fd6140f3be0 con 0x7fd61c159a80 2024-05-27T00:15:36.774 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-27T00:15:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 -- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd5fc0678c0 msgr2=0x7fd5fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd5fc0678c0 0x7fd5fc069d80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0074a0 tx=0x7fd60c00b040 comp rx=0 tx=0).stop 2024-05-27T00:15:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 -- 172.21.15.98:0/454440258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c159a80 msgr2=0x7fd61c1f16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c159a80 0x7fd61c1f16f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd6140b3040 tx=0x7fd61401ba30 comp rx=0 tx=0).stop 2024-05-27T00:15:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 -- 172.21.15.98:0/454440258 shutdown_connections 2024-05-27T00:15:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd61c159a80 0x7fd61c1f16f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd61c1526e0 0x7fd61c1e2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd5fc0678c0 0x7fd5fc069d80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 --2- 172.21.15.98:0/454440258 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd61c0c3090 0x7fd61c14e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 -- 172.21.15.98:0/454440258 >> 172.21.15.98:0/454440258 conn(0x7fd61c06d680 msgr2=0x7fd61c159e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:36.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 -- 172.21.15.98:0/454440258 shutdown_connections 2024-05-27T00:15:36.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.773+0000 7fd622200640 1 -- 172.21.15.98:0/454440258 wait complete. 2024-05-27T00:15:36.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T00:15:36.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a1a200640 1 Processor -- start 2024-05-27T00:15:36.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a1a200640 1 -- start start 2024-05-27T00:15:36.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a140b1e10 0x7f7a1406e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a1a200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7a140c2b10 0x7f7a1406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a1a200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a140c8420 con 0x7f7a141516b0 2024-05-27T00:15:36.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a1a200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7a140c1df0 con 0x7f7a140c2b10 2024-05-27T00:15:36.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a1a200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7a140c3000 con 0x7f7a140b1e10 2024-05-27T00:15:36.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a13e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7a140c2b10 0x7f7a1406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a13e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7a140c2b10 0x7f7a1406ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51196/0 (socket says 172.21.15.98:51196) 2024-05-27T00:15:36.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a13e00640 1 -- 172.21.15.98:0/4192815204 learned_addr learned my addr 172.21.15.98:0/4192815204 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:36.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a12a00640 1 --2- 172.21.15.98:0/4192815204 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a140b1e10 0x7f7a1406e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a13e00640 1 -- 172.21.15.98:0/4192815204 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a140b1e10 msgr2=0x7f7a1406e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:36.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a13e00640 1 --2- 172.21.15.98:0/4192815204 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a140b1e10 0x7f7a1406e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.929+0000 7f7a13e00640 1 -- 172.21.15.98:0/4192815204 >> v1:172.21.15.204:6789/0 conn(0x7f7a141516b0 legacy=0x7f7a140c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a13e00640 1 -- 172.21.15.98:0/4192815204 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a140c7e60 con 0x7f7a140c2b10 2024-05-27T00:15:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a13e00640 1 --2- 172.21.15.98:0/4192815204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140c2b10 0x7f7a1406ecc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009e80 tx=0x7f7a0c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9197b79738cfb788 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a12000640 1 -- 172.21.15.98:0/4192815204 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0a57d0 con 0x7f7a140c2b10 2024-05-27T00:15:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a12000640 1 -- 172.21.15.98:0/4192815204 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0c0a5970 con 0x7f7a140c2b10 2024-05-27T00:15:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a12000640 1 -- 172.21.15.98:0/4192815204 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0a5d40 con 0x7f7a140c2b10 2024-05-27T00:15:36.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a1a200640 1 -- 172.21.15.98:0/4192815204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140c2b10 msgr2=0x7f7a1406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a1a200640 1 --2- 172.21.15.98:0/4192815204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140c2b10 0x7f7a1406ecc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009e80 tx=0x7f7a0c01ca10 comp rx=0 tx=0).stop 2024-05-27T00:15:36.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a1a200640 1 -- 172.21.15.98:0/4192815204 shutdown_connections 2024-05-27T00:15:36.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a1a200640 1 --2- 172.21.15.98:0/4192815204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140c2b10 0x7f7a1406ecc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a1a200640 1 --2- 172.21.15.98:0/4192815204 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a140b1e10 0x7f7a1406e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a1a200640 1 -- 172.21.15.98:0/4192815204 >> 172.21.15.98:0/4192815204 conn(0x7f7a140cbf00 msgr2=0x7f7a140cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:36.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a1a200640 1 -- 172.21.15.98:0/4192815204 shutdown_connections 2024-05-27T00:15:36.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.933+0000 7f7a1a200640 1 -- 172.21.15.98:0/4192815204 wait complete. 2024-05-27T00:15:36.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a1a200640 1 Processor -- start 2024-05-27T00:15:36.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a1a200640 1 -- start start 2024-05-27T00:15:36.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140b1e10 0x7f7a140caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a140c2b10 0x7f7a140ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a141516b0 0x7f7a1414c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a1a200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a14151c10 con 0x7f7a140b1e10 2024-05-27T00:15:36.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a1a200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7a14151a90 con 0x7f7a140c2b10 2024-05-27T00:15:36.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a1a200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7a14164540 con 0x7f7a141516b0 2024-05-27T00:15:36.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140b1e10 0x7f7a140caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140b1e10 0x7f7a140caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51202/0 (socket says 172.21.15.98:51202) 2024-05-27T00:15:36.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13400640 1 -- 172.21.15.98:0/2045870798 learned_addr learned my addr 172.21.15.98:0/2045870798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:36.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a12a00640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a140c2b10 0x7f7a140ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13e00640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a141516b0 0x7f7a1414c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13400640 1 -- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a141516b0 msgr2=0x7f7a1414c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13400640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a141516b0 0x7f7a1414c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13400640 1 -- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a140c2b10 msgr2=0x7f7a140ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13400640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a140c2b10 0x7f7a140ca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13400640 1 -- 172.21.15.98:0/2045870798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a140cb940 con 0x7f7a140b1e10 2024-05-27T00:15:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13e00640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a141516b0 0x7f7a1414c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a13400640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140b1e10 0x7f7a140caba0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400c450 tx=0x7f7a0400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a0b400640 1 -- 172.21.15.98:0/2045870798 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a040052d0 con 0x7f7a140b1e10 2024-05-27T00:15:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a0b400640 1 -- 172.21.15.98:0/2045870798 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a04005470 con 0x7f7a140b1e10 2024-05-27T00:15:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a0b400640 1 -- 172.21.15.98:0/2045870798 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a04007260 con 0x7f7a140b1e10 2024-05-27T00:15:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a1a200640 1 -- 172.21.15.98:0/2045870798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a141f59d0 con 0x7f7a140b1e10 2024-05-27T00:15:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.937+0000 7f7a1a200640 1 -- 172.21.15.98:0/2045870798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a141f56f0 con 0x7f7a140b1e10 2024-05-27T00:15:36.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.941+0000 7f7a1a200640 1 -- 172.21.15.98:0/2045870798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a14075cb0 con 0x7f7a140b1e10 2024-05-27T00:15:36.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.941+0000 7f7a0b400640 1 -- 172.21.15.98:0/2045870798 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7a04007400 con 0x7f7a140b1e10 2024-05-27T00:15:36.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.941+0000 7f7a0b400640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79f40678c0 0x7f79f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:36.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.941+0000 7f7a0b400640 1 -- 172.21.15.98:0/2045870798 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f7a040850d0 con 0x7f7a140b1e10 2024-05-27T00:15:36.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.941+0000 7f7a12a00640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79f40678c0 0x7f79f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:36.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.945+0000 7f7a12a00640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79f40678c0 0x7f79f4069d80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7a14148b80 tx=0x7f7a00009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:36.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:36.945+0000 7f7a0b400640 1 -- 172.21.15.98:0/2045870798 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7a040555b0 con 0x7f7a140b1e10 2024-05-27T00:15:37.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 -- 172.21.15.98:0/2045870798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7a141488d0 con 0x7f7a140b1e10 2024-05-27T00:15:37.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a0b400640 1 -- 172.21.15.98:0/2045870798 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f7a040598b0 con 0x7f7a140b1e10 2024-05-27T00:15:37.104 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-27T00:15:37.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 -- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79f40678c0 msgr2=0x7f79f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79f40678c0 0x7f79f4069d80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7a14148b80 tx=0x7f7a00009190 comp rx=0 tx=0).stop 2024-05-27T00:15:37.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 -- 172.21.15.98:0/2045870798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140b1e10 msgr2=0x7f7a140caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140b1e10 0x7f7a140caba0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400c450 tx=0x7f7a0400c910 comp rx=0 tx=0).stop 2024-05-27T00:15:37.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 -- 172.21.15.98:0/2045870798 shutdown_connections 2024-05-27T00:15:37.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a141516b0 0x7f7a1414c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79f40678c0 0x7f79f4069d80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a140c2b10 0x7f7a140ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 --2- 172.21.15.98:0/2045870798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a140b1e10 0x7f7a140caba0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 -- 172.21.15.98:0/2045870798 >> 172.21.15.98:0/2045870798 conn(0x7f7a140cbf00 msgr2=0x7f7a140cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:37.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 -- 172.21.15.98:0/2045870798 shutdown_connections 2024-05-27T00:15:37.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.101+0000 7f7a1a200640 1 -- 172.21.15.98:0/2045870798 wait complete. 2024-05-27T00:15:37.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T00:15:37.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06c400640 1 Processor -- start 2024-05-27T00:15:37.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06c400640 1 -- start start 2024-05-27T00:15:37.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06c400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb0641510d0 0x7fb0640c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06c400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0640c8350 0x7fb06415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06c400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb06415af10 0x7fb06406cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06c400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb0640710a0 con 0x7fb06415af10 2024-05-27T00:15:37.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06c400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb0640c7f70 con 0x7fb0641510d0 2024-05-27T00:15:37.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06c400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb0641514b0 con 0x7fb0640c8350 2024-05-27T00:15:37.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06a400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb06415af10 0x7fb06406cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06a400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb06415af10 0x7fb06406cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51214/0 (socket says 172.21.15.98:51214) 2024-05-27T00:15:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb06a400640 1 -- 172.21.15.98:0/3092405715 learned_addr learned my addr 172.21.15.98:0/3092405715 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb069a00640 1 --2- 172.21.15.98:0/3092405715 >> v2:172.21.15.204:3300/0 conn(0x7fb0641510d0 0x7fb0640c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.261+0000 7fb069000640 1 --2- 172.21.15.98:0/3092405715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0640c8350 0x7fb06415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb06a400640 1 -- 172.21.15.98:0/3092405715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0640c8350 msgr2=0x7fb06415a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb06a400640 1 --2- 172.21.15.98:0/3092405715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0640c8350 0x7fb06415a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb06a400640 1 -- 172.21.15.98:0/3092405715 >> v2:172.21.15.204:3300/0 conn(0x7fb0641510d0 msgr2=0x7fb0640c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb06a400640 1 --2- 172.21.15.98:0/3092405715 >> v2:172.21.15.204:3300/0 conn(0x7fb0641510d0 0x7fb0640c7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb06a400640 1 -- 172.21.15.98:0/3092405715 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb064072760 con 0x7fb06415af10 2024-05-27T00:15:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb06a400640 1 --2- 172.21.15.98:0/3092405715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 0x7fb06406cfe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb0640c65c0 tx=0x7fb05c01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e849a4fbd4ba239a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb063e00640 1 -- 172.21.15.98:0/3092405715 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0b4070 con 0x7fb06415af10 2024-05-27T00:15:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb063e00640 1 -- 172.21.15.98:0/3092405715 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb05c0af400 con 0x7fb06415af10 2024-05-27T00:15:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb063e00640 1 -- 172.21.15.98:0/3092405715 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0af7d0 con 0x7fb06415af10 2024-05-27T00:15:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb06c400640 1 -- 172.21.15.98:0/3092405715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 msgr2=0x7fb06406cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.265+0000 7fb06c400640 1 --2- 172.21.15.98:0/3092405715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 0x7fb06406cfe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb0640c65c0 tx=0x7fb05c01f1e0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 -- 172.21.15.98:0/3092405715 shutdown_connections 2024-05-27T00:15:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 --2- 172.21.15.98:0/3092405715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 0x7fb06406cfe0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 --2- 172.21.15.98:0/3092405715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0640c8350 0x7fb06415a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 --2- 172.21.15.98:0/3092405715 >> v2:172.21.15.204:3300/0 conn(0x7fb0641510d0 0x7fb0640c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 -- 172.21.15.98:0/3092405715 >> 172.21.15.98:0/3092405715 conn(0x7fb0640cbf00 msgr2=0x7fb0640cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 -- 172.21.15.98:0/3092405715 shutdown_connections 2024-05-27T00:15:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 -- 172.21.15.98:0/3092405715 wait complete. 2024-05-27T00:15:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 Processor -- start 2024-05-27T00:15:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 -- start start 2024-05-27T00:15:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0640c8350 0x7fb0641f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0641510d0 0x7fb064140290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 0x7fb064141670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb064073000 con 0x7fb06415af10 2024-05-27T00:15:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb064072d20 con 0x7fb0641510d0 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb06415a4b0 con 0x7fb0640c8350 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06a400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 0x7fb064141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06a400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 0x7fb064141670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51222/0 (socket says 172.21.15.98:51222) 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06a400640 1 -- 172.21.15.98:0/1182516054 learned_addr learned my addr 172.21.15.98:0/1182516054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb069a00640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0640c8350 0x7fb0641f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb069000640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0641510d0 0x7fb064140290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06a400640 1 -- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0640c8350 msgr2=0x7fb0641f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06a400640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0640c8350 0x7fb0641f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06a400640 1 -- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0641510d0 msgr2=0x7fb064140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06a400640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0641510d0 0x7fb064140290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06a400640 1 -- 172.21.15.98:0/1182516054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0641434c0 con 0x7fb06415af10 2024-05-27T00:15:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06a400640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 0x7fb064141670 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb05c01d890 tx=0x7fb05c01bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb061600640 1 -- 172.21.15.98:0/1182516054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0b4040 con 0x7fb06415af10 2024-05-27T00:15:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb061600640 1 -- 172.21.15.98:0/1182516054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb05c0090d0 con 0x7fb06415af10 2024-05-27T00:15:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb06c400640 1 -- 172.21.15.98:0/1182516054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0641f0670 con 0x7fb06415af10 2024-05-27T00:15:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.269+0000 7fb061600640 1 -- 172.21.15.98:0/1182516054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c009510 con 0x7fb06415af10 2024-05-27T00:15:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.273+0000 7fb06c400640 1 -- 172.21.15.98:0/1182516054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0641f0c10 con 0x7fb06415af10 2024-05-27T00:15:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.273+0000 7fb06c400640 1 -- 172.21.15.98:0/1182516054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb064075cb0 con 0x7fb06415af10 2024-05-27T00:15:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.273+0000 7fb061600640 1 -- 172.21.15.98:0/1182516054 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb05c0c3020 con 0x7fb06415af10 2024-05-27T00:15:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.273+0000 7fb061600640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0440678c0 0x7fb044069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.273+0000 7fb061600640 1 -- 172.21.15.98:0/1182516054 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fb05c1236c0 con 0x7fb06415af10 2024-05-27T00:15:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.273+0000 7fb069a00640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0440678c0 0x7fb044069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.277+0000 7fb069a00640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0440678c0 0x7fb044069d80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb058009a70 tx=0x7fb058008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.277+0000 7fb061600640 1 -- 172.21.15.98:0/1182516054 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb05c0ca020 con 0x7fb06415af10 2024-05-27T00:15:37.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.433+0000 7fb06c400640 1 -- 172.21.15.98:0/1182516054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb0640c8130 con 0x7fb06415af10 2024-05-27T00:15:37.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.433+0000 7fb061600640 1 -- 172.21.15.98:0/1182516054 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7fb05c018d40 con 0x7fb06415af10 2024-05-27T00:15:37.437 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-27T00:15:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 -- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0440678c0 msgr2=0x7fb044069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0440678c0 0x7fb044069d80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb058009a70 tx=0x7fb058008040 comp rx=0 tx=0).stop 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 -- 172.21.15.98:0/1182516054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 msgr2=0x7fb064141670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 0x7fb064141670 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb05c01d890 tx=0x7fb05c01bf80 comp rx=0 tx=0).stop 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 -- 172.21.15.98:0/1182516054 shutdown_connections 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb06415af10 0x7fb064141670 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0440678c0 0x7fb044069d80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0641510d0 0x7fb064140290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 --2- 172.21.15.98:0/1182516054 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0640c8350 0x7fb0641f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 -- 172.21.15.98:0/1182516054 >> 172.21.15.98:0/1182516054 conn(0x7fb0640cbf00 msgr2=0x7fb0640c10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 -- 172.21.15.98:0/1182516054 shutdown_connections 2024-05-27T00:15:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.437+0000 7fb06c400640 1 -- 172.21.15.98:0/1182516054 wait complete. 2024-05-27T00:15:37.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:15:37.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.589+0000 7f16ae600640 1 Processor -- start 2024-05-27T00:15:37.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.589+0000 7f16ae600640 1 -- start start 2024-05-27T00:15:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16ae600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f16a80735d0 0x7f16a8151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16ae600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80c79f0 0x7f16a8151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16ae600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16a80c17d0 con 0x7f16a80731f0 2024-05-27T00:15:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16ae600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f16a8071990 con 0x7f16a80735d0 2024-05-27T00:15:37.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16ae600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f16a8070e10 con 0x7f16a80c79f0 2024-05-27T00:15:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f169ea00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f16a80735d0 0x7f16a8151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16a7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80c79f0 0x7f16a8151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f169ea00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f16a80735d0 0x7f16a8151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:39240/0 (socket says 172.21.15.98:39240) 2024-05-27T00:15:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f169ea00640 1 -- 172.21.15.98:0/4082231827 learned_addr learned my addr 172.21.15.98:0/4082231827 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16a7400640 1 --1- 172.21.15.98:0/4082231827 >> v1:172.21.15.204:6789/0 conn(0x7f16a80731f0 0x7f16a80c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:41940/0 (socket says 172.21.15.98:41940) 2024-05-27T00:15:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16a7e00640 1 -- 172.21.15.98:0/4082231827 >> v2:172.21.15.204:3300/0 conn(0x7f16a80735d0 msgr2=0x7f16a8151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16a7e00640 1 --2- 172.21.15.98:0/4082231827 >> v2:172.21.15.204:3300/0 conn(0x7f16a80735d0 0x7f16a8151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16a7e00640 1 -- 172.21.15.98:0/4082231827 >> v1:172.21.15.204:6789/0 conn(0x7f16a80731f0 legacy=0x7f16a80c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f16a7e00640 1 -- 172.21.15.98:0/4082231827 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a80710f0 con 0x7f16a80c79f0 2024-05-27T00:15:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.593+0000 7f169ea00640 1 --2- 172.21.15.98:0/4082231827 >> v2:172.21.15.204:3300/0 conn(0x7f16a80735d0 0x7f16a8151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:15:37.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16a7e00640 1 --2- 172.21.15.98:0/4082231827 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80c79f0 0x7f16a8151970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009d90 tx=0x7f16a001c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3761c62b8aac472e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16a6a00640 1 -- 172.21.15.98:0/4082231827 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a00b2070 con 0x7f16a80c79f0 2024-05-27T00:15:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16a6a00640 1 -- 172.21.15.98:0/4082231827 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a00a58b0 con 0x7f16a80c79f0 2024-05-27T00:15:37.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16a6a00640 1 -- 172.21.15.98:0/4082231827 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a00a5c80 con 0x7f16a80c79f0 2024-05-27T00:15:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16ae600640 1 -- 172.21.15.98:0/4082231827 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80c79f0 msgr2=0x7f16a8151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16ae600640 1 --2- 172.21.15.98:0/4082231827 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80c79f0 0x7f16a8151970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009d90 tx=0x7f16a001c890 comp rx=0 tx=0).stop 2024-05-27T00:15:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16ae600640 1 -- 172.21.15.98:0/4082231827 shutdown_connections 2024-05-27T00:15:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16ae600640 1 --2- 172.21.15.98:0/4082231827 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80c79f0 0x7f16a8151970 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16ae600640 1 --2- 172.21.15.98:0/4082231827 >> v2:172.21.15.204:3300/0 conn(0x7f16a80735d0 0x7f16a8151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16ae600640 1 -- 172.21.15.98:0/4082231827 >> 172.21.15.98:0/4082231827 conn(0x7f16a8067530 msgr2=0x7f16a806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:37.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16ae600640 1 -- 172.21.15.98:0/4082231827 shutdown_connections 2024-05-27T00:15:37.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16ae600640 1 -- 172.21.15.98:0/4082231827 wait complete. 2024-05-27T00:15:37.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16ae600640 1 Processor -- start 2024-05-27T00:15:37.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.597+0000 7f16ae600640 1 -- start start 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16ae600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80731f0 0x7f16a81546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16ae600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a80735d0 0x7f16a81ef450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16ae600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16a80c79f0 0x7f16a81ef990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16ae600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f16a80c8400 con 0x7f16a80c79f0 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16ae600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f16a80716b0 con 0x7f16a80735d0 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16ae600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f16a8157970 con 0x7f16a80731f0 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16a7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16a80c79f0 0x7f16a81ef990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16a7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16a80c79f0 0x7f16a81ef990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51226/0 (socket says 172.21.15.98:51226) 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16a7e00640 1 -- 172.21.15.98:0/3395331225 learned_addr learned my addr 172.21.15.98:0/3395331225 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16a7400640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80731f0 0x7f16a81546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f169ea00640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a80735d0 0x7f16a81ef450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16a7e00640 1 -- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80731f0 msgr2=0x7f16a81546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16a7e00640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80731f0 0x7f16a81546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16a7e00640 1 -- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a80735d0 msgr2=0x7f16a81ef450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16a7e00640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a80735d0 0x7f16a81ef450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16a7e00640 1 -- 172.21.15.98:0/3395331225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a81f00c0 con 0x7f16a80c79f0 2024-05-27T00:15:37.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16a7e00640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16a80c79f0 0x7f16a81ef990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f16a001d130 tx=0x7f16a0019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:37.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f169fe00640 1 -- 172.21.15.98:0/3395331225 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a00b2050 con 0x7f16a80c79f0 2024-05-27T00:15:37.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f169fe00640 1 -- 172.21.15.98:0/3395331225 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a00be070 con 0x7f16a80c79f0 2024-05-27T00:15:37.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16ae600640 1 -- 172.21.15.98:0/3395331225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a81e3190 con 0x7f16a80c79f0 2024-05-27T00:15:37.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f169fe00640 1 -- 172.21.15.98:0/3395331225 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0009450 con 0x7f16a80c79f0 2024-05-27T00:15:37.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16ae600640 1 -- 172.21.15.98:0/3395331225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a81e3750 con 0x7f16a80c79f0 2024-05-27T00:15:37.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.601+0000 7f16ae600640 1 -- 172.21.15.98:0/3395331225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a8075cb0 con 0x7f16a80c79f0 2024-05-27T00:15:37.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.605+0000 7f169fe00640 1 -- 172.21.15.98:0/3395331225 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f16a00c3460 con 0x7f16a80c79f0 2024-05-27T00:15:37.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.605+0000 7f169fe00640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16840678c0 0x7f1684069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.605+0000 7f169fe00640 1 -- 172.21.15.98:0/3395331225 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f16a0121150 con 0x7f16a80c79f0 2024-05-27T00:15:37.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.605+0000 7f16a7400640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16840678c0 0x7f1684069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.605+0000 7f16a7400640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16840678c0 0x7f1684069d80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1698000c10 tx=0x7f169800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.609+0000 7f169fe00640 1 -- 172.21.15.98:0/3395331225 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f16a0016480 con 0x7f16a80c79f0 2024-05-27T00:15:37.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.797+0000 7f16ae600640 1 -- 172.21.15.98:0/3395331225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16a8158510 con 0x7f16a80c79f0 2024-05-27T00:15:37.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.797+0000 7f169fe00640 1 -- 172.21.15.98:0/3395331225 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f16a00f1630 con 0x7f16a80c79f0 2024-05-27T00:15:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:37.800 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:15:37.801 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:15:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.797+0000 7f16ae600640 1 -- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16840678c0 msgr2=0x7f1684069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.797+0000 7f16ae600640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16840678c0 0x7f1684069d80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1698000c10 tx=0x7f169800b040 comp rx=0 tx=0).stop 2024-05-27T00:15:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.797+0000 7f16ae600640 1 -- 172.21.15.98:0/3395331225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16a80c79f0 msgr2=0x7f16a81ef990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.797+0000 7f16ae600640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16a80c79f0 0x7f16a81ef990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f16a001d130 tx=0x7f16a0019630 comp rx=0 tx=0).stop 2024-05-27T00:15:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.801+0000 7f16ae600640 1 -- 172.21.15.98:0/3395331225 shutdown_connections 2024-05-27T00:15:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.801+0000 7f16ae600640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16a80c79f0 0x7f16a81ef990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.801+0000 7f16ae600640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16840678c0 0x7f1684069d80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.801+0000 7f16ae600640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a80735d0 0x7f16a81ef450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.801+0000 7f16ae600640 1 --2- 172.21.15.98:0/3395331225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16a80731f0 0x7f16a81546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.801+0000 7f16ae600640 1 -- 172.21.15.98:0/3395331225 >> 172.21.15.98:0/3395331225 conn(0x7f16a8067530 msgr2=0x7f16a8157d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.801+0000 7f16ae600640 1 -- 172.21.15.98:0/3395331225 shutdown_connections 2024-05-27T00:15:37.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.801+0000 7f16ae600640 1 -- 172.21.15.98:0/3395331225 wait complete. 2024-05-27T00:15:37.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4614600640 1 Processor -- start 2024-05-27T00:15:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4614600640 1 -- start start 2024-05-27T00:15:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4614600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 0x7f460c158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4614600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f460c0b3ed0 0x7f460c158c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4614600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f460c15be40 con 0x7f460c152a50 2024-05-27T00:15:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4614600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f460c0c8a00 con 0x7f460c0b3ed0 2024-05-27T00:15:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4614600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f460c0c50e0 con 0x7f460c0cb7e0 2024-05-27T00:15:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4611200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 0x7f460c158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4611200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 0x7f460c158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:53480/0 (socket says 172.21.15.98:53480) 2024-05-27T00:15:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4611200640 1 -- 172.21.15.98:0/3874795951 learned_addr learned my addr 172.21.15.98:0/3874795951 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4612600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f460c0b3ed0 0x7f460c158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4611200640 1 -- 172.21.15.98:0/3874795951 >> v2:172.21.15.204:3300/0 conn(0x7f460c0b3ed0 msgr2=0x7f460c158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4611200640 1 --2- 172.21.15.98:0/3874795951 >> v2:172.21.15.204:3300/0 conn(0x7f460c0b3ed0 0x7f460c158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4611200640 1 -- 172.21.15.98:0/3874795951 >> v1:172.21.15.204:6789/0 conn(0x7f460c152a50 legacy=0x7f460c0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.953+0000 7f4611200640 1 -- 172.21.15.98:0/3874795951 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460c0c8c50 con 0x7f460c0cb7e0 2024-05-27T00:15:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f4611200640 1 --2- 172.21.15.98:0/3874795951 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 0x7f460c158750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f45f8009870 tx=0x7f45f801c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95b7c1284ef7f453 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f460be00640 1 -- 172.21.15.98:0/3874795951 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f80b0070 con 0x7f460c0cb7e0 2024-05-27T00:15:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f460be00640 1 -- 172.21.15.98:0/3874795951 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45f80ab600 con 0x7f460c0cb7e0 2024-05-27T00:15:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f460be00640 1 -- 172.21.15.98:0/3874795951 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f80ab9d0 con 0x7f460c0cb7e0 2024-05-27T00:15:37.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f4614600640 1 -- 172.21.15.98:0/3874795951 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 msgr2=0x7f460c158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f4614600640 1 --2- 172.21.15.98:0/3874795951 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 0x7f460c158750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f45f8009870 tx=0x7f45f801c330 comp rx=0 tx=0).stop 2024-05-27T00:15:37.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f4614600640 1 -- 172.21.15.98:0/3874795951 shutdown_connections 2024-05-27T00:15:37.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f4614600640 1 --2- 172.21.15.98:0/3874795951 >> v2:172.21.15.204:3300/0 conn(0x7f460c0b3ed0 0x7f460c158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f4614600640 1 --2- 172.21.15.98:0/3874795951 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 0x7f460c158750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f4614600640 1 -- 172.21.15.98:0/3874795951 >> 172.21.15.98:0/3874795951 conn(0x7f460c072f90 msgr2=0x7f460c0733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f4614600640 1 -- 172.21.15.98:0/3874795951 shutdown_connections 2024-05-27T00:15:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.957+0000 7f4614600640 1 -- 172.21.15.98:0/3874795951 wait complete. 2024-05-27T00:15:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4614600640 1 Processor -- start 2024-05-27T00:15:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4614600640 1 -- start start 2024-05-27T00:15:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4614600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f460c0b3ed0 0x7f460c0c5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4614600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 0x7f460c0ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4614600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460c152a50 0x7f460c0709a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4614600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f460c152e30 con 0x7f460c152a50 2024-05-27T00:15:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4614600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f460c0cb570 con 0x7f460c0b3ed0 2024-05-27T00:15:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4614600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f460c0c9c10 con 0x7f460c0cb7e0 2024-05-27T00:15:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4612600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460c152a50 0x7f460c0709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4611c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f460c0b3ed0 0x7f460c0c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4612600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460c152a50 0x7f460c0709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51228/0 (socket says 172.21.15.98:51228) 2024-05-27T00:15:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4612600640 1 -- 172.21.15.98:0/3712031182 learned_addr learned my addr 172.21.15.98:0/3712031182 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4611200640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 0x7f460c0ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4612600640 1 -- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 msgr2=0x7f460c0ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4612600640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 0x7f460c0ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4612600640 1 -- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f460c0b3ed0 msgr2=0x7f460c0c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:37.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4612600640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f460c0b3ed0 0x7f460c0c5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:37.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4612600640 1 -- 172.21.15.98:0/3712031182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460c1f3070 con 0x7f460c152a50 2024-05-27T00:15:37.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4612600640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460c152a50 0x7f460c0709a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4604009dd0 tx=0x7f460400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:37.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4609600640 1 -- 172.21.15.98:0/3712031182 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4604018020 con 0x7f460c152a50 2024-05-27T00:15:37.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4609600640 1 -- 172.21.15.98:0/3712031182 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4604014480 con 0x7f460c152a50 2024-05-27T00:15:37.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4614600640 1 -- 172.21.15.98:0/3712031182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f460c1f3350 con 0x7f460c152a50 2024-05-27T00:15:37.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4609600640 1 -- 172.21.15.98:0/3712031182 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4604014870 con 0x7f460c152a50 2024-05-27T00:15:37.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.961+0000 7f4614600640 1 -- 172.21.15.98:0/3712031182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f460c1f3910 con 0x7f460c152a50 2024-05-27T00:15:37.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.965+0000 7f4614600640 1 -- 172.21.15.98:0/3712031182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f460c075cb0 con 0x7f460c152a50 2024-05-27T00:15:37.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.965+0000 7f4609600640 1 -- 172.21.15.98:0/3712031182 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f460401b020 con 0x7f460c152a50 2024-05-27T00:15:37.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.965+0000 7f4609600640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45ec0678c0 0x7f45ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:37.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.965+0000 7f4609600640 1 -- 172.21.15.98:0/3712031182 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f46040867a0 con 0x7f460c152a50 2024-05-27T00:15:37.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.965+0000 7f4611c00640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45ec0678c0 0x7f45ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:37.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.965+0000 7f4611c00640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45ec0678c0 0x7f45ec069d80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4600007fd0 tx=0x7f4600009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:37.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:37.969+0000 7f4609600640 1 -- 172.21.15.98:0/3712031182 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4604022030 con 0x7f460c152a50 2024-05-27T00:15:38.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.125+0000 7f4614600640 1 -- 172.21.15.98:0/3712031182 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f460c073510 con 0x7f460c152a50 2024-05-27T00:15:38.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.125+0000 7f4609600640 1 -- 172.21.15.98:0/3712031182 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16581 (secure 0 0 0) 0x7f4604010040 con 0x7f460c152a50 2024-05-27T00:15:38.128 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:38.129 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:26.579229+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:38.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 -- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45ec0678c0 msgr2=0x7f45ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45ec0678c0 0x7f45ec069d80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4600007fd0 tx=0x7f4600009040 comp rx=0 tx=0).stop 2024-05-27T00:15:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 -- 172.21.15.98:0/3712031182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460c152a50 msgr2=0x7f460c0709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460c152a50 0x7f460c0709a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4604009dd0 tx=0x7f460400c490 comp rx=0 tx=0).stop 2024-05-27T00:15:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 -- 172.21.15.98:0/3712031182 shutdown_connections 2024-05-27T00:15:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f460c152a50 0x7f460c0709a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f460c0cb7e0 0x7f460c0ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45ec0678c0 0x7f45ec069d80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 --2- 172.21.15.98:0/3712031182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f460c0b3ed0 0x7f460c0c5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 -- 172.21.15.98:0/3712031182 >> 172.21.15.98:0/3712031182 conn(0x7f460c072f90 msgr2=0x7f460c1624e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 -- 172.21.15.98:0/3712031182 shutdown_connections 2024-05-27T00:15:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.129+0000 7f4614600640 1 -- 172.21.15.98:0/3712031182 wait complete. 2024-05-27T00:15:38.150 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T00:15:13.211703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T00:15:23.224464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T00:15:23.950896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T00:15:38.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T00:15:38.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.285+0000 7fdf91600640 1 Processor -- start 2024-05-27T00:15:38.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.285+0000 7fdf91600640 1 -- start start 2024-05-27T00:15:38.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.285+0000 7fdf91600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf8c1516b0 0x7fdf8c0c1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.285+0000 7fdf91600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf8c0b1e10 0x7fdf8c06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf91600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf8c0c8420 con 0x7fdf8c0c2b10 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf91600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdf8c0c1df0 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf91600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdf8c0c3000 con 0x7fdf8c1516b0 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf8a000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf8c0b1e10 0x7fdf8c06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf8a000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf8c0b1e10 0x7fdf8c06e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51244/0 (socket says 172.21.15.98:51244) 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf8a000640 1 -- 172.21.15.98:0/3468723899 learned_addr learned my addr 172.21.15.98:0/3468723899 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf8aa00640 1 --2- 172.21.15.98:0/3468723899 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf8c1516b0 0x7fdf8c0c1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf8a000640 1 -- 172.21.15.98:0/3468723899 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf8c1516b0 msgr2=0x7fdf8c0c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf8a000640 1 --2- 172.21.15.98:0/3468723899 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf8c1516b0 0x7fdf8c0c1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf8a000640 1 -- 172.21.15.98:0/3468723899 >> v1:172.21.15.204:6789/0 conn(0x7fdf8c0c2b10 legacy=0x7fdf8c15a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf8a000640 1 -- 172.21.15.98:0/3468723899 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8c0c7e60 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf8a000640 1 --2- 172.21.15.98:0/3468723899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 0x7fdf8c06e780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdf780098a0 tx=0x7fdf7801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815f29e617f51ebb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:38.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf89600640 1 -- 172.21.15.98:0/3468723899 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780ab430 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf89600640 1 -- 172.21.15.98:0/3468723899 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf780b0070 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.289+0000 7fdf89600640 1 -- 172.21.15.98:0/3468723899 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780ab7f0 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 -- 172.21.15.98:0/3468723899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 msgr2=0x7fdf8c06e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 --2- 172.21.15.98:0/3468723899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 0x7fdf8c06e780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdf780098a0 tx=0x7fdf7801c540 comp rx=0 tx=0).stop 2024-05-27T00:15:38.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 -- 172.21.15.98:0/3468723899 shutdown_connections 2024-05-27T00:15:38.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 --2- 172.21.15.98:0/3468723899 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 0x7fdf8c06e780 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 --2- 172.21.15.98:0/3468723899 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf8c1516b0 0x7fdf8c0c1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 -- 172.21.15.98:0/3468723899 >> 172.21.15.98:0/3468723899 conn(0x7fdf8c0cbf00 msgr2=0x7fdf8c0cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:38.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 -- 172.21.15.98:0/3468723899 shutdown_connections 2024-05-27T00:15:38.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 -- 172.21.15.98:0/3468723899 wait complete. 2024-05-27T00:15:38.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 Processor -- start 2024-05-27T00:15:38.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 -- start start 2024-05-27T00:15:38.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 0x7fdf8c0af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf8c0c2b10 0x7fdf8c143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf8c1516b0 0x7fdf8c0c5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8c073030 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8c072eb0 con 0x7fdf8c1516b0 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf91600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdf8c151a90 con 0x7fdf8c0c2b10 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 0x7fdf8c0af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 0x7fdf8c0af710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51250/0 (socket says 172.21.15.98:51250) 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8aa00640 1 -- 172.21.15.98:0/2191012380 learned_addr learned my addr 172.21.15.98:0/2191012380 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8a000640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf8c0c2b10 0x7fdf8c143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8b400640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf8c1516b0 0x7fdf8c0c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8aa00640 1 -- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf8c0c2b10 msgr2=0x7fdf8c143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8aa00640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf8c0c2b10 0x7fdf8c143a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8aa00640 1 -- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf8c1516b0 msgr2=0x7fdf8c0c5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8aa00640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf8c1516b0 0x7fdf8c0c5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8aa00640 1 -- 172.21.15.98:0/2191012380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8c145770 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.293+0000 7fdf8b400640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf8c1516b0 0x7fdf8c0c5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf8aa00640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 0x7fdf8c0af710 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c007ec0 tx=0x7fdf7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf82a00640 1 -- 172.21.15.98:0/2191012380 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c014620 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf82a00640 1 -- 172.21.15.98:0/2191012380 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf7c019070 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf91600640 1 -- 172.21.15.98:0/2191012380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf8c1f3ef0 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf82a00640 1 -- 172.21.15.98:0/2191012380 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c014a20 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf91600640 1 -- 172.21.15.98:0/2191012380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf8c1f44b0 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf91600640 1 -- 172.21.15.98:0/2191012380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf8c075cb0 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf82a00640 1 -- 172.21.15.98:0/2191012380 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf7c014bc0 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf82a00640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf6c0678c0 0x7fdf6c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf82a00640 1 -- 172.21.15.98:0/2191012380 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fdf7c086550 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.297+0000 7fdf8a000640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf6c0678c0 0x7fdf6c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.301+0000 7fdf8a000640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf6c0678c0 0x7fdf6c069d80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf780af000 tx=0x7fdf78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:38.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.301+0000 7fdf82a00640 1 -- 172.21.15.98:0/2191012380 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fdf7c057090 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.457+0000 7fdf91600640 1 -- 172.21.15.98:0/2191012380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdf8c150dd0 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.457+0000 7fdf82a00640 1 -- 172.21.15.98:0/2191012380 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fdf7c056990 con 0x7fdf8c0b1e10 2024-05-27T00:15:38.461 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-27T00:15:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 -- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf6c0678c0 msgr2=0x7fdf6c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf6c0678c0 0x7fdf6c069d80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf780af000 tx=0x7fdf78008040 comp rx=0 tx=0).stop 2024-05-27T00:15:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 -- 172.21.15.98:0/2191012380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 msgr2=0x7fdf8c0af710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 0x7fdf8c0af710 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c007ec0 tx=0x7fdf7c00e4b0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 -- 172.21.15.98:0/2191012380 shutdown_connections 2024-05-27T00:15:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf6c0678c0 0x7fdf6c069d80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf8c1516b0 0x7fdf8c0c5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf8c0c2b10 0x7fdf8c143a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 --2- 172.21.15.98:0/2191012380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf8c0b1e10 0x7fdf8c0af710 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 -- 172.21.15.98:0/2191012380 >> 172.21.15.98:0/2191012380 conn(0x7fdf8c0cbf00 msgr2=0x7fdf8c0cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 -- 172.21.15.98:0/2191012380 shutdown_connections 2024-05-27T00:15:38.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.461+0000 7fdf91600640 1 -- 172.21.15.98:0/2191012380 wait complete. 2024-05-27T00:15:38.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T00:15:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.617+0000 7fb386e00640 1 Processor -- start 2024-05-27T00:15:38.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.617+0000 7fb386e00640 1 -- start start 2024-05-27T00:15:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb386e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3800b1e10 0x7fb38006e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb386e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb3800c2b10 0x7fb38006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb386e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3800c8420 con 0x7fb3801516b0 2024-05-27T00:15:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb386e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb3800c1df0 con 0x7fb3800b1e10 2024-05-27T00:15:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb386e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb3800c3000 con 0x7fb3800c2b10 2024-05-27T00:15:38.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb37f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3800b1e10 0x7fb38006e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb37f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3800b1e10 0x7fb38006e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51266/0 (socket says 172.21.15.98:51266) 2024-05-27T00:15:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb37f400640 1 -- 172.21.15.98:0/2706010844 learned_addr learned my addr 172.21.15.98:0/2706010844 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb384e00640 1 --2- 172.21.15.98:0/2706010844 >> v2:172.21.15.204:3300/0 conn(0x7fb3800c2b10 0x7fb38006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb37f400640 1 -- 172.21.15.98:0/2706010844 >> v2:172.21.15.204:3300/0 conn(0x7fb3800c2b10 msgr2=0x7fb38006ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb37f400640 1 --2- 172.21.15.98:0/2706010844 >> v2:172.21.15.204:3300/0 conn(0x7fb3800c2b10 0x7fb38006ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb37f400640 1 -- 172.21.15.98:0/2706010844 >> v1:172.21.15.204:6789/0 conn(0x7fb3801516b0 legacy=0x7fb3800c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.621+0000 7fb37f400640 1 -- 172.21.15.98:0/2706010844 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3800c7e60 con 0x7fb3800b1e10 2024-05-27T00:15:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb37f400640 1 --2- 172.21.15.98:0/2706010844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800b1e10 0x7fb38006e780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb380158590 tx=0x7fb36c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe0a02ae8e82608 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb37ea00640 1 -- 172.21.15.98:0/2706010844 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c0ab610 con 0x7fb3800b1e10 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb37ea00640 1 -- 172.21.15.98:0/2706010844 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb36c0b0070 con 0x7fb3800b1e10 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb37ea00640 1 -- 172.21.15.98:0/2706010844 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c0ab9d0 con 0x7fb3800b1e10 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb386e00640 1 -- 172.21.15.98:0/2706010844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800b1e10 msgr2=0x7fb38006e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb386e00640 1 --2- 172.21.15.98:0/2706010844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800b1e10 0x7fb38006e780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb380158590 tx=0x7fb36c01c520 comp rx=0 tx=0).stop 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb386e00640 1 -- 172.21.15.98:0/2706010844 shutdown_connections 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb386e00640 1 --2- 172.21.15.98:0/2706010844 >> v2:172.21.15.204:3300/0 conn(0x7fb3800c2b10 0x7fb38006ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb386e00640 1 --2- 172.21.15.98:0/2706010844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800b1e10 0x7fb38006e780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb386e00640 1 -- 172.21.15.98:0/2706010844 >> 172.21.15.98:0/2706010844 conn(0x7fb3800cbf00 msgr2=0x7fb3800cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb386e00640 1 -- 172.21.15.98:0/2706010844 shutdown_connections 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb386e00640 1 -- 172.21.15.98:0/2706010844 wait complete. 2024-05-27T00:15:38.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb386e00640 1 Processor -- start 2024-05-27T00:15:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.625+0000 7fb386e00640 1 -- start start 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb386e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3800b1e10 0x7fb380143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb386e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800c2b10 0x7fb3800cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb386e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3801516b0 0x7fb3800cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb386e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb380151c10 con 0x7fb3800c2b10 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb386e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb380151a90 con 0x7fb3800b1e10 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb386e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb380162530 con 0x7fb3801516b0 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb37f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800c2b10 0x7fb3800cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb37f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800c2b10 0x7fb3800cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51276/0 (socket says 172.21.15.98:51276) 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb37f400640 1 -- 172.21.15.98:0/665277478 learned_addr learned my addr 172.21.15.98:0/665277478 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb37fe00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3800b1e10 0x7fb380143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb384e00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3801516b0 0x7fb3800cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb37f400640 1 -- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3801516b0 msgr2=0x7fb3800cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb37f400640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3801516b0 0x7fb3800cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb37f400640 1 -- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3800b1e10 msgr2=0x7fb380143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb37f400640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3800b1e10 0x7fb380143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb37f400640 1 -- 172.21.15.98:0/665277478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3801f7540 con 0x7fb3800c2b10 2024-05-27T00:15:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb37f400640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800c2b10 0x7fb3800cb0f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0ad040 tx=0x7fb36c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb377e00640 1 -- 172.21.15.98:0/665277478 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c0be5f0 con 0x7fb3800c2b10 2024-05-27T00:15:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb377e00640 1 -- 172.21.15.98:0/665277478 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb36c0be790 con 0x7fb3800c2b10 2024-05-27T00:15:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb377e00640 1 -- 172.21.15.98:0/665277478 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c0beb40 con 0x7fb3800c2b10 2024-05-27T00:15:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb386e00640 1 -- 172.21.15.98:0/665277478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3801f7820 con 0x7fb3800c2b10 2024-05-27T00:15:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb386e00640 1 -- 172.21.15.98:0/665277478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3801f7de0 con 0x7fb3800c2b10 2024-05-27T00:15:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb386e00640 1 -- 172.21.15.98:0/665277478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb380075cb0 con 0x7fb3800c2b10 2024-05-27T00:15:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.629+0000 7fb377e00640 1 -- 172.21.15.98:0/665277478 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb36c0b43f0 con 0x7fb3800c2b10 2024-05-27T00:15:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.633+0000 7fb377e00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3600678f0 0x7fb360069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.633+0000 7fb377e00640 1 -- 172.21.15.98:0/665277478 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fb36c11ffa0 con 0x7fb3800c2b10 2024-05-27T00:15:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.633+0000 7fb37fe00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3600678f0 0x7fb360069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.633+0000 7fb37fe00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3600678f0 0x7fb360069db0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb3700015b0 tx=0x7fb370009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:38.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.637+0000 7fb377e00640 1 -- 172.21.15.98:0/665277478 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb36c0a8050 con 0x7fb3800c2b10 2024-05-27T00:15:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 -- 172.21.15.98:0/665277478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb38006e550 con 0x7fb3800c2b10 2024-05-27T00:15:38.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb377e00640 1 -- 172.21.15.98:0/665277478 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fb36c016110 con 0x7fb3800c2b10 2024-05-27T00:15:38.795 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-27T00:15:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 -- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3600678f0 msgr2=0x7fb360069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3600678f0 0x7fb360069db0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb3700015b0 tx=0x7fb370009040 comp rx=0 tx=0).stop 2024-05-27T00:15:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 -- 172.21.15.98:0/665277478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800c2b10 msgr2=0x7fb3800cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800c2b10 0x7fb3800cb0f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0ad040 tx=0x7fb36c01ad60 comp rx=0 tx=0).stop 2024-05-27T00:15:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 -- 172.21.15.98:0/665277478 shutdown_connections 2024-05-27T00:15:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3801516b0 0x7fb3800cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3800c2b10 0x7fb3800cb0f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3600678f0 0x7fb360069db0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 --2- 172.21.15.98:0/665277478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3800b1e10 0x7fb380143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 -- 172.21.15.98:0/665277478 >> 172.21.15.98:0/665277478 conn(0x7fb3800cbf00 msgr2=0x7fb38014e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:38.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 -- 172.21.15.98:0/665277478 shutdown_connections 2024-05-27T00:15:38.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.793+0000 7fb386e00640 1 -- 172.21.15.98:0/665277478 wait complete. 2024-05-27T00:15:38.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T00:15:38.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f850a400640 1 Processor -- start 2024-05-27T00:15:38.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f850a400640 1 -- start start 2024-05-27T00:15:38.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f850a400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8504154b00 0x7f85040c1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f850a400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f850415e090 0x7f850415e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f850a400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f850406f290 con 0x7f85040c82a0 2024-05-27T00:15:38.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f850a400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8504154ee0 con 0x7f8504154b00 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f850a400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f850415a8f0 con 0x7f850415e090 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f8503400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8504154b00 0x7f85040c1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f8503400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8504154b00 0x7f85040c1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51278/0 (socket says 172.21.15.98:51278) 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f8503400640 1 -- 172.21.15.98:0/257402804 learned_addr learned my addr 172.21.15.98:0/257402804 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f8502a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f85040c82a0 0x7f85040c6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47136/0 (socket says 172.21.15.98:47136) 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f8503e00640 1 --2- 172.21.15.98:0/257402804 >> v2:172.21.15.204:3300/0 conn(0x7f850415e090 0x7f850415e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f8503400640 1 -- 172.21.15.98:0/257402804 >> v2:172.21.15.204:3300/0 conn(0x7f850415e090 msgr2=0x7f850415e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f8503400640 1 --2- 172.21.15.98:0/257402804 >> v2:172.21.15.204:3300/0 conn(0x7f850415e090 0x7f850415e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f8503400640 1 -- 172.21.15.98:0/257402804 >> v1:172.21.15.98:6789/0 conn(0x7f85040c82a0 legacy=0x7f85040c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f8503400640 1 -- 172.21.15.98:0/257402804 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850415ea00 con 0x7f8504154b00 2024-05-27T00:15:38.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.953+0000 7f8502000640 1 -- 172.21.15.98:0/257402804 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2902746078 0 0) 0x7f84f0001120 con 0x7f85040c82a0 2024-05-27T00:15:38.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f8503400640 1 --2- 172.21.15.98:0/257402804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 0x7f85040c1260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f85040c11d0 tx=0x7f84f401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50dca942ca678fe9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:38.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f8502000640 1 -- 172.21.15.98:0/257402804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f40b0070 con 0x7f8504154b00 2024-05-27T00:15:38.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f8502000640 1 -- 172.21.15.98:0/257402804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f40ab600 con 0x7f8504154b00 2024-05-27T00:15:38.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f8502000640 1 -- 172.21.15.98:0/257402804 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f40ab9d0 con 0x7f8504154b00 2024-05-27T00:15:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f850a400640 1 -- 172.21.15.98:0/257402804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 msgr2=0x7f85040c1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f850a400640 1 --2- 172.21.15.98:0/257402804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 0x7f85040c1260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f85040c11d0 tx=0x7f84f401c520 comp rx=0 tx=0).stop 2024-05-27T00:15:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f850a400640 1 -- 172.21.15.98:0/257402804 shutdown_connections 2024-05-27T00:15:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f850a400640 1 --2- 172.21.15.98:0/257402804 >> v2:172.21.15.204:3300/0 conn(0x7f850415e090 0x7f850415e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f850a400640 1 --2- 172.21.15.98:0/257402804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 0x7f85040c1260 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f850a400640 1 -- 172.21.15.98:0/257402804 >> 172.21.15.98:0/257402804 conn(0x7f85040cbf00 msgr2=0x7f85040cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f850a400640 1 -- 172.21.15.98:0/257402804 shutdown_connections 2024-05-27T00:15:38.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f850a400640 1 -- 172.21.15.98:0/257402804 wait complete. 2024-05-27T00:15:38.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f850a400640 1 Processor -- start 2024-05-27T00:15:38.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.957+0000 7f850a400640 1 -- start start 2024-05-27T00:15:38.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f850a400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85040c82a0 0x7f850415bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f850a400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 0x7f85040cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f850a400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f850415e090 0x7f85040c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f850a400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85040c17a0 con 0x7f8504154b00 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f850a400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f850415b430 con 0x7f850415e090 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f850a400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f85040c1920 con 0x7f85040c82a0 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8502a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 0x7f85040cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8502a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 0x7f85040cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51292/0 (socket says 172.21.15.98:51292) 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8502a00640 1 -- 172.21.15.98:0/1825097158 learned_addr learned my addr 172.21.15.98:0/1825097158 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8503400640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85040c82a0 0x7f850415bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8503e00640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f850415e090 0x7f85040c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8502a00640 1 -- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85040c82a0 msgr2=0x7f850415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8502a00640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85040c82a0 0x7f850415bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8502a00640 1 -- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f850415e090 msgr2=0x7f85040c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8502a00640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f850415e090 0x7f85040c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:38.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8502a00640 1 -- 172.21.15.98:0/1825097158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850415da70 con 0x7f8504154b00 2024-05-27T00:15:38.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f8502a00640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 0x7f85040cb440 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f84f000c4b0 tx=0x7f84f000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:38.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f84fb400640 1 -- 172.21.15.98:0/1825097158 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0018070 con 0x7f8504154b00 2024-05-27T00:15:38.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f84fb400640 1 -- 172.21.15.98:0/1825097158 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f0007500 con 0x7f8504154b00 2024-05-27T00:15:38.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f84fb400640 1 -- 172.21.15.98:0/1825097158 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0004a30 con 0x7f8504154b00 2024-05-27T00:15:38.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f850a400640 1 -- 172.21.15.98:0/1825097158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850414a1c0 con 0x7f8504154b00 2024-05-27T00:15:38.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f850a400640 1 -- 172.21.15.98:0/1825097158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85040c91a0 con 0x7f8504154b00 2024-05-27T00:15:38.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.961+0000 7f850a400640 1 -- 172.21.15.98:0/1825097158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8504075cb0 con 0x7f8504154b00 2024-05-27T00:15:38.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.965+0000 7f84fb400640 1 -- 172.21.15.98:0/1825097158 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f84f001b070 con 0x7f8504154b00 2024-05-27T00:15:38.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.965+0000 7f84fb400640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84e40678c0 0x7f84e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:38.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.965+0000 7f84fb400640 1 -- 172.21.15.98:0/1825097158 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f84f0084ee0 con 0x7f8504154b00 2024-05-27T00:15:38.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.965+0000 7f8503400640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84e40678c0 0x7f84e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:38.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.965+0000 7f8503400640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84e40678c0 0x7f84e4069d80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f84f40ad000 tx=0x7f84f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:38.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:38.969+0000 7f84fb400640 1 -- 172.21.15.98:0/1825097158 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f850406f290 con 0x7f8504154b00 2024-05-27T00:15:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.121+0000 7f850a400640 1 -- 172.21.15.98:0/1825097158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f850415b910 con 0x7f8504154b00 2024-05-27T00:15:39.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f84fb400640 1 -- 172.21.15.98:0/1825097158 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f84f0055330 con 0x7f8504154b00 2024-05-27T00:15:39.129 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-27T00:15:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 -- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84e40678c0 msgr2=0x7f84e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84e40678c0 0x7f84e4069d80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f84f40ad000 tx=0x7f84f4008040 comp rx=0 tx=0).stop 2024-05-27T00:15:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 -- 172.21.15.98:0/1825097158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 msgr2=0x7f85040cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 0x7f85040cb440 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f84f000c4b0 tx=0x7f84f000c970 comp rx=0 tx=0).stop 2024-05-27T00:15:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 -- 172.21.15.98:0/1825097158 shutdown_connections 2024-05-27T00:15:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84e40678c0 0x7f84e4069d80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f850415e090 0x7f85040c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8504154b00 0x7f85040cb440 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 --2- 172.21.15.98:0/1825097158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85040c82a0 0x7f850415bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 -- 172.21.15.98:0/1825097158 >> 172.21.15.98:0/1825097158 conn(0x7f85040cbf00 msgr2=0x7f850414b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 -- 172.21.15.98:0/1825097158 shutdown_connections 2024-05-27T00:15:39.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.125+0000 7f850a400640 1 -- 172.21.15.98:0/1825097158 wait complete. 2024-05-27T00:15:39.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:15:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6df0e00640 1 Processor -- start 2024-05-27T00:15:39.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6df0e00640 1 -- start start 2024-05-27T00:15:39.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6df0e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f6dec06c790 0x7f6dec1591d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6df0e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dec0677a0 con 0x7f6dec152be0 2024-05-27T00:15:39.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6df0e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dec067a80 con 0x7f6dec067130 2024-05-27T00:15:39.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6df0e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f6dec159710 con 0x7f6dec06c790 2024-05-27T00:15:39.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de9e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6dec152be0 0x7f6dec0c79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47146/0 (socket says 172.21.15.98:47146) 2024-05-27T00:15:39.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de9e00640 1 -- 172.21.15.98:0/2512450408 learned_addr learned my addr 172.21.15.98:0/2512450408 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:39.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6dea800640 1 --2- 172.21.15.98:0/2512450408 >> v2:172.21.15.204:3300/0 conn(0x7f6dec06c790 0x7f6dec1591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1629471993 0 0) 0x7f6dec0677a0 con 0x7f6dec152be0 2024-05-27T00:15:39.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dd0003530 con 0x7f6dec152be0 2024-05-27T00:15:39.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1903783974 0 0) 0x7f6dd0003530 con 0x7f6dec152be0 2024-05-27T00:15:39.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 >> v2:172.21.15.204:3300/0 conn(0x7f6dec06c790 msgr2=0x7f6dec1591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:39.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 --2- 172.21.15.98:0/2512450408 >> v2:172.21.15.204:3300/0 conn(0x7f6dec06c790 0x7f6dec1591d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 >> v1:172.21.15.204:6789/0 conn(0x7f6dec067130 legacy=0x7f6dec06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec06b5a0 con 0x7f6dec152be0 2024-05-27T00:15:39.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f6ddc001680 con 0x7f6dec152be0 2024-05-27T00:15:39.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 >> v1:172.21.15.98:6789/0 conn(0x7f6dec152be0 legacy=0x7f6dec0c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dd00039c0 0x7f6dd0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dd0004810 0x7f6dd0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dd00052f0 0x7f6dd000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec159710 con 0x7f6dd0004810 2024-05-27T00:15:39.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd000c1c0 con 0x7f6dd00039c0 2024-05-27T00:15:39.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd000c3c0 con 0x7f6dd00052f0 2024-05-27T00:15:39.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de9400640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dd0004810 0x7f6dd0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de9e00640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dd00039c0 0x7f6dd0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6dea800640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dd00052f0 0x7f6dd000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de9400640 1 -- 172.21.15.98:0/2512450408 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dd00052f0 msgr2=0x7f6dd000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:39.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de9400640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dd00052f0 0x7f6dd000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de9400640 1 -- 172.21.15.98:0/2512450408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dd00039c0 msgr2=0x7f6dd0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:39.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de9400640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dd00039c0 0x7f6dd0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6de9400640 1 -- 172.21.15.98:0/2512450408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd0003530 con 0x7f6dd0004810 2024-05-27T00:15:39.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.293+0000 7f6dea800640 1 -- 172.21.15.98:0/2512450408 reap_dead start 2024-05-27T00:15:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6de9400640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dd0004810 0x7f6dd0004c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8001750 tx=0x7f6dd801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744d578f322b7d86 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:39.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd80013a0 con 0x7f6dd0004810 2024-05-27T00:15:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd80052f0 con 0x7f6dd0004810 2024-05-27T00:15:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6de8a00640 1 -- 172.21.15.98:0/2512450408 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8003840 con 0x7f6dd0004810 2024-05-27T00:15:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6df0e00640 1 -- 172.21.15.98:0/2512450408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dd0004810 msgr2=0x7f6dd0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6df0e00640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dd0004810 0x7f6dd0004c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8001750 tx=0x7f6dd801c3a0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6df0e00640 1 -- 172.21.15.98:0/2512450408 shutdown_connections 2024-05-27T00:15:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6df0e00640 1 --2- 172.21.15.98:0/2512450408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dd0004810 0x7f6dd0004c90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6df0e00640 1 -- 172.21.15.98:0/2512450408 >> 172.21.15.98:0/2512450408 conn(0x7f6dec0c9af0 msgr2=0x7f6dec0c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6df0e00640 1 -- 172.21.15.98:0/2512450408 shutdown_connections 2024-05-27T00:15:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.297+0000 7f6df0e00640 1 -- 172.21.15.98:0/2512450408 wait complete. 2024-05-27T00:15:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 Processor -- start 2024-05-27T00:15:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 -- start start 2024-05-27T00:15:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dd0004810 0x7f6dec13dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec13d790 0x7f6dec1517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dec1f8030 0x7f6dec1f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec06c490 con 0x7f6dec1f8030 2024-05-27T00:15:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec067d60 con 0x7f6dec13d790 2024-05-27T00:15:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6dec06c610 con 0x7f6dd0004810 2024-05-27T00:15:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6dea800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dec1f8030 0x7f6dec1f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6dea800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dec1f8030 0x7f6dec1f48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51316/0 (socket says 172.21.15.98:51316) 2024-05-27T00:15:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6dea800640 1 -- 172.21.15.98:0/1736528302 learned_addr learned my addr 172.21.15.98:0/1736528302 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6de9400640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec13d790 0x7f6dec1517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6de9e00640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dd0004810 0x7f6dec13dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6dea800640 1 -- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dd0004810 msgr2=0x7f6dec13dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6dea800640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dd0004810 0x7f6dec13dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6dea800640 1 -- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec13d790 msgr2=0x7f6dec1517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6dea800640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec13d790 0x7f6dec1517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6dea800640 1 -- 172.21.15.98:0/1736528302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec15c040 con 0x7f6dec1f8030 2024-05-27T00:15:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6de9e00640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dd0004810 0x7f6dec13dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6dea800640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dec1f8030 0x7f6dec1f48c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6dec15cd80 tx=0x7f6de40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6de2000640 1 -- 172.21.15.98:0/1736528302 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4011e40 con 0x7f6dec1f8030 2024-05-27T00:15:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6de2000640 1 -- 172.21.15.98:0/1736528302 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de401b070 con 0x7f6dec1f8030 2024-05-27T00:15:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 -- 172.21.15.98:0/1736528302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dec15c320 con 0x7f6dec1f8030 2024-05-27T00:15:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6de2000640 1 -- 172.21.15.98:0/1736528302 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de40039a0 con 0x7f6dec1f8030 2024-05-27T00:15:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 -- 172.21.15.98:0/1736528302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dec15c8e0 con 0x7f6dec1f8030 2024-05-27T00:15:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.301+0000 7f6df0e00640 1 -- 172.21.15.98:0/1736528302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dec075cb0 con 0x7f6dec1f8030 2024-05-27T00:15:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.305+0000 7f6de2000640 1 -- 172.21.15.98:0/1736528302 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6de4003b40 con 0x7f6dec1f8030 2024-05-27T00:15:39.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.305+0000 7f6de2000640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6dcc0678c0 0x7f6dcc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.305+0000 7f6de2000640 1 -- 172.21.15.98:0/1736528302 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f6de4086680 con 0x7f6dec1f8030 2024-05-27T00:15:39.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.305+0000 7f6de9e00640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6dcc0678c0 0x7f6dcc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.305+0000 7f6de9e00640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6dcc0678c0 0x7f6dcc069d80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6dec069d30 tx=0x7f6ddc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:39.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.309+0000 7f6de2000640 1 -- 172.21.15.98:0/1736528302 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f6de4057120 con 0x7f6dec1f8030 2024-05-27T00:15:39.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.497+0000 7f6df0e00640 1 -- 172.21.15.98:0/1736528302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dec156320 con 0x7f6dec1f8030 2024-05-27T00:15:39.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.497+0000 7f6de2000640 1 -- 172.21.15.98:0/1736528302 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f6de4056a20 con 0x7f6dec1f8030 2024-05-27T00:15:39.501 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:39.501 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:15:39.501 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 -- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6dcc0678c0 msgr2=0x7f6dcc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6dcc0678c0 0x7f6dcc069d80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6dec069d30 tx=0x7f6ddc009040 comp rx=0 tx=0).stop 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 -- 172.21.15.98:0/1736528302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dec1f8030 msgr2=0x7f6dec1f48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dec1f8030 0x7f6dec1f48c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6dec15cd80 tx=0x7f6de40115a0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 -- 172.21.15.98:0/1736528302 shutdown_connections 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6dec1f8030 0x7f6dec1f48c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6dcc0678c0 0x7f6dcc069d80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec13d790 0x7f6dec1517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 --2- 172.21.15.98:0/1736528302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dd0004810 0x7f6dec13dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 -- 172.21.15.98:0/1736528302 >> 172.21.15.98:0/1736528302 conn(0x7f6dec0c9af0 msgr2=0x7f6dec072b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 -- 172.21.15.98:0/1736528302 shutdown_connections 2024-05-27T00:15:39.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.501+0000 7f6df0e00640 1 -- 172.21.15.98:0/1736528302 wait complete. 2024-05-27T00:15:39.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:39.545 INFO:tasks.ceph.mds.c.smithi098.stderr:2024-05-27T00:15:39.541+0000 7fe387c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:15:39.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.653+0000 7f99edc00640 1 Processor -- start 2024-05-27T00:15:39.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.653+0000 7f99edc00640 1 -- start start 2024-05-27T00:15:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99edc00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f99e80731f0 0x7f99e80c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99edc00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f99e80735d0 0x7f99e8151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99edc00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99e80c17d0 con 0x7f99e80c79f0 2024-05-27T00:15:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99edc00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f99e8071990 con 0x7f99e80735d0 2024-05-27T00:15:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99edc00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f99e8070e10 con 0x7f99e80731f0 2024-05-27T00:15:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99de000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f99e80735d0 0x7f99e8151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99de000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f99e80735d0 0x7f99e8151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51320/0 (socket says 172.21.15.98:51320) 2024-05-27T00:15:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99de000640 1 -- 172.21.15.98:0/2066261865 learned_addr learned my addr 172.21.15.98:0/2066261865 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99e6a00640 1 --2- 172.21.15.98:0/2066261865 >> v2:172.21.15.204:3300/0 conn(0x7f99e80731f0 0x7f99e80c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99de000640 1 -- 172.21.15.98:0/2066261865 >> v2:172.21.15.204:3300/0 conn(0x7f99e80731f0 msgr2=0x7f99e80c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99de000640 1 --2- 172.21.15.98:0/2066261865 >> v2:172.21.15.204:3300/0 conn(0x7f99e80731f0 0x7f99e80c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99de000640 1 -- 172.21.15.98:0/2066261865 >> v1:172.21.15.204:6789/0 conn(0x7f99e80c79f0 legacy=0x7f99e8157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.657+0000 7f99de000640 1 -- 172.21.15.98:0/2066261865 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e80710f0 con 0x7f99e80735d0 2024-05-27T00:15:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99de000640 1 --2- 172.21.15.98:0/2066261865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80735d0 0x7f99e8151430 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99e80cca90 tx=0x7f99d001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bde328662172545 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99e6000640 1 -- 172.21.15.98:0/2066261865 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d00aa650 con 0x7f99e80735d0 2024-05-27T00:15:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99e6000640 1 -- 172.21.15.98:0/2066261865 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99d00af070 con 0x7f99e80735d0 2024-05-27T00:15:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99e6000640 1 -- 172.21.15.98:0/2066261865 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d00aaa10 con 0x7f99e80735d0 2024-05-27T00:15:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 -- 172.21.15.98:0/2066261865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80735d0 msgr2=0x7f99e8151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 --2- 172.21.15.98:0/2066261865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80735d0 0x7f99e8151430 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99e80cca90 tx=0x7f99d001c4f0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 -- 172.21.15.98:0/2066261865 shutdown_connections 2024-05-27T00:15:39.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 --2- 172.21.15.98:0/2066261865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80735d0 0x7f99e8151430 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 --2- 172.21.15.98:0/2066261865 >> v2:172.21.15.204:3300/0 conn(0x7f99e80731f0 0x7f99e80c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 -- 172.21.15.98:0/2066261865 >> 172.21.15.98:0/2066261865 conn(0x7f99e8067530 msgr2=0x7f99e806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 -- 172.21.15.98:0/2066261865 shutdown_connections 2024-05-27T00:15:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 -- 172.21.15.98:0/2066261865 wait complete. 2024-05-27T00:15:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 Processor -- start 2024-05-27T00:15:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 -- start start 2024-05-27T00:15:39.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80731f0 0x7f99e80cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99e80735d0 0x7f99e80cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99e80c79f0 0x7f99e81595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f99e80c13f0 con 0x7f99e80731f0 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f99e8151d80 con 0x7f99e80735d0 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99edc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f99e80c1570 con 0x7f99e80c79f0 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.661+0000 7f99e6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80731f0 0x7f99e80cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99e6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80731f0 0x7f99e80cad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51324/0 (socket says 172.21.15.98:51324) 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99e6a00640 1 -- 172.21.15.98:0/1951107903 learned_addr learned my addr 172.21.15.98:0/1951107903 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99de000640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99e80735d0 0x7f99e80cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99e7400640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99e80c79f0 0x7f99e81595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99e6a00640 1 -- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99e80c79f0 msgr2=0x7f99e81595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99e6a00640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99e80c79f0 0x7f99e81595d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99e6a00640 1 -- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99e80735d0 msgr2=0x7f99e80cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99e6a00640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99e80735d0 0x7f99e80cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99e6a00640 1 -- 172.21.15.98:0/1951107903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e815b210 con 0x7f99e80731f0 2024-05-27T00:15:39.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99de000640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99e80735d0 0x7f99e80cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:39.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99e6a00640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80731f0 0x7f99e80cad30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f99d800d800 tx=0x7f99d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:39.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99df400640 1 -- 172.21.15.98:0/1951107903 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d800b6c0 con 0x7f99e80731f0 2024-05-27T00:15:39.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99df400640 1 -- 172.21.15.98:0/1951107903 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99d8019070 con 0x7f99e80731f0 2024-05-27T00:15:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99df400640 1 -- 172.21.15.98:0/1951107903 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d800ba60 con 0x7f99e80731f0 2024-05-27T00:15:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99edc00640 1 -- 172.21.15.98:0/1951107903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e8158ca0 con 0x7f99e80731f0 2024-05-27T00:15:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99edc00640 1 -- 172.21.15.98:0/1951107903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e815b480 con 0x7f99e80731f0 2024-05-27T00:15:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99edc00640 1 -- 172.21.15.98:0/1951107903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99e8075cb0 con 0x7f99e80731f0 2024-05-27T00:15:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99df400640 1 -- 172.21.15.98:0/1951107903 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f99d800bc00 con 0x7f99e80731f0 2024-05-27T00:15:39.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99df400640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99c40678c0 0x7f99c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:39.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.665+0000 7f99df400640 1 -- 172.21.15.98:0/1951107903 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f99d8086110 con 0x7f99e80731f0 2024-05-27T00:15:39.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.669+0000 7f99de000640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99c40678c0 0x7f99c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:39.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.669+0000 7f99de000640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99c40678c0 0x7f99c4069d80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f99d00ae000 tx=0x7f99d00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.673+0000 7f99df400640 1 -- 172.21.15.98:0/1951107903 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f99d8056c40 con 0x7f99e80731f0 2024-05-27T00:15:39.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 -- 172.21.15.98:0/1951107903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f99e8158510 con 0x7f99e80731f0 2024-05-27T00:15:39.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99df400640 1 -- 172.21.15.98:0/1951107903 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16581 (secure 0 0 0) 0x7f99d8056540 con 0x7f99e80731f0 2024-05-27T00:15:39.832 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:39.832 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:26.579229+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:39.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 -- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99c40678c0 msgr2=0x7f99c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99c40678c0 0x7f99c4069d80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f99d00ae000 tx=0x7f99d00a7000 comp rx=0 tx=0).stop 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 -- 172.21.15.98:0/1951107903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80731f0 msgr2=0x7f99e80cad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80731f0 0x7f99e80cad30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f99d800d800 tx=0x7f99d800dcc0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 -- 172.21.15.98:0/1951107903 shutdown_connections 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99e80c79f0 0x7f99e81595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99c40678c0 0x7f99c4069d80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99e80735d0 0x7f99e80cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 --2- 172.21.15.98:0/1951107903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99e80731f0 0x7f99e80cad30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 -- 172.21.15.98:0/1951107903 >> 172.21.15.98:0/1951107903 conn(0x7f99e8067530 msgr2=0x7f99e806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 -- 172.21.15.98:0/1951107903 shutdown_connections 2024-05-27T00:15:39.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:39.829+0000 7f99edc00640 1 -- 172.21.15.98:0/1951107903 wait complete. 2024-05-27T00:15:39.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:15:40.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a9b800640 1 Processor -- start 2024-05-27T00:15:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a9b800640 1 -- start start 2024-05-27T00:15:40.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a9b800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a941528a0 0x7f1a940c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a9b800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1a940c3020 0x7f1a9406e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a9b800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a940ca550 con 0x7f1a9406e4a0 2024-05-27T00:15:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a9b800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1a940ce3a0 con 0x7f1a940c3020 2024-05-27T00:15:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a9b800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1a940c51e0 con 0x7f1a941528a0 2024-05-27T00:15:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a99800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1a940c3020 0x7f1a9406e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a99800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1a940c3020 0x7f1a9406e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51336/0 (socket says 172.21.15.98:51336) 2024-05-27T00:15:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a99800640 1 -- 172.21.15.98:0/1773971606 learned_addr learned my addr 172.21.15.98:0/1773971606 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:40.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.025+0000 7f1a98e00640 1 --2- 172.21.15.98:0/1773971606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a941528a0 0x7f1a940c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a99800640 1 -- 172.21.15.98:0/1773971606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a941528a0 msgr2=0x7f1a940c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a99800640 1 --2- 172.21.15.98:0/1773971606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a941528a0 0x7f1a940c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a99800640 1 -- 172.21.15.98:0/1773971606 >> v1:172.21.15.204:6789/0 conn(0x7f1a9406e4a0 legacy=0x7f1a940c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:40.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a99800640 1 -- 172.21.15.98:0/1773971606 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a940c8aa0 con 0x7f1a940c3020 2024-05-27T00:15:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a99800640 1 --2- 172.21.15.98:0/1773971606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 0x7f1a9406e990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009e50 tx=0x7f1a8c01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bc508b9da85752 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a93e00640 1 -- 172.21.15.98:0/1773971606 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0a57e0 con 0x7f1a940c3020 2024-05-27T00:15:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a93e00640 1 -- 172.21.15.98:0/1773971606 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a8c0a5980 con 0x7f1a940c3020 2024-05-27T00:15:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a93e00640 1 -- 172.21.15.98:0/1773971606 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c0a5d50 con 0x7f1a940c3020 2024-05-27T00:15:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a9b800640 1 -- 172.21.15.98:0/1773971606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 msgr2=0x7f1a9406e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a9b800640 1 --2- 172.21.15.98:0/1773971606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 0x7f1a9406e990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009e50 tx=0x7f1a8c01ca00 comp rx=0 tx=0).stop 2024-05-27T00:15:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a9b800640 1 -- 172.21.15.98:0/1773971606 shutdown_connections 2024-05-27T00:15:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a9b800640 1 --2- 172.21.15.98:0/1773971606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 0x7f1a9406e990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a9b800640 1 --2- 172.21.15.98:0/1773971606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a941528a0 0x7f1a940c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.029+0000 7f1a9b800640 1 -- 172.21.15.98:0/1773971606 >> 172.21.15.98:0/1773971606 conn(0x7f1a9406d680 msgr2=0x7f1a9406da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:40.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 -- 172.21.15.98:0/1773971606 shutdown_connections 2024-05-27T00:15:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 -- 172.21.15.98:0/1773971606 wait complete. 2024-05-27T00:15:40.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 Processor -- start 2024-05-27T00:15:40.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 -- start start 2024-05-27T00:15:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9406e4a0 0x7f1a94153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 0x7f1a941f7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a941528a0 0x7f1a940c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94152c80 con 0x7f1a940c3020 2024-05-27T00:15:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1a940ce810 con 0x7f1a9406e4a0 2024-05-27T00:15:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1a941656c0 con 0x7f1a941528a0 2024-05-27T00:15:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a8be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 0x7f1a941f7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a8be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 0x7f1a941f7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51346/0 (socket says 172.21.15.98:51346) 2024-05-27T00:15:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a8be00640 1 -- 172.21.15.98:0/297306085 learned_addr learned my addr 172.21.15.98:0/297306085 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a98e00640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9406e4a0 0x7f1a94153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a99800640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a941528a0 0x7f1a940c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a8be00640 1 -- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a941528a0 msgr2=0x7f1a940c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a8be00640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a941528a0 0x7f1a940c6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a8be00640 1 -- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9406e4a0 msgr2=0x7f1a94153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a8be00640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9406e4a0 0x7f1a94153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a8be00640 1 -- 172.21.15.98:0/297306085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a94155f80 con 0x7f1a940c3020 2024-05-27T00:15:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a8be00640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 0x7f1a941f7480 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c00c370 tx=0x7f1a7c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a91600640 1 -- 172.21.15.98:0/297306085 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c0052d0 con 0x7f1a940c3020 2024-05-27T00:15:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a91600640 1 -- 172.21.15.98:0/297306085 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a7c005470 con 0x7f1a940c3020 2024-05-27T00:15:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a91600640 1 -- 172.21.15.98:0/297306085 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c007260 con 0x7f1a940c3020 2024-05-27T00:15:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.033+0000 7f1a9b800640 1 -- 172.21.15.98:0/297306085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a94152f20 con 0x7f1a940c3020 2024-05-27T00:15:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.037+0000 7f1a9b800640 1 -- 172.21.15.98:0/297306085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a941534e0 con 0x7f1a940c3020 2024-05-27T00:15:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.037+0000 7f1a9b800640 1 -- 172.21.15.98:0/297306085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a94075cb0 con 0x7f1a940c3020 2024-05-27T00:15:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.037+0000 7f1a91600640 1 -- 172.21.15.98:0/297306085 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1a7c007400 con 0x7f1a940c3020 2024-05-27T00:15:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.037+0000 7f1a91600640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a700678f0 0x7f1a70069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.037+0000 7f1a91600640 1 -- 172.21.15.98:0/297306085 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f1a7c0850c0 con 0x7f1a940c3020 2024-05-27T00:15:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.037+0000 7f1a98e00640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a700678f0 0x7f1a70069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.041+0000 7f1a98e00640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a700678f0 0x7f1a70069db0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1a84006aa0 tx=0x7f1a84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:40.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.041+0000 7f1a91600640 1 -- 172.21.15.98:0/297306085 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1a7c055510 con 0x7f1a940c3020 2024-05-27T00:15:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.229+0000 7f1a9b800640 1 -- 172.21.15.98:0/297306085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a9406f4b0 con 0x7f1a940c3020 2024-05-27T00:15:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.229+0000 7f1a91600640 1 -- 172.21.15.98:0/297306085 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f1a7c059810 con 0x7f1a940c3020 2024-05-27T00:15:40.234 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:40.234 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:15:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:15:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 -- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a700678f0 msgr2=0x7f1a70069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a700678f0 0x7f1a70069db0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1a84006aa0 tx=0x7f1a84008040 comp rx=0 tx=0).stop 2024-05-27T00:15:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 -- 172.21.15.98:0/297306085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 msgr2=0x7f1a941f7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 0x7f1a941f7480 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c00c370 tx=0x7f1a7c00c830 comp rx=0 tx=0).stop 2024-05-27T00:15:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 -- 172.21.15.98:0/297306085 shutdown_connections 2024-05-27T00:15:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a941528a0 0x7f1a940c6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a940c3020 0x7f1a941f7480 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a700678f0 0x7f1a70069db0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 --2- 172.21.15.98:0/297306085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9406e4a0 0x7f1a94153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 -- 172.21.15.98:0/297306085 >> 172.21.15.98:0/297306085 conn(0x7f1a9406d680 msgr2=0x7f1a940cd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 -- 172.21.15.98:0/297306085 shutdown_connections 2024-05-27T00:15:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.233+0000 7f1a9b800640 1 -- 172.21.15.98:0/297306085 wait complete. 2024-05-27T00:15:40.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-27T00:15:24.956429+0000', 'modified': '2024-05-27T00:15:31.625115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4290': {'gid': 4290, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.98:6837/3824013488', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6836', 'nonce': 3824013488}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 3824013488}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4290, 'qdb_cluster': [4290]} 2024-05-27T00:15:40.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-27T00:15:40.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-27T00:15:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2256400640 1 Processor -- start 2024-05-27T00:15:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2256400640 1 -- start start 2024-05-27T00:15:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2256400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f22500c6360 0x7f22500c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2256400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22500c3970 con 0x7f225015d770 2024-05-27T00:15:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2256400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f225015db50 con 0x7f22500c23f0 2024-05-27T00:15:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2256400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f22500c6850 con 0x7f22500c6360 2024-05-27T00:15:40.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2255000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f225015d770 0x7f22500c7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47162/0 (socket says 172.21.15.98:47162) 2024-05-27T00:15:40.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2255000640 1 -- 172.21.15.98:0/883502965 learned_addr learned my addr 172.21.15.98:0/883502965 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:40.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2247e00640 1 --2- 172.21.15.98:0/883502965 >> v2:172.21.15.204:3300/0 conn(0x7f22500c6360 0x7f22500c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:40.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3853557507 0 0) 0x7f22500c3970 con 0x7f225015d770 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2234003530 con 0x7f225015d770 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1353958951 0 0) 0x7f2234003530 con 0x7f225015d770 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 >> v2:172.21.15.204:3300/0 conn(0x7f22500c6360 msgr2=0x7f22500c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 --2- 172.21.15.98:0/883502965 >> v2:172.21.15.204:3300/0 conn(0x7f22500c6360 0x7f22500c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 >> v1:172.21.15.204:6789/0 conn(0x7f22500c23f0 legacy=0x7f22500c8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22500c3f50 con 0x7f225015d770 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f22400016a0 con 0x7f225015d770 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 >> v1:172.21.15.98:6789/0 conn(0x7f225015d770 legacy=0x7f22500c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2234003a10 0x7f2234003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2234004810 0x7f2234004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f223400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22500c6850 con 0x7f22340051d0 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f223400c0b0 con 0x7f2234004810 2024-05-27T00:15:40.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f223400c2b0 con 0x7f2234003a10 2024-05-27T00:15:41.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2255a00640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f223400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2255000640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2234003a10 0x7f2234003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2247e00640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2234004810 0x7f2234004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2255a00640 1 -- 172.21.15.98:0/883502965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2234003a10 msgr2=0x7f2234003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.993+0000 7f2255a00640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2234003a10 0x7f2234003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2255a00640 1 -- 172.21.15.98:0/883502965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2234004810 msgr2=0x7f2234004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2255a00640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2234004810 0x7f2234004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2255a00640 1 -- 172.21.15.98:0/883502965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2234003530 con 0x7f22340051d0 2024-05-27T00:15:41.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2255a00640 1 -- 172.21.15.98:0/883502965 reap_dead start 2024-05-27T00:15:41.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2255a00640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f223400b9d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2248001e70 tx=0x7f224801c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2d3d01754f8ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:41.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2248001a90 con 0x7f22340051d0 2024-05-27T00:15:41.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2248003bc0 con 0x7f22340051d0 2024-05-27T00:15:41.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f224fe00640 1 -- 172.21.15.98:0/883502965 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2248005020 con 0x7f22340051d0 2024-05-27T00:15:41.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2256400640 1 -- 172.21.15.98:0/883502965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 msgr2=0x7f223400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2256400640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f223400b9d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2248001e70 tx=0x7f224801c780 comp rx=0 tx=0).stop 2024-05-27T00:15:41.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2256400640 1 -- 172.21.15.98:0/883502965 shutdown_connections 2024-05-27T00:15:41.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2256400640 1 --2- 172.21.15.98:0/883502965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f223400b9d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2256400640 1 -- 172.21.15.98:0/883502965 >> 172.21.15.98:0/883502965 conn(0x7f22500cdeb0 msgr2=0x7f22500ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:41.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2256400640 1 -- 172.21.15.98:0/883502965 shutdown_connections 2024-05-27T00:15:41.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:40.997+0000 7f2256400640 1 -- 172.21.15.98:0/883502965 wait complete. 2024-05-27T00:15:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2256400640 1 Processor -- start 2024-05-27T00:15:41.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2256400640 1 -- start start 2024-05-27T00:15:41.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2256400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f225006e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2256400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2250159f00 0x7f22501e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2256400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250158280 0x7f2250156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2256400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22500c69d0 con 0x7f22340051d0 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2256400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f22500c4510 con 0x7f2250158280 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2256400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f22500c4f40 con 0x7f2250159f00 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2255000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f225006e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2255000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f225006e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51358/0 (socket says 172.21.15.98:51358) 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2255000640 1 -- 172.21.15.98:0/3113355462 learned_addr learned my addr 172.21.15.98:0/3113355462 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2247e00640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2250159f00 0x7f22501e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2255a00640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250158280 0x7f2250156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2255000640 1 -- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2250159f00 msgr2=0x7f22501e5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2255000640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2250159f00 0x7f22501e5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2255000640 1 -- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250158280 msgr2=0x7f2250156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2255000640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250158280 0x7f2250156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2255000640 1 -- 172.21.15.98:0/3113355462 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225006f0b0 con 0x7f22340051d0 2024-05-27T00:15:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2255000640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f225006e7e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2240004e80 tx=0x7f2240007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f224d600640 1 -- 172.21.15.98:0/3113355462 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224000f750 con 0x7f22340051d0 2024-05-27T00:15:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f224d600640 1 -- 172.21.15.98:0/3113355462 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2240018070 con 0x7f22340051d0 2024-05-27T00:15:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f224d600640 1 -- 172.21.15.98:0/3113355462 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224000fb50 con 0x7f22340051d0 2024-05-27T00:15:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2256400640 1 -- 172.21.15.98:0/3113355462 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f225006f390 con 0x7f22340051d0 2024-05-27T00:15:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.001+0000 7f2256400640 1 -- 172.21.15.98:0/3113355462 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22501f0a40 con 0x7f22340051d0 2024-05-27T00:15:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f224d600640 1 -- 172.21.15.98:0/3113355462 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f224000fcf0 con 0x7f22340051d0 2024-05-27T00:15:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f224d600640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f222c0678c0 0x7f222c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f224d600640 1 -- 172.21.15.98:0/3113355462 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2240085d00 con 0x7f22340051d0 2024-05-27T00:15:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f225af9f000 1 Processor -- start 2024-05-27T00:15:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f2247e00640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f222c0678c0 0x7f222c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f225af9f000 1 -- start start 2024-05-27T00:15:41.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f225af9f000 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400254650 0x55f40021a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f2247e00640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f222c0678c0 0x7f222c069d80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f22500cdc10 tx=0x7f2238008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f225af9f000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f40021add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f225af9f000 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400255d60 0x55f400256140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f225af9f000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f400062ee0 con 0x55f40021a9f0 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f225af9f000 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f3ffffef70 con 0x55f400255d60 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.005+0000 7f225af9f000 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x55f40005eea0 con 0x55f400254650 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2247e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f40021add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2247e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f40021add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51366/0 (socket says 172.21.15.98:51366) 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2247e00640 1 -- 172.21.15.98:0/1294998123 learned_addr learned my addr 172.21.15.98:0/1294998123 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2255000640 1 --2- 172.21.15.98:0/1294998123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400254650 0x55f40021a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2255a00640 1 --2- 172.21.15.98:0/1294998123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400255d60 0x55f400256140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2247e00640 1 -- 172.21.15.98:0/1294998123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400254650 msgr2=0x55f40021a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2247e00640 1 --2- 172.21.15.98:0/1294998123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400254650 0x55f40021a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2247e00640 1 -- 172.21.15.98:0/1294998123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400255d60 msgr2=0x55f400256140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2247e00640 1 --2- 172.21.15.98:0/1294998123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400255d60 0x55f400256140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2247e00640 1 -- 172.21.15.98:0/1294998123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f40002f950 con 0x55f40021a9f0 2024-05-27T00:15:41.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2247e00640 1 --2- 172.21.15.98:0/1294998123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f40021add0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2238004350 tx=0x7f2238016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d53c2fc96fe9657 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2245600640 1 -- 172.21.15.98:0/1294998123 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2238029070 con 0x55f40021a9f0 2024-05-27T00:15:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2245600640 1 -- 172.21.15.98:0/1294998123 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f223801f050 con 0x55f40021a9f0 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2245600640 1 -- 172.21.15.98:0/1294998123 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2238003da0 con 0x55f40021a9f0 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 -- 172.21.15.98:0/1294998123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 msgr2=0x55f40021add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 --2- 172.21.15.98:0/1294998123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f40021add0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2238004350 tx=0x7f2238016000 comp rx=0 tx=0).stop 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 -- 172.21.15.98:0/1294998123 shutdown_connections 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 --2- 172.21.15.98:0/1294998123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400255d60 0x55f400256140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 --2- 172.21.15.98:0/1294998123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f40021add0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 --2- 172.21.15.98:0/1294998123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400254650 0x55f40021a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 -- 172.21.15.98:0/1294998123 >> 172.21.15.98:0/1294998123 conn(0x55f4000fc990 msgr2=0x55f400253b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 -- 172.21.15.98:0/1294998123 shutdown_connections 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 -- 172.21.15.98:0/1294998123 wait complete. 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 -- start start 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 Processor -- start 2024-05-27T00:15:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f40021a9f0 0x55f400219380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400254650 0x55f40025c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400255d60 0x55f40025c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f4000e53e0 con 0x55f400255d60 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f40002c0c0 con 0x55f40021a9f0 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x55f40002c3c0 con 0x55f400254650 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f225af9f000 10 client.0 _ll_register_callbacks cb 0x55f4002002b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2255a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400255d60 0x55f40025c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2255a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400255d60 0x55f40025c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51378/0 (socket says 172.21.15.98:51378) 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2255a00640 1 -- 172.21.15.98:0/1513036038 learned_addr learned my addr 172.21.15.98:0/1513036038 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.009+0000 7f2255000640 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f40021a9f0 0x55f400219380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f2247e00640 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400254650 0x55f40025c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f2255a00640 1 -- 172.21.15.98:0/1513036038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400254650 msgr2=0x55f40025c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f2255a00640 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400254650 0x55f40025c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f2255a00640 1 -- 172.21.15.98:0/1513036038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f40021a9f0 msgr2=0x55f400219380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f2255a00640 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f40021a9f0 0x55f400219380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f2255a00640 1 -- 172.21.15.98:0/1513036038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3fffeae80 con 0x55f400255d60 2024-05-27T00:15:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f2255a00640 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400255d60 0x55f40025c8f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f224801c5b0 tx=0x7f2248019520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c6d5ffc0ff46bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f223e000640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-27T00:15:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f223e000640 1 -- 172.21.15.98:0/1513036038 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22480b7060 con 0x55f400255d60 2024-05-27T00:15:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 -- 172.21.15.98:0/1513036038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f223801f050 con 0x55f400255d60 2024-05-27T00:15:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f223e000640 1 -- 172.21.15.98:0/1513036038 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2248006d60 con 0x55f400255d60 2024-05-27T00:15:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f223e000640 1 -- 172.21.15.98:0/1513036038 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f22480bd050 con 0x55f400255d60 2024-05-27T00:15:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 10 client.4668 fetch_fsmap learned FSMap version 8 2024-05-27T00:15:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 -- 172.21.15.98:0/1513036038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55f4000e5870 con 0x55f400255d60 2024-05-27T00:15:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f223e000640 1 -- 172.21.15.98:0/1513036038 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f223801f050 con 0x55f400255d60 2024-05-27T00:15:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f223e000640 1 -- 172.21.15.98:0/1513036038 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 8) ==== 4704+0+0 (secure 0 0 0) 0x7f2248005020 con 0x55f400255d60 2024-05-27T00:15:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 10 client.4668 fetch_fsmap finished waiting for FSMap version 8 2024-05-27T00:15:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 10 client.4668 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 20 client.4668 populate_metadata read hostname 'smithi098' 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.98:6836/3824013488,v1:172.21.15.98:6837/3824013488] conn(0x55f40024fa80 0x55f40024fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 4 client.4668 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 -- 172.21.15.98:0/1513036038 --> [v2:172.21.15.98:6836/3824013488,v1:172.21.15.98:6837/3824013488] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3ffffdab0 con 0x55f40024fa80 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f2255000640 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.98:6836/3824013488,v1:172.21.15.98:6837/3824013488] conn(0x55f40024fa80 0x55f40024fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f2255000640 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.98:6836/3824013488,v1:172.21.15.98:6837/3824013488] conn(0x55f40024fa80 0x55f40024fe60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d35de5c3a5bade4f server_cookie=b638b15d2f28e5a2 in_seq=0 out_seq=0 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f223e000640 10 client.4668 ms_handle_connect on v2:172.21.15.98:6836/3824013488 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f223e000640 1 -- 172.21.15.98:0/1513036038 <== mds.0 v2:172.21.15.98:6836/3824013488 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f3ffffdab0 con 0x55f40024fa80 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f223e000640 10 client.4668 handle_command_reply: tid=0 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 client.4668 shutdown 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 -- 172.21.15.98:0/1513036038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400255d60 msgr2=0x55f40025c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400255d60 0x55f40025c8f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f224801c5b0 tx=0x7f2248019520 comp rx=0 tx=0).stop 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 -- 172.21.15.98:0/1513036038 shutdown_connections 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.98:6836/3824013488,v1:172.21.15.98:6837/3824013488] conn(0x55f40024fa80 0x55f40024fe60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400255d60 0x55f40025c8f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400254650 0x55f40025c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f223e000640 0 client.4668 ms_handle_reset on v2:172.21.15.98:6836/3824013488 2024-05-27T00:15:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 --2- 172.21.15.98:0/1513036038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f40021a9f0 0x55f400219380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 -- 172.21.15.98:0/1513036038 >> 172.21.15.98:0/1513036038 conn(0x55f4000fc990 msgr2=0x55f4002555e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 -- 172.21.15.98:0/1513036038 shutdown_connections 2024-05-27T00:15:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 1 -- 172.21.15.98:0/1513036038 wait complete. 2024-05-27T00:15:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.013+0000 7f225af9f000 20 client.4668 trim_cache size 0 max 16384 2024-05-27T00:15:41.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f225af9f000 1 Processor -- start 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f225af9f000 1 -- start start 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f225af9f000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f400267410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f225af9f000 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400254650 0x55f400267950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f225af9f000 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 0x55f40025c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f225af9f000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22500c8d10 con 0x55f40021a9f0 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f225af9f000 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f22501fbf00 con 0x55f400254650 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f225af9f000 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x55f40006e000 con 0x55f400255d60 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f400267410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f400267410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51392/0 (socket says 172.21.15.98:51392) 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255000640 1 -- 172.21.15.98:0/3546798396 learned_addr learned my addr 172.21.15.98:0/3546798396 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255a00640 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 0x55f40025c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2247e00640 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400254650 0x55f400267950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255000640 1 -- 172.21.15.98:0/3546798396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 msgr2=0x55f40025c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255000640 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 0x55f40025c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255000640 1 -- 172.21.15.98:0/3546798396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400254650 msgr2=0x55f400267950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255000640 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400254650 0x55f400267950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255000640 1 -- 172.21.15.98:0/3546798396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4000bc590 con 0x55f40021a9f0 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255a00640 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 0x55f40025c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2247e00640 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400254650 0x55f400267950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2255000640 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f400267410 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2240055fa0 tx=0x7f2240084060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb15d64840fade25 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2244c00640 1 -- 172.21.15.98:0/3546798396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224005a3c0 con 0x55f40021a9f0 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2244c00640 1 -- 172.21.15.98:0/3546798396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f224005ec80 con 0x55f40021a9f0 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f2244c00640 1 -- 172.21.15.98:0/3546798396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22400842a0 con 0x55f40021a9f0 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f225af9f000 1 -- 172.21.15.98:0/3546798396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 msgr2=0x55f400267410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.033+0000 7f225af9f000 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f400267410 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2240055fa0 tx=0x7f2240084060 comp rx=0 tx=0).stop 2024-05-27T00:15:41.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- 172.21.15.98:0/3546798396 shutdown_connections 2024-05-27T00:15:41.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 0x55f40025c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f400254650 0x55f400267950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 --2- 172.21.15.98:0/3546798396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f40021a9f0 0x55f400267410 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- 172.21.15.98:0/3546798396 >> 172.21.15.98:0/3546798396 conn(0x55f4000fc990 msgr2=0x55f4002506f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- 172.21.15.98:0/3546798396 shutdown_connections 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- 172.21.15.98:0/3546798396 wait complete. 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- start start 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 Processor -- start 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f40021a9f0 0x55f4002cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400254650 0x55f4002cc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 0x55f40025ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f3ffffa7b0 con 0x55f400254650 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f400021e10 con 0x55f40021a9f0 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x55f4000621f0 con 0x55f400255d60 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 10 client.0 _ll_register_callbacks cb 0x7f2250071980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T00:15:41.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2247e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400254650 0x55f4002cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2247e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400254650 0x55f4002cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51394/0 (socket says 172.21.15.98:51394) 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2247e00640 1 -- 172.21.15.98:0/2932308011 learned_addr learned my addr 172.21.15.98:0/2932308011 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2255000640 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f40021a9f0 0x55f4002cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2255a00640 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 0x55f40025ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2247e00640 1 -- 172.21.15.98:0/2932308011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 msgr2=0x55f40025ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2247e00640 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 0x55f40025ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2247e00640 1 -- 172.21.15.98:0/2932308011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f40021a9f0 msgr2=0x55f4002cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2247e00640 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f40021a9f0 0x55f4002cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2247e00640 1 -- 172.21.15.98:0/2932308011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f400014f50 con 0x55f400254650 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f2247e00640 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400254650 0x55f4002cc420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2238027040 tx=0x7f2238004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c0741e070e9836 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f223e000640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-05-27T00:15:41.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f223e000640 1 -- 172.21.15.98:0/2932308011 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2238034070 con 0x55f400254650 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f223e000640 1 -- 172.21.15.98:0/2932308011 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2238003da0 con 0x55f400254650 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- 172.21.15.98:0/2932308011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f224005ec80 con 0x55f400254650 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f223e000640 1 -- 172.21.15.98:0/2932308011 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f2238029040 con 0x55f400254650 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 10 client.4674 fetch_fsmap learned FSMap version 8 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- 172.21.15.98:0/2932308011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55f4000097e0 con 0x55f400254650 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f223e000640 1 -- 172.21.15.98:0/2932308011 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f224005ec80 con 0x55f400254650 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f223e000640 1 -- 172.21.15.98:0/2932308011 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 8) ==== 4704+0+0 (secure 0 0 0) 0x7f2238030460 con 0x55f400254650 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 10 client.4674 fetch_fsmap finished waiting for FSMap version 8 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 10 client.4674 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 20 client.4674 populate_metadata read hostname 'smithi098' 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.98:6836/3824013488,v1:172.21.15.98:6837/3824013488] conn(0x55f40025b7c0 0x55f40025bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 4 client.4674 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T00:15:41.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.037+0000 7f225af9f000 1 -- 172.21.15.98:0/2932308011 --> [v2:172.21.15.98:6836/3824013488,v1:172.21.15.98:6837/3824013488] -- command(tid 0: {"prefix": "status"}) -- 0x55f400070610 con 0x55f40025b7c0 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2255000640 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.98:6836/3824013488,v1:172.21.15.98:6837/3824013488] conn(0x55f40025b7c0 0x55f40025bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f223e000640 10 client.4674 ms_handle_connect on v2:172.21.15.98:6836/3824013488 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2255000640 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.98:6836/3824013488,v1:172.21.15.98:6837/3824013488] conn(0x55f40025b7c0 0x55f40025bba0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=366d540b2d9ce8f4 server_cookie=b39e5dc8f5c7bc89 in_seq=0 out_seq=0 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f223e000640 1 -- 172.21.15.98:0/2932308011 <== mds.0 v2:172.21.15.98:6836/3824013488 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55f400070610 con 0x55f40025b7c0 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f223e000640 10 client.4674 handle_command_reply: tid=0 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 client.4674 shutdown 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 -- 172.21.15.98:0/2932308011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400254650 msgr2=0x55f4002cc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400254650 0x55f4002cc420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2238027040 tx=0x7f2238004740 comp rx=0 tx=0).stop 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 -- 172.21.15.98:0/2932308011 shutdown_connections 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.98:6836/3824013488,v1:172.21.15.98:6837/3824013488] conn(0x55f40025b7c0 0x55f40025bba0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x55f400255d60 0x55f40025ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f400254650 0x55f4002cc420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f223e000640 0 client.4674 ms_handle_reset on v2:172.21.15.98:6836/3824013488 2024-05-27T00:15:41.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 --2- 172.21.15.98:0/2932308011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f40021a9f0 0x55f4002cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 -- 172.21.15.98:0/2932308011 >> 172.21.15.98:0/2932308011 conn(0x55f4000fc990 msgr2=0x55f40024fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:41.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 -- 172.21.15.98:0/2932308011 shutdown_connections 2024-05-27T00:15:41.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 1 -- 172.21.15.98:0/2932308011 wait complete. 2024-05-27T00:15:41.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f225af9f000 20 client.4674 trim_cache size 0 max 16384 2024-05-27T00:15:41.048 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-27T00:15:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "f6d0c69d-6040-4984-af90-0cab9df1ba47", 2024-05-27T00:15:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-05-27T00:15:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4290, 2024-05-27T00:15:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-05-27T00:15:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-05-27T00:15:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-05-27T00:15:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 15.438790922000001, 2024-05-27T00:15:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 8, 2024-05-27T00:15:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 22, 2024-05-27T00:15:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 22, 2024-05-27T00:15:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 22.494891648999999 2024-05-27T00:15:41.049 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-27T00:15:41.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 -- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f222c0678c0 msgr2=0x7f222c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f222c0678c0 0x7f222c069d80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f22500cdc10 tx=0x7f2238008040 comp rx=0 tx=0).stop 2024-05-27T00:15:41.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 -- 172.21.15.98:0/3113355462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 msgr2=0x7f225006e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f225006e7e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2240004e80 tx=0x7f2240007430 comp rx=0 tx=0).stop 2024-05-27T00:15:41.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 -- 172.21.15.98:0/3113355462 shutdown_connections 2024-05-27T00:15:41.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f222c0678c0 0x7f222c069d80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250158280 0x7f2250156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2250159f00 0x7f22501e5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 --2- 172.21.15.98:0/3113355462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22340051d0 0x7f225006e7e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 -- 172.21.15.98:0/3113355462 >> 172.21.15.98:0/3113355462 conn(0x7f22500cdeb0 msgr2=0x7f225015ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:41.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 -- 172.21.15.98:0/3113355462 shutdown_connections 2024-05-27T00:15:41.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.041+0000 7f2256400640 1 -- 172.21.15.98:0/3113355462 wait complete. 2024-05-27T00:15:41.063 INFO:teuthology.run_tasks:Running task sequential... 2024-05-27T00:15:41.100 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-27T00:15:41.100 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-27T00:15:41.100 INFO:teuthology.task.print:Enabling mgr modules 2024-05-27T00:15:41.101 INFO:teuthology.run_tasks:Running task kclient... 2024-05-27T00:15:41.139 INFO:tasks.kclient:Mounting kernel clients... 2024-05-27T00:15:41.139 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-27T00:15:41.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:15:41.139 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-05-27T00:15:41.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:15:41.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-05-27T00:15:41.212 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2024-05-27T00:15:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:15:41.214 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-05-27T00:15:41.214 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T00:15:41.215 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T00:15:41.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-27T00:15:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7950c00640 1 Processor -- start 2024-05-27T00:15:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7950c00640 1 -- start start 2024-05-27T00:15:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7950c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c0c3020 0x7f794c0c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7950c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794c0ca550 con 0x7f794c06e4a0 2024-05-27T00:15:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7950c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794c0c51e0 con 0x7f794c1528a0 2024-05-27T00:15:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7950c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f794c0ce3a0 con 0x7f794c0c3020 2024-05-27T00:15:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7941400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f794c06e4a0 0x7f794c06e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47176/0 (socket says 172.21.15.98:47176) 2024-05-27T00:15:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7941400640 1 -- 172.21.15.98:0/2386561100 learned_addr learned my addr 172.21.15.98:0/2386561100 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f794a800640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c0c3020 0x7f794c0c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530533243 0 0) 0x7f794c0ca550 con 0x7f794c06e4a0 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7930003530 con 0x7f794c06e4a0 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 448318376 0 0) 0x7f7930003530 con 0x7f794c06e4a0 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c0c3020 msgr2=0x7f794c0c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7949400640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c0c3020 0x7f794c0c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 >> v1:172.21.15.204:6789/0 conn(0x7f794c1528a0 legacy=0x7f794c0c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c0c8aa0 con 0x7f794c06e4a0 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.393+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f7934001680 con 0x7f794c06e4a0 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 >> v1:172.21.15.98:6789/0 conn(0x7f794c06e4a0 legacy=0x7f794c06e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949400640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 0x7f7930003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949400640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c0c3020 0x7f7930004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949400640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7930004d50 0x7f79300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f794c0ce3a0 con 0x7f79300039f0 2024-05-27T00:15:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f793000b9d0 con 0x7f7930004d50 2024-05-27T00:15:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f793000bbd0 con 0x7f794c0c3020 2024-05-27T00:15:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949e00640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 0x7f7930003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7941400640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c0c3020 0x7f7930004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f794a800640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7930004d50 0x7f79300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949e00640 1 -- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c0c3020 msgr2=0x7f7930004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949e00640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c0c3020 0x7f7930004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949e00640 1 -- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7930004d50 msgr2=0x7f79300051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949e00640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7930004d50 0x7f79300051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949e00640 1 -- 172.21.15.98:0/2386561100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7930003530 con 0x7f79300039f0 2024-05-27T00:15:41.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949e00640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 0x7f7930003dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f793c001860 tx=0x7f793c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3f6caa3ef117d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:41.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c0b0070 con 0x7f79300039f0 2024-05-27T00:15:41.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f793c0ab600 con 0x7f79300039f0 2024-05-27T00:15:41.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.397+0000 7f7949400640 1 -- 172.21.15.98:0/2386561100 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c001460 con 0x7f79300039f0 2024-05-27T00:15:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 -- 172.21.15.98:0/2386561100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 msgr2=0x7f7930003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 --2- 172.21.15.98:0/2386561100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 0x7f7930003dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f793c001860 tx=0x7f793c01c430 comp rx=0 tx=0).stop 2024-05-27T00:15:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f794a800640 1 -- 172.21.15.98:0/2386561100 reap_dead start 2024-05-27T00:15:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 -- 172.21.15.98:0/2386561100 shutdown_connections 2024-05-27T00:15:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 -- 172.21.15.98:0/2386561100 >> 172.21.15.98:0/2386561100 conn(0x7f794c06d680 msgr2=0x7f794c06da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 -- 172.21.15.98:0/2386561100 shutdown_connections 2024-05-27T00:15:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 -- 172.21.15.98:0/2386561100 wait complete. 2024-05-27T00:15:41.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 Processor -- start 2024-05-27T00:15:41.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 -- start start 2024-05-27T00:15:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 0x7f794c1f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c1473a0 0x7f794c06cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f794c06c080 0x7f794c13e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f794c0ce830 con 0x7f79300039f0 2024-05-27T00:15:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f794c0ce030 con 0x7f794c06c080 2024-05-27T00:15:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7950c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f794c152c80 con 0x7f794c1473a0 2024-05-27T00:15:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7949e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 0x7f794c1f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7949e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 0x7f794c1f3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51412/0 (socket says 172.21.15.98:51412) 2024-05-27T00:15:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7949e00640 1 -- 172.21.15.98:0/366874320 learned_addr learned my addr 172.21.15.98:0/366874320 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7941400640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c1473a0 0x7f794c06cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f794a800640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f794c06c080 0x7f794c13e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7949e00640 1 -- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c1473a0 msgr2=0x7f794c06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7949e00640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c1473a0 0x7f794c06cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7949e00640 1 -- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f794c06c080 msgr2=0x7f794c13e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7949e00640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f794c06c080 0x7f794c13e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.401+0000 7f7949e00640 1 -- 172.21.15.98:0/366874320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c1f3d70 con 0x7f79300039f0 2024-05-27T00:15:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7949e00640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 0x7f794c1f3830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f793c0ae040 tx=0x7f7930005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7942a00640 1 -- 172.21.15.98:0/366874320 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c0bc070 con 0x7f79300039f0 2024-05-27T00:15:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7942a00640 1 -- 172.21.15.98:0/366874320 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f793c001440 con 0x7f79300039f0 2024-05-27T00:15:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7950c00640 1 -- 172.21.15.98:0/366874320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794c147ea0 con 0x7f79300039f0 2024-05-27T00:15:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7942a00640 1 -- 172.21.15.98:0/366874320 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c0b0040 con 0x7f79300039f0 2024-05-27T00:15:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7950c00640 1 -- 172.21.15.98:0/366874320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f794c147bc0 con 0x7f79300039f0 2024-05-27T00:15:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7950c00640 1 -- 172.21.15.98:0/366874320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f794c075cb0 con 0x7f79300039f0 2024-05-27T00:15:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7942a00640 1 -- 172.21.15.98:0/366874320 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f793c0bf070 con 0x7f79300039f0 2024-05-27T00:15:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7942a00640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79280678c0 0x7f7928069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7942a00640 1 -- 172.21.15.98:0/366874320 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f793c11f120 con 0x7f79300039f0 2024-05-27T00:15:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.405+0000 7f7941400640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79280678c0 0x7f7928069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.409+0000 7f7941400640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79280678c0 0x7f7928069d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f794c1f0a30 tx=0x7f7934009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:41.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.413+0000 7f7942a00640 1 -- 172.21.15.98:0/366874320 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f793c016170 con 0x7f79300039f0 2024-05-27T00:15:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.601+0000 7f7950c00640 1 -- 172.21.15.98:0/366874320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f794c06dde0 con 0x7f79300039f0 2024-05-27T00:15:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.601+0000 7f7942a00640 1 -- 172.21.15.98:0/366874320 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f793c0efbc0 con 0x7f79300039f0 2024-05-27T00:15:41.604 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-27T00:15:41.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 -- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79280678c0 msgr2=0x7f7928069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79280678c0 0x7f7928069d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f794c1f0a30 tx=0x7f7934009040 comp rx=0 tx=0).stop 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 -- 172.21.15.98:0/366874320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 msgr2=0x7f794c1f3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 0x7f794c1f3830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f793c0ae040 tx=0x7f7930005840 comp rx=0 tx=0).stop 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 -- 172.21.15.98:0/366874320 shutdown_connections 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f79280678c0 0x7f7928069d80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f794c06c080 0x7f794c13e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f794c1473a0 0x7f794c06cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 --2- 172.21.15.98:0/366874320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79300039f0 0x7f794c1f3830 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 -- 172.21.15.98:0/366874320 >> 172.21.15.98:0/366874320 conn(0x7f794c06d680 msgr2=0x7f794c0cbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 -- 172.21.15.98:0/366874320 shutdown_connections 2024-05-27T00:15:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41.605+0000 7f7950c00640 1 -- 172.21.15.98:0/366874320 wait complete. 2024-05-27T00:15:41.627 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-27T00:15:41.627 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T00:15:41.627 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-05-27T00:15:41.627 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-27T00:15:41.627 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T00:15:41.627 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-27T00:15:41.627 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T00:15:41.627 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T00:15:41.628 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-27T00:15:41.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:15:41.628 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-05-27T00:15:41.637 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42467sec preferred_lft 42467sec 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 02:42:84:e1:98:d3 brd ff:ff:ff:ff:ff:ff 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-05-27T00:15:41.638 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-05-27T00:15:41.640 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T00:15:41.640 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T00:15:41.640 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-27T00:15:41.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-05-27T00:15:41.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-05-27T00:15:41.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-05-27T00:15:41.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T00:15:41.640 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-27T00:15:41.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T00:15:41.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T00:15:41.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:15:41.795 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T00:15:41.862 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-05-27T00:15:41.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:15:41.864 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-05-27T00:15:41.943 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T00:15:41.944 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T00:15:41.944 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-27T00:15:41.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-27T00:15:41.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-27T00:15:41.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-27T00:15:41.944 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-27T00:15:42.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T00:15:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T00:15:42.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:15:42.079 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-05-27T00:15:42.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:15:42.089 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-05-27T00:15:42.142 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T00:15:42.142 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-27T00:15:42.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T00:15:42.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-27T00:15:42.142 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-27T00:15:42.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T00:15:42.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T00:15:42.272 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-27T00:15:42.272 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T00:15:42.272 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-27T00:15:42.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-27T00:15:42.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T00:15:42.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-27T00:15:42.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-27T00:15:42.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-27T00:15:42.273 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-27T00:15:42.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T00:15:42.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T00:15:42.440 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T00:15:42.440 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-05-27T00:15:42.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-05-27T00:15:42.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T00:15:42.440 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-05-27T00:15:42.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T00:15:42.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T00:15:42.570 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T00:15:42.570 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-27T00:15:42.579 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-27T00:15:42.579 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T00:15:42.579 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-27T00:15:42.633 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-05-27T00:15:42.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:15:42.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-27T00:15:42.721 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-27T00:15:42.722 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-27T00:15:42.722 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-05-27T00:15:43.289 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-27T00:15:43.289 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-27T00:15:43.289 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-05-27T00:15:43.289 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.204:3300,172.21.15.204:3301" 2024-05-27T00:15:43.289 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-27T00:15:43.303 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-05-27T00:15:43.303 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-05-27T00:15:43.303 INFO:teuthology.orchestra.run.smithi098.stdout:invalid new device string format 2024-05-27T00:15:43.303 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.204:3300,172.21.15.204:3301" 2024-05-27T00:15:43.303 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2024-05-27T00:15:43.303 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:6789,172.21.15.204:6789,172.21.15.204:6790" 2024-05-27T00:15:43.304 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:6789,172.21.15.204:6789,172.21.15.204:6790:/ 2024-05-27T00:15:43.304 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6d0c69d-6040-4984-af90-0cab9df1ba47" will pass to kernel 2024-05-27T00:15:43.307 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-27T00:15:43.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:15:43.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-27T00:15:43.329 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T00:15:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T00:15:43.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T00:15:43.449 INFO:teuthology.orchestra.run.smithi098.stdout:client4678 2024-05-27T00:15:43.450 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:15:43.450 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T00:15:43.458 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T00:15:43.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T00:15:43.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T00:15:43.565 INFO:teuthology.orchestra.run.smithi098.stdout:client4678 2024-05-27T00:15:43.565 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-27T00:15:43.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-27T00:15:43.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T00:15:43.630 INFO:teuthology.orchestra.run.smithi098.stdout:client4678 2024-05-27T00:15:43.631 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/f6d0c69d-6040-4984-af90-0cab9df1ba47.client4678/status') 2024-05-27T00:15:43.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:15:43.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:15:43.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T00:15:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:instance: client.4678 (3)172.21.15.98:0/1256063699 2024-05-27T00:15:43.696 INFO:teuthology.orchestra.run.smithi098.stdout:blocklisted: false 2024-05-27T00:15:43.696 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records in 2024-05-27T00:15:43.696 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records out 2024-05-27T00:15:43.696 INFO:teuthology.orchestra.run.smithi098.stderr:70 bytes copied, 6.019e-05 s, 1.2 MB/s 2024-05-27T00:15:43.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T00:15:43.704 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-05-27T00:15:43.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a40e00640 1 Processor -- start 2024-05-27T00:15:43.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a40e00640 1 -- start start 2024-05-27T00:15:43.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a40e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5a3c070fe0 0x7f5a3c154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:43.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a40e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a3c15a950 con 0x7f5a3c0cb570 2024-05-27T00:15:43.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a40e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a3c15e060 con 0x7f5a3c070c00 2024-05-27T00:15:43.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a40e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5a3c15e340 con 0x7f5a3c070fe0 2024-05-27T00:15:43.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a3a800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5a3c0cb570 0x7f5a3c15ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45806/0 (socket says 172.21.15.98:45806) 2024-05-27T00:15:43.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a3a800640 1 -- 172.21.15.98:0/3584441472 learned_addr learned my addr 172.21.15.98:0/3584441472 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:43.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a31400640 1 --2- 172.21.15.98:0/3584441472 >> v2:172.21.15.98:3300/0 conn(0x7f5a3c070fe0 0x7f5a3c154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:43.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a31400640 1 -- 172.21.15.98:0/3584441472 >> v1:172.21.15.204:6789/0 conn(0x7f5a3c070c00 legacy=0x7f5a3c0c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:43.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a31400640 1 -- 172.21.15.98:0/3584441472 >> v1:172.21.15.98:6789/0 conn(0x7f5a3c0cb570 legacy=0x7f5a3c15ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:43.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a39400640 1 -- 172.21.15.98:0/3584441472 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1860490022 0 0) 0x7f5a3c15a950 con 0x7f5a3c0cb570 2024-05-27T00:15:43.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.905+0000 7f5a31400640 1 -- 172.21.15.98:0/3584441472 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3c15e620 con 0x7f5a3c070fe0 2024-05-27T00:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a31400640 1 --2- 172.21.15.98:0/3584441472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c070fe0 0x7f5a3c154780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5a240097c0 tx=0x7f5a2401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46722d1aca4af6b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a39400640 1 -- 172.21.15.98:0/3584441472 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a240b5070 con 0x7f5a3c070fe0 2024-05-27T00:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a39400640 1 -- 172.21.15.98:0/3584441472 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a240a8990 con 0x7f5a3c070fe0 2024-05-27T00:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a39400640 1 -- 172.21.15.98:0/3584441472 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a240a8d60 con 0x7f5a3c070fe0 2024-05-27T00:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a40e00640 1 -- 172.21.15.98:0/3584441472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c070fe0 msgr2=0x7f5a3c154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a40e00640 1 --2- 172.21.15.98:0/3584441472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c070fe0 0x7f5a3c154780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5a240097c0 tx=0x7f5a2401ec90 comp rx=0 tx=0).stop 2024-05-27T00:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a40e00640 1 -- 172.21.15.98:0/3584441472 shutdown_connections 2024-05-27T00:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a40e00640 1 --2- 172.21.15.98:0/3584441472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c070fe0 0x7f5a3c154780 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:43.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a40e00640 1 -- 172.21.15.98:0/3584441472 >> 172.21.15.98:0/3584441472 conn(0x7f5a3c072f90 msgr2=0x7f5a3c0733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a40e00640 1 -- 172.21.15.98:0/3584441472 shutdown_connections 2024-05-27T00:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.909+0000 7f5a40e00640 1 -- 172.21.15.98:0/3584441472 wait complete. 2024-05-27T00:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 Processor -- start 2024-05-27T00:15:43.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 -- start start 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3c070c00 0x7f5a3c0cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a3c070fe0 0x7f5a3c0cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c0cb570 0x7f5a3c15c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c0c9110 con 0x7f5a3c0cb570 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c15ebe0 con 0x7f5a3c070c00 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5a3c15b560 con 0x7f5a3c070fe0 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a3a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c0cb570 0x7f5a3c15c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a3a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c0cb570 0x7f5a3c15c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53888/0 (socket says 172.21.15.98:53888) 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a3a800640 1 -- 172.21.15.98:0/12282193 learned_addr learned my addr 172.21.15.98:0/12282193 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a31400640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a3c070fe0 0x7f5a3c0cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:43.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a39e00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3c070c00 0x7f5a3c0cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a3a800640 1 -- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a3c070fe0 msgr2=0x7f5a3c0cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a3a800640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a3c070fe0 0x7f5a3c0cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a3a800640 1 -- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3c070c00 msgr2=0x7f5a3c0cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a3a800640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3c070c00 0x7f5a3c0cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a3a800640 1 -- 172.21.15.98:0/12282193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3c1f12f0 con 0x7f5a3c0cb570 2024-05-27T00:15:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a39e00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3c070c00 0x7f5a3c0cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a3a800640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c0cb570 0x7f5a3c15c9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c730 tx=0x7f5a3400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:43.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a32a00640 1 -- 172.21.15.98:0/12282193 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34018070 con 0x7f5a3c0cb570 2024-05-27T00:15:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a32a00640 1 -- 172.21.15.98:0/12282193 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a3400ce40 con 0x7f5a3c0cb570 2024-05-27T00:15:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a32a00640 1 -- 172.21.15.98:0/12282193 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400a860 con 0x7f5a3c0cb570 2024-05-27T00:15:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 -- 172.21.15.98:0/12282193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a3c15bdf0 con 0x7f5a3c0cb570 2024-05-27T00:15:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 -- 172.21.15.98:0/12282193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a3c1f0810 con 0x7f5a3c0cb570 2024-05-27T00:15:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.913+0000 7f5a40e00640 1 -- 172.21.15.98:0/12282193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a3c075cb0 con 0x7f5a3c0cb570 2024-05-27T00:15:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.917+0000 7f5a32a00640 1 -- 172.21.15.98:0/12282193 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a3401b070 con 0x7f5a3c0cb570 2024-05-27T00:15:43.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.917+0000 7f5a32a00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a180678f0 0x7f5a18069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.917+0000 7f5a32a00640 1 -- 172.21.15.98:0/12282193 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5a340851b0 con 0x7f5a3c0cb570 2024-05-27T00:15:43.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.917+0000 7f5a39e00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a180678f0 0x7f5a18069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:43.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.917+0000 7f5a39e00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a180678f0 0x7f5a18069db0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c000f70 tx=0x7f5a2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:43.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:43.921+0000 7f5a32a00640 1 -- 172.21.15.98:0/12282193 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f5a34055ce0 con 0x7f5a3c0cb570 2024-05-27T00:15:44.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.077+0000 7f5a40e00640 1 -- 172.21.15.98:0/12282193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a3c0c51b0 con 0x7f5a3c0cb570 2024-05-27T00:15:44.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.077+0000 7f5a32a00640 1 -- 172.21.15.98:0/12282193 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16581 (secure 0 0 0) 0x7f5a340555e0 con 0x7f5a3c0cb570 2024-05-27T00:15:44.081 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:44.082 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:26.579229+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:44.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 -- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a180678f0 msgr2=0x7f5a18069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a180678f0 0x7f5a18069db0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c000f70 tx=0x7f5a2c009040 comp rx=0 tx=0).stop 2024-05-27T00:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 -- 172.21.15.98:0/12282193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c0cb570 msgr2=0x7f5a3c15c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c0cb570 0x7f5a3c15c9c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c730 tx=0x7f5a3400cbf0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 -- 172.21.15.98:0/12282193 shutdown_connections 2024-05-27T00:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a3c0cb570 0x7f5a3c15c9c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a3c070fe0 0x7f5a3c0cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a180678f0 0x7f5a18069db0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 --2- 172.21.15.98:0/12282193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3c070c00 0x7f5a3c0cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 -- 172.21.15.98:0/12282193 >> 172.21.15.98:0/12282193 conn(0x7f5a3c072f90 msgr2=0x7f5a3c152860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:44.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 -- 172.21.15.98:0/12282193 shutdown_connections 2024-05-27T00:15:44.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.081+0000 7f5a40e00640 1 -- 172.21.15.98:0/12282193 wait complete. 2024-05-27T00:15:44.103 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T00:15:13.211703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T00:15:23.224464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T00:15:23.950896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T00:15:44.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T00:15:44.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1ef400640 1 Processor -- start 2024-05-27T00:15:44.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1ef400640 1 -- start start 2024-05-27T00:15:44.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1ef400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1e8157ff0 0x7fa1e806d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1ef400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1e806ee50 0x7fa1e806c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1ef400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1e80c7c00 con 0x7fa1e80734f0 2024-05-27T00:15:44.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1ef400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa1e8073aa0 con 0x7fa1e8157ff0 2024-05-27T00:15:44.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1ef400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa1e8072700 con 0x7fa1e806ee50 2024-05-27T00:15:44.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1eca00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1e8157ff0 0x7fa1e806d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1ed400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa1e80734f0 0x7fa1e80c8f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45814/0 (socket says 172.21.15.98:45814) 2024-05-27T00:15:44.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1ed400640 1 -- 172.21.15.98:0/2074200369 learned_addr learned my addr 172.21.15.98:0/2074200369 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.241+0000 7fa1eca00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1e8157ff0 0x7fa1e806d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53892/0 (socket says 172.21.15.98:53892) 2024-05-27T00:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1e7e00640 1 --2- 172.21.15.98:0/2074200369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1e806ee50 0x7fa1e806c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1eca00640 1 -- 172.21.15.98:0/2074200369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1e806ee50 msgr2=0x7fa1e806c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1eca00640 1 --2- 172.21.15.98:0/2074200369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1e806ee50 0x7fa1e806c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1eca00640 1 -- 172.21.15.98:0/2074200369 >> v1:172.21.15.98:6789/0 conn(0x7fa1e80734f0 legacy=0x7fa1e80c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1e7400640 1 -- 172.21.15.98:0/2074200369 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2657019276 0 0) 0x7fa1e80c7c00 con 0x7fa1e80734f0 2024-05-27T00:15:44.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1eca00640 1 -- 172.21.15.98:0/2074200369 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e81583d0 con 0x7fa1e8157ff0 2024-05-27T00:15:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1eca00640 1 --2- 172.21.15.98:0/2074200369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e8157ff0 0x7fa1e806d310 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80097c0 tx=0x7fa1d801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcba575a28940290 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1e7400640 1 -- 172.21.15.98:0/2074200369 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d80b2070 con 0x7fa1e8157ff0 2024-05-27T00:15:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1e7400640 1 -- 172.21.15.98:0/2074200369 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d80ad640 con 0x7fa1e8157ff0 2024-05-27T00:15:44.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1e7400640 1 -- 172.21.15.98:0/2074200369 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d80ada10 con 0x7fa1e8157ff0 2024-05-27T00:15:44.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1ef400640 1 -- 172.21.15.98:0/2074200369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e8157ff0 msgr2=0x7fa1e806d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.245+0000 7fa1ef400640 1 --2- 172.21.15.98:0/2074200369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e8157ff0 0x7fa1e806d310 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80097c0 tx=0x7fa1d801c4a0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 -- 172.21.15.98:0/2074200369 shutdown_connections 2024-05-27T00:15:44.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 --2- 172.21.15.98:0/2074200369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1e806ee50 0x7fa1e806c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 --2- 172.21.15.98:0/2074200369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e8157ff0 0x7fa1e806d310 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 -- 172.21.15.98:0/2074200369 >> 172.21.15.98:0/2074200369 conn(0x7fa1e80c9b60 msgr2=0x7fa1e80c9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:44.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 -- 172.21.15.98:0/2074200369 shutdown_connections 2024-05-27T00:15:44.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 -- 172.21.15.98:0/2074200369 wait complete. 2024-05-27T00:15:44.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 Processor -- start 2024-05-27T00:15:44.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 -- start start 2024-05-27T00:15:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e806ee50 0x7fa1e815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1e80734f0 0x7fa1e815b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1e8157ff0 0x7fa1e8145740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e8167a40 con 0x7fa1e806ee50 2024-05-27T00:15:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e8072c30 con 0x7fa1e8157ff0 2024-05-27T00:15:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ef400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa1e8167bc0 con 0x7fa1e80734f0 2024-05-27T00:15:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1eca00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e806ee50 0x7fa1e815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1eca00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e806ee50 0x7fa1e815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53898/0 (socket says 172.21.15.98:53898) 2024-05-27T00:15:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1eca00640 1 -- 172.21.15.98:0/1297275906 learned_addr learned my addr 172.21.15.98:0/1297275906 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1e7e00640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1e80734f0 0x7fa1e815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ed400640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1e8157ff0 0x7fa1e8145740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1eca00640 1 -- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1e80734f0 msgr2=0x7fa1e815b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1eca00640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1e80734f0 0x7fa1e815b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1eca00640 1 -- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1e8157ff0 msgr2=0x7fa1e8145740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1eca00640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1e8157ff0 0x7fa1e8145740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1eca00640 1 -- 172.21.15.98:0/1297275906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e80cbd30 con 0x7fa1e806ee50 2024-05-27T00:15:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1ed400640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1e8157ff0 0x7fa1e8145740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1eca00640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e806ee50 0x7fa1e815ae40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80b0040 tx=0x7fa1d8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1e4c00640 1 -- 172.21.15.98:0/1297275906 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d80b2040 con 0x7fa1e806ee50 2024-05-27T00:15:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1e4c00640 1 -- 172.21.15.98:0/1297275906 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d80bc070 con 0x7fa1e806ee50 2024-05-27T00:15:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.249+0000 7fa1e4c00640 1 -- 172.21.15.98:0/1297275906 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d80b6870 con 0x7fa1e806ee50 2024-05-27T00:15:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.253+0000 7fa1ef400640 1 -- 172.21.15.98:0/1297275906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1e80cbfd0 con 0x7fa1e806ee50 2024-05-27T00:15:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.253+0000 7fa1ef400640 1 -- 172.21.15.98:0/1297275906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1e80cc550 con 0x7fa1e806ee50 2024-05-27T00:15:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.253+0000 7fa1ef400640 1 -- 172.21.15.98:0/1297275906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1e8075cb0 con 0x7fa1e806ee50 2024-05-27T00:15:44.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.253+0000 7fa1e4c00640 1 -- 172.21.15.98:0/1297275906 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa1d80ad500 con 0x7fa1e806ee50 2024-05-27T00:15:44.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.253+0000 7fa1e4c00640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1c80678c0 0x7fa1c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.253+0000 7fa1e4c00640 1 -- 172.21.15.98:0/1297275906 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa1d811fdc0 con 0x7fa1e806ee50 2024-05-27T00:15:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.253+0000 7fa1e7e00640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1c80678c0 0x7fa1c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.257+0000 7fa1e7e00640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1c80678c0 0x7fa1c8069d80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa1d400b930 tx=0x7fa1d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.257+0000 7fa1e4c00640 1 -- 172.21.15.98:0/1297275906 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa1d8016090 con 0x7fa1e806ee50 2024-05-27T00:15:44.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 -- 172.21.15.98:0/1297275906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa1e8151c10 con 0x7fa1e806ee50 2024-05-27T00:15:44.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1e4c00640 1 -- 172.21.15.98:0/1297275906 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fa1d80f07e0 con 0x7fa1e806ee50 2024-05-27T00:15:44.419 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-27T00:15:44.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 -- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1c80678c0 msgr2=0x7fa1c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1c80678c0 0x7fa1c8069d80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa1d400b930 tx=0x7fa1d4008040 comp rx=0 tx=0).stop 2024-05-27T00:15:44.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 -- 172.21.15.98:0/1297275906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e806ee50 msgr2=0x7fa1e815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e806ee50 0x7fa1e815ae40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80b0040 tx=0x7fa1d8019240 comp rx=0 tx=0).stop 2024-05-27T00:15:44.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 -- 172.21.15.98:0/1297275906 shutdown_connections 2024-05-27T00:15:44.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1c80678c0 0x7fa1c8069d80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1e8157ff0 0x7fa1e8145740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1e80734f0 0x7fa1e815b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 --2- 172.21.15.98:0/1297275906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1e806ee50 0x7fa1e815ae40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 -- 172.21.15.98:0/1297275906 >> 172.21.15.98:0/1297275906 conn(0x7fa1e80c9b60 msgr2=0x7fa1e8151e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:44.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 -- 172.21.15.98:0/1297275906 shutdown_connections 2024-05-27T00:15:44.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.417+0000 7fa1ef400640 1 -- 172.21.15.98:0/1297275906 wait complete. 2024-05-27T00:15:44.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T00:15:44.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.577+0000 7f534c800640 1 Processor -- start 2024-05-27T00:15:44.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.577+0000 7f534c800640 1 -- start start 2024-05-27T00:15:44.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.577+0000 7f534c800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5344158640 0x7f5344158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f534c800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5344159010 con 0x7f5344152990 2024-05-27T00:15:44.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f534c800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f534415bc40 con 0x7f53440b1c20 2024-05-27T00:15:44.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f534c800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f534415c850 con 0x7f5344158640 2024-05-27T00:15:44.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f534a800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5344158640 0x7f5344158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f534a800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5344158640 0x7f5344158ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53902/0 (socket says 172.21.15.98:53902) 2024-05-27T00:15:44.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f534a800640 1 -- 172.21.15.98:0/3716521588 learned_addr learned my addr 172.21.15.98:0/3716521588 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:44.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f534a800640 1 -- 172.21.15.98:0/3716521588 >> v1:172.21.15.204:6789/0 conn(0x7f53440b1c20 legacy=0x7f53440c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:44.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f534a800640 1 -- 172.21.15.98:0/3716521588 >> v1:172.21.15.98:6789/0 conn(0x7f5344152990 legacy=0x7f534406e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f534a800640 1 -- 172.21.15.98:0/3716521588 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534415bf20 con 0x7f5344158640 2024-05-27T00:15:44.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f5348a00640 1 -- 172.21.15.98:0/3716521588 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 188712751 0 0) 0x7f5344159010 con 0x7f5344152990 2024-05-27T00:15:44.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f534a800640 1 --2- 172.21.15.98:0/3716521588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344158640 0x7f5344158ad0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5340009e50 tx=0x7f534001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24844a4a404490b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:44.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f5348a00640 1 -- 172.21.15.98:0/3716521588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53400a5700 con 0x7f5344158640 2024-05-27T00:15:44.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f5348a00640 1 -- 172.21.15.98:0/3716521588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53400a58a0 con 0x7f5344158640 2024-05-27T00:15:44.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.581+0000 7f5348a00640 1 -- 172.21.15.98:0/3716521588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53400a5c70 con 0x7f5344158640 2024-05-27T00:15:44.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 -- 172.21.15.98:0/3716521588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344158640 msgr2=0x7f5344158ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 --2- 172.21.15.98:0/3716521588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344158640 0x7f5344158ad0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5340009e50 tx=0x7f534001ca30 comp rx=0 tx=0).stop 2024-05-27T00:15:44.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 -- 172.21.15.98:0/3716521588 shutdown_connections 2024-05-27T00:15:44.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 --2- 172.21.15.98:0/3716521588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5344158640 0x7f5344158ad0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 -- 172.21.15.98:0/3716521588 >> 172.21.15.98:0/3716521588 conn(0x7f53440cbf00 msgr2=0x7f53440cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:44.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 -- 172.21.15.98:0/3716521588 shutdown_connections 2024-05-27T00:15:44.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 -- 172.21.15.98:0/3716521588 wait complete. 2024-05-27T00:15:44.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 Processor -- start 2024-05-27T00:15:44.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 -- start start 2024-05-27T00:15:44.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53440b1c20 0x7f53440c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5344152990 0x7f53440c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5344158640 0x7f53440c1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53440c87b0 con 0x7f53440b1c20 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f53440c8630 con 0x7f5344152990 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534c800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5344166660 con 0x7f5344158640 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f5349e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53440b1c20 0x7f53440c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f5349e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53440b1c20 0x7f53440c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53918/0 (socket says 172.21.15.98:53918) 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f5349e00640 1 -- 172.21.15.98:0/3985410493 learned_addr learned my addr 172.21.15.98:0/3985410493 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f5349400640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5344152990 0x7f53440c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f534a800640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5344158640 0x7f53440c1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f5349e00640 1 -- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5344158640 msgr2=0x7f53440c1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f5349e00640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5344158640 0x7f53440c1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f5349e00640 1 -- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5344152990 msgr2=0x7f53440c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f5349e00640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5344152990 0x7f53440c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.585+0000 7f5349e00640 1 -- 172.21.15.98:0/3985410493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53440c5bc0 con 0x7f53440b1c20 2024-05-27T00:15:44.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f5349e00640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53440b1c20 0x7f53440c54f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f533800c4b0 tx=0x7f533800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f533e000640 1 -- 172.21.15.98:0/3985410493 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338005480 con 0x7f53440b1c20 2024-05-27T00:15:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f533e000640 1 -- 172.21.15.98:0/3985410493 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5338003600 con 0x7f53440b1c20 2024-05-27T00:15:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f534c800640 1 -- 172.21.15.98:0/3985410493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53440c37d0 con 0x7f53440b1c20 2024-05-27T00:15:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f533e000640 1 -- 172.21.15.98:0/3985410493 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53380039f0 con 0x7f53440b1c20 2024-05-27T00:15:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f534c800640 1 -- 172.21.15.98:0/3985410493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53440c3d90 con 0x7f53440b1c20 2024-05-27T00:15:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f534c800640 1 -- 172.21.15.98:0/3985410493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53440c3f90 con 0x7f53440b1c20 2024-05-27T00:15:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f533e000640 1 -- 172.21.15.98:0/3985410493 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5338003b90 con 0x7f53440b1c20 2024-05-27T00:15:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f533e000640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53280678f0 0x7f5328069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f533e000640 1 -- 172.21.15.98:0/3985410493 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5338085930 con 0x7f53440b1c20 2024-05-27T00:15:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.589+0000 7f5349400640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53280678f0 0x7f5328069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.593+0000 7f5349400640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53280678f0 0x7f5328069db0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f53440b0f50 tx=0x7f5334009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:44.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.593+0000 7f533e000640 1 -- 172.21.15.98:0/3985410493 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f5338055e20 con 0x7f53440b1c20 2024-05-27T00:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.749+0000 7f534c800640 1 -- 172.21.15.98:0/3985410493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f53440ca700 con 0x7f53440b1c20 2024-05-27T00:15:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.749+0000 7f533e000640 1 -- 172.21.15.98:0/3985410493 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f533805a120 con 0x7f53440b1c20 2024-05-27T00:15:44.753 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-27T00:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 -- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53280678f0 msgr2=0x7f5328069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53280678f0 0x7f5328069db0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f53440b0f50 tx=0x7f5334009040 comp rx=0 tx=0).stop 2024-05-27T00:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 -- 172.21.15.98:0/3985410493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53440b1c20 msgr2=0x7f53440c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53440b1c20 0x7f53440c54f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f533800c4b0 tx=0x7f533800c970 comp rx=0 tx=0).stop 2024-05-27T00:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 -- 172.21.15.98:0/3985410493 shutdown_connections 2024-05-27T00:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5344158640 0x7f53440c1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53280678f0 0x7f5328069db0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5344152990 0x7f53440c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 --2- 172.21.15.98:0/3985410493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53440b1c20 0x7f53440c54f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 -- 172.21.15.98:0/3985410493 >> 172.21.15.98:0/3985410493 conn(0x7f53440cbf00 msgr2=0x7f534414a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 -- 172.21.15.98:0/3985410493 shutdown_connections 2024-05-27T00:15:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.753+0000 7f534c800640 1 -- 172.21.15.98:0/3985410493 wait complete. 2024-05-27T00:15:44.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T00:15:44.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.909+0000 7f1783200640 1 Processor -- start 2024-05-27T00:15:44.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.909+0000 7f1783200640 1 -- start start 2024-05-27T00:15:44.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1783200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1784070930 0x7f17840c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1783200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17840ca4e0 con 0x7f178415be60 2024-05-27T00:15:44.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1783200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17840c7350 con 0x7f1784070d10 2024-05-27T00:15:44.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1783200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f17840c83f0 con 0x7f1784070930 2024-05-27T00:15:44.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1782800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f178415be60 0x7f178406f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45824/0 (socket says 172.21.15.98:45824) 2024-05-27T00:15:44.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1782800640 1 -- 172.21.15.98:0/3402280909 learned_addr learned my addr 172.21.15.98:0/3402280909 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:44.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781e00640 1 --2- 172.21.15.98:0/3402280909 >> v2:172.21.15.204:3300/0 conn(0x7f1784070930 0x7f17840c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1772421743 0 0) 0x7f17840ca4e0 con 0x7f178415be60 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1768003530 con 0x7f178415be60 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 168814164 0 0) 0x7f1768003530 con 0x7f178415be60 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 >> v2:172.21.15.204:3300/0 conn(0x7f1784070930 msgr2=0x7f17840c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 --2- 172.21.15.98:0/3402280909 >> v2:172.21.15.204:3300/0 conn(0x7f1784070930 0x7f17840c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 >> v1:172.21.15.204:6789/0 conn(0x7f1784070d10 legacy=0x7f17840c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17840c7630 con 0x7f178415be60 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f177c001680 con 0x7f178415be60 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 >> v1:172.21.15.98:6789/0 conn(0x7f178415be60 legacy=0x7f178406f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17680039c0 0x7f1768003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1768004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17680051d0 0x7f176800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17840c83f0 con 0x7f1768004810 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f176800c080 con 0x7f17680051d0 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f176800c280 con 0x7f17680039c0 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1779400640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1768004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781e00640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17680039c0 0x7f1768003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1782800640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17680051d0 0x7f176800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1779400640 1 -- 172.21.15.98:0/3402280909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17680039c0 msgr2=0x7f1768003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1779400640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17680039c0 0x7f1768003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1779400640 1 -- 172.21.15.98:0/3402280909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17680051d0 msgr2=0x7f176800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1779400640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17680051d0 0x7f176800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1779400640 1 -- 172.21.15.98:0/3402280909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1768003530 con 0x7f1768004810 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1781e00640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17680039c0 0x7f1768003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1782800640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17680051d0 0x7f176800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:44.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.913+0000 7f1782800640 1 -- 172.21.15.98:0/3402280909 reap_dead start 2024-05-27T00:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1779400640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1768004c90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f176c001780 tx=0x7f176c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1abe056a77b36d5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c0013a0 con 0x7f1768004810 2024-05-27T00:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f176c0052f0 con 0x7f1768004810 2024-05-27T00:15:44.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1781400640 1 -- 172.21.15.98:0/3402280909 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c003840 con 0x7f1768004810 2024-05-27T00:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1783200640 1 -- 172.21.15.98:0/3402280909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 msgr2=0x7f1768004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1783200640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1768004c90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f176c001780 tx=0x7f176c01c400 comp rx=0 tx=0).stop 2024-05-27T00:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1783200640 1 -- 172.21.15.98:0/3402280909 shutdown_connections 2024-05-27T00:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1783200640 1 --2- 172.21.15.98:0/3402280909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1768004c90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1783200640 1 -- 172.21.15.98:0/3402280909 >> 172.21.15.98:0/3402280909 conn(0x7f17840cdf20 msgr2=0x7f17840ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1783200640 1 -- 172.21.15.98:0/3402280909 shutdown_connections 2024-05-27T00:15:44.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1783200640 1 -- 172.21.15.98:0/3402280909 wait complete. 2024-05-27T00:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1783200640 1 Processor -- start 2024-05-27T00:15:44.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.917+0000 7f1783200640 1 -- start start 2024-05-27T00:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1783200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1784159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1783200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17841542d0 0x7f1784154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1783200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17841f0c70 0x7f17841f10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1783200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17840c7100 con 0x7f1768004810 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1783200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17840c5830 con 0x7f17841f0c70 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1783200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f17840c7f60 con 0x7f17841542d0 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1781e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1784159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1781e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1784159b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53946/0 (socket says 172.21.15.98:53946) 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1781e00640 1 -- 172.21.15.98:0/2429441765 learned_addr learned my addr 172.21.15.98:0/2429441765 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1779400640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17841542d0 0x7f1784154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1782800640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17841f0c70 0x7f17841f10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1781e00640 1 -- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17841542d0 msgr2=0x7f1784154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1781e00640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17841542d0 0x7f1784154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1781e00640 1 -- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17841f0c70 msgr2=0x7f17841f10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1781e00640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17841f0c70 0x7f17841f10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1781e00640 1 -- 172.21.15.98:0/2429441765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17840af310 con 0x7f1768004810 2024-05-27T00:15:44.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1781e00640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1784159b60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1774009fc0 tx=0x7f1774010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f177aa00640 1 -- 172.21.15.98:0/2429441765 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177401a030 con 0x7f1768004810 2024-05-27T00:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f177aa00640 1 -- 172.21.15.98:0/2429441765 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17740039a0 con 0x7f1768004810 2024-05-27T00:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f177aa00640 1 -- 172.21.15.98:0/2429441765 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1774003d70 con 0x7f1768004810 2024-05-27T00:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1783200640 1 -- 172.21.15.98:0/2429441765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17840af5f0 con 0x7f1768004810 2024-05-27T00:15:44.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1783200640 1 -- 172.21.15.98:0/2429441765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17840afbb0 con 0x7f1768004810 2024-05-27T00:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f1783200640 1 -- 172.21.15.98:0/2429441765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17840af060 con 0x7f1768004810 2024-05-27T00:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.921+0000 7f177aa00640 1 -- 172.21.15.98:0/2429441765 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f177401d070 con 0x7f1768004810 2024-05-27T00:15:44.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.925+0000 7f177aa00640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17600678f0 0x7f1760069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.925+0000 7f177aa00640 1 -- 172.21.15.98:0/2429441765 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f1774086a90 con 0x7f1768004810 2024-05-27T00:15:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.925+0000 7f1779400640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17600678f0 0x7f1760069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:44.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.925+0000 7f1779400640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17600678f0 0x7f1760069db0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f176c004000 tx=0x7f176c0011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:44.929+0000 7f177aa00640 1 -- 172.21.15.98:0/2429441765 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f17740575d0 con 0x7f1768004810 2024-05-27T00:15:45.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.081+0000 7f1783200640 1 -- 172.21.15.98:0/2429441765 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1784075c40 con 0x7f1768004810 2024-05-27T00:15:45.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.081+0000 7f177aa00640 1 -- 172.21.15.98:0/2429441765 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f1774056ed0 con 0x7f1768004810 2024-05-27T00:15:45.085 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-27T00:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 -- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17600678f0 msgr2=0x7f1760069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17600678f0 0x7f1760069db0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f176c004000 tx=0x7f176c0011a0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 -- 172.21.15.98:0/2429441765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 msgr2=0x7f1784159b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1784159b60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1774009fc0 tx=0x7f1774010590 comp rx=0 tx=0).stop 2024-05-27T00:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 -- 172.21.15.98:0/2429441765 shutdown_connections 2024-05-27T00:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17600678f0 0x7f1760069db0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17841f0c70 0x7f17841f10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17841542d0 0x7f1784154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 --2- 172.21.15.98:0/2429441765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1768004810 0x7f1784159b60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 -- 172.21.15.98:0/2429441765 >> 172.21.15.98:0/2429441765 conn(0x7f17840cdf20 msgr2=0x7f17840c3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 -- 172.21.15.98:0/2429441765 shutdown_connections 2024-05-27T00:15:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.085+0000 7f1783200640 1 -- 172.21.15.98:0/2429441765 wait complete. 2024-05-27T00:15:45.106 INFO:tasks.mds_thrash:mds thrasher using random seed: 1716768945 2024-05-27T00:15:45.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:45.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc724e00640 1 Processor -- start 2024-05-27T00:15:45.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc724e00640 1 -- start start 2024-05-27T00:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc724e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc7200cb7e0 0x7fc720158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc724e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc7200b3ed0 0x7fc720158c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc724e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc72015be40 con 0x7fc720152a50 2024-05-27T00:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc724e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc7200c8a00 con 0x7fc7200cb7e0 2024-05-27T00:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc724e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fc7200c50e0 con 0x7fc7200b3ed0 2024-05-27T00:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc717e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc7200cb7e0 0x7fc720158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc71de00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc720152a50 0x7fc7200709a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45840/0 (socket says 172.21.15.98:45840) 2024-05-27T00:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc71de00640 1 -- 172.21.15.98:0/840088393 learned_addr learned my addr 172.21.15.98:0/840088393 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:45.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc717e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc7200cb7e0 0x7fc720158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53950/0 (socket says 172.21.15.98:53950) 2024-05-27T00:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc71e800640 1 --2- 172.21.15.98:0/840088393 >> v2:172.21.15.204:3300/0 conn(0x7fc7200b3ed0 0x7fc720158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc717e00640 1 -- 172.21.15.98:0/840088393 >> v2:172.21.15.204:3300/0 conn(0x7fc7200b3ed0 msgr2=0x7fc720158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc717e00640 1 --2- 172.21.15.98:0/840088393 >> v2:172.21.15.204:3300/0 conn(0x7fc7200b3ed0 0x7fc720158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc717e00640 1 -- 172.21.15.98:0/840088393 >> v1:172.21.15.98:6789/0 conn(0x7fc720152a50 legacy=0x7fc7200709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc717e00640 1 -- 172.21.15.98:0/840088393 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7200c8c50 con 0x7fc7200cb7e0 2024-05-27T00:15:45.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.241+0000 7fc71d400640 1 -- 172.21.15.98:0/840088393 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2655472290 0 0) 0x7fc72015be40 con 0x7fc720152a50 2024-05-27T00:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc717e00640 1 --2- 172.21.15.98:0/840088393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200cb7e0 0x7fc720158750 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc7200c8b80 tx=0x7fc70801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d24706cf2119c3c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:45.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc71d400640 1 -- 172.21.15.98:0/840088393 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7080b1070 con 0x7fc7200cb7e0 2024-05-27T00:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc71d400640 1 -- 172.21.15.98:0/840088393 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7080ac600 con 0x7fc7200cb7e0 2024-05-27T00:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc71d400640 1 -- 172.21.15.98:0/840088393 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7080ac9d0 con 0x7fc7200cb7e0 2024-05-27T00:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc724e00640 1 -- 172.21.15.98:0/840088393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200cb7e0 msgr2=0x7fc720158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc724e00640 1 --2- 172.21.15.98:0/840088393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200cb7e0 0x7fc720158750 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc7200c8b80 tx=0x7fc70801c3a0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc724e00640 1 -- 172.21.15.98:0/840088393 shutdown_connections 2024-05-27T00:15:45.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc724e00640 1 --2- 172.21.15.98:0/840088393 >> v2:172.21.15.204:3300/0 conn(0x7fc7200b3ed0 0x7fc720158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc724e00640 1 --2- 172.21.15.98:0/840088393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200cb7e0 0x7fc720158750 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc724e00640 1 -- 172.21.15.98:0/840088393 >> 172.21.15.98:0/840088393 conn(0x7fc720072f90 msgr2=0x7fc7200733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc724e00640 1 -- 172.21.15.98:0/840088393 shutdown_connections 2024-05-27T00:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc724e00640 1 -- 172.21.15.98:0/840088393 wait complete. 2024-05-27T00:15:45.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc724e00640 1 Processor -- start 2024-05-27T00:15:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.245+0000 7fc724e00640 1 -- start start 2024-05-27T00:15:45.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc724e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200b3ed0 0x7fc7201f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc724e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7200cb7e0 0x7fc7201f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc724e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc720152a50 0x7fc720147700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc724e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc720152e30 con 0x7fc7200b3ed0 2024-05-27T00:15:45.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc724e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc7200cb570 con 0x7fc7200cb7e0 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc724e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc7200c9c10 con 0x7fc720152a50 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc71de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200b3ed0 0x7fc7201f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc71de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200b3ed0 0x7fc7201f6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53952/0 (socket says 172.21.15.98:53952) 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc71de00640 1 -- 172.21.15.98:0/2811097922 learned_addr learned my addr 172.21.15.98:0/2811097922 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc717e00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7200cb7e0 0x7fc7201f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc71e800640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc720152a50 0x7fc720147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc71de00640 1 -- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc720152a50 msgr2=0x7fc720147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc71de00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc720152a50 0x7fc720147700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc71de00640 1 -- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7200cb7e0 msgr2=0x7fc7201f3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc71de00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7200cb7e0 0x7fc7201f3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc71de00640 1 -- 172.21.15.98:0/2811097922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7201f2e00 con 0x7fc7200b3ed0 2024-05-27T00:15:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc71de00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200b3ed0 0x7fc7201f6d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc71000c4b0 tx=0x7fc71000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc716000640 1 -- 172.21.15.98:0/2811097922 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710018070 con 0x7fc7200b3ed0 2024-05-27T00:15:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc716000640 1 -- 172.21.15.98:0/2811097922 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc710007500 con 0x7fc7200b3ed0 2024-05-27T00:15:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc716000640 1 -- 172.21.15.98:0/2811097922 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710004a30 con 0x7fc7200b3ed0 2024-05-27T00:15:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc724e00640 1 -- 172.21.15.98:0/2811097922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc72006b9b0 con 0x7fc7200b3ed0 2024-05-27T00:15:45.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc724e00640 1 -- 172.21.15.98:0/2811097922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc72006bf70 con 0x7fc7200b3ed0 2024-05-27T00:15:45.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.249+0000 7fc724e00640 1 -- 172.21.15.98:0/2811097922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc720075cb0 con 0x7fc7200b3ed0 2024-05-27T00:15:45.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.253+0000 7fc716000640 1 -- 172.21.15.98:0/2811097922 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc710004bd0 con 0x7fc7200b3ed0 2024-05-27T00:15:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.253+0000 7fc716000640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc6fc0678c0 0x7fc6fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.253+0000 7fc716000640 1 -- 172.21.15.98:0/2811097922 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc710084ed0 con 0x7fc7200b3ed0 2024-05-27T00:15:45.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.253+0000 7fc717e00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc6fc0678c0 0x7fc6fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.253+0000 7fc717e00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc6fc0678c0 0x7fc6fc069d80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc7080ae000 tx=0x7fc708008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:45.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.257+0000 7fc716000640 1 -- 172.21.15.98:0/2811097922 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc710055a10 con 0x7fc7200b3ed0 2024-05-27T00:15:45.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.417+0000 7fc724e00640 1 -- 172.21.15.98:0/2811097922 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc72014b3b0 con 0x7fc7200b3ed0 2024-05-27T00:15:45.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.417+0000 7fc716000640 1 -- 172.21.15.98:0/2811097922 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16581 (secure 0 0 0) 0x7fc710055310 con 0x7fc7200b3ed0 2024-05-27T00:15:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:45.421 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:26.579229+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:45.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 -- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc6fc0678c0 msgr2=0x7fc6fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc6fc0678c0 0x7fc6fc069d80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc7080ae000 tx=0x7fc708008040 comp rx=0 tx=0).stop 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 -- 172.21.15.98:0/2811097922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200b3ed0 msgr2=0x7fc7201f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200b3ed0 0x7fc7201f6d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc71000c4b0 tx=0x7fc71000c970 comp rx=0 tx=0).stop 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 -- 172.21.15.98:0/2811097922 shutdown_connections 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc720152a50 0x7fc720147700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc6fc0678c0 0x7fc6fc069d80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7200cb7e0 0x7fc7201f3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 --2- 172.21.15.98:0/2811097922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7200b3ed0 0x7fc7201f6d40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 -- 172.21.15.98:0/2811097922 >> 172.21.15.98:0/2811097922 conn(0x7fc720072f90 msgr2=0x7fc720073370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 -- 172.21.15.98:0/2811097922 shutdown_connections 2024-05-27T00:15:45.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.421+0000 7fc724e00640 1 -- 172.21.15.98:0/2811097922 wait complete. 2024-05-27T00:15:45.442 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T00:15:13.211703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T00:15:23.224464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T00:15:23.950896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T00:15:45.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T00:15:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.577+0000 7fa930600640 1 Processor -- start 2024-05-27T00:15:45.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.577+0000 7fa930600640 1 -- start start 2024-05-27T00:15:45.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa930600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa9280b1c20 0x7fa9280c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa930600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9280c8460 0x7fa928158640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa930600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa92815bc40 con 0x7fa928152990 2024-05-27T00:15:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa930600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa92815c850 con 0x7fa9280b1c20 2024-05-27T00:15:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa930600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa92815bf20 con 0x7fa9280c8460 2024-05-27T00:15:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa92d200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa9280b1c20 0x7fa9280c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa92e600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9280c8460 0x7fa928158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa92e600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9280c8460 0x7fa928158640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:46760/0 (socket says 172.21.15.98:46760) 2024-05-27T00:15:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa92d200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa9280b1c20 0x7fa9280c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:56556/0 (socket says 172.21.15.98:56556) 2024-05-27T00:15:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa92e600640 1 -- 172.21.15.98:0/4049879518 learned_addr learned my addr 172.21.15.98:0/4049879518 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa92d200640 1 -- 172.21.15.98:0/4049879518 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9280c8460 msgr2=0x7fa928158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa92d200640 1 --2- 172.21.15.98:0/4049879518 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9280c8460 0x7fa928158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa92d200640 1 -- 172.21.15.98:0/4049879518 >> v1:172.21.15.204:6789/0 conn(0x7fa928152990 legacy=0x7fa92806e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.581+0000 7fa92d200640 1 -- 172.21.15.98:0/4049879518 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92815c200 con 0x7fa9280b1c20 2024-05-27T00:15:45.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa92d200640 1 --2- 172.21.15.98:0/4049879518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9280b1c20 0x7fa9280c7f20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa9140097c0 tx=0x7fa91401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42fec3c13bd1a41b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa927e00640 1 -- 172.21.15.98:0/4049879518 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140b0070 con 0x7fa9280b1c20 2024-05-27T00:15:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa927e00640 1 -- 172.21.15.98:0/4049879518 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9140ab600 con 0x7fa9280b1c20 2024-05-27T00:15:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa927e00640 1 -- 172.21.15.98:0/4049879518 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140ab9d0 con 0x7fa9280b1c20 2024-05-27T00:15:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa930600640 1 -- 172.21.15.98:0/4049879518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9280b1c20 msgr2=0x7fa9280c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa930600640 1 --2- 172.21.15.98:0/4049879518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9280b1c20 0x7fa9280c7f20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa9140097c0 tx=0x7fa91401c4a0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa930600640 1 -- 172.21.15.98:0/4049879518 shutdown_connections 2024-05-27T00:15:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa930600640 1 --2- 172.21.15.98:0/4049879518 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9280c8460 0x7fa928158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa930600640 1 --2- 172.21.15.98:0/4049879518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9280b1c20 0x7fa9280c7f20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa930600640 1 -- 172.21.15.98:0/4049879518 >> 172.21.15.98:0/4049879518 conn(0x7fa9280cbf00 msgr2=0x7fa9280cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa930600640 1 -- 172.21.15.98:0/4049879518 shutdown_connections 2024-05-27T00:15:45.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa930600640 1 -- 172.21.15.98:0/4049879518 wait complete. 2024-05-27T00:15:45.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa930600640 1 Processor -- start 2024-05-27T00:15:45.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.585+0000 7fa930600640 1 -- start start 2024-05-27T00:15:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa930600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9280b1c20 0x7fa92813f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa930600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9280c8460 0x7fa92813ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa930600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa928152990 0x7fa9281492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa930600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9281657d0 con 0x7fa9280c8460 2024-05-27T00:15:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa930600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa928165650 con 0x7fa9280b1c20 2024-05-27T00:15:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa930600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa928165950 con 0x7fa928152990 2024-05-27T00:15:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9280c8460 0x7fa92813ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9280c8460 0x7fa92813ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53960/0 (socket says 172.21.15.98:53960) 2024-05-27T00:15:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92d200640 1 -- 172.21.15.98:0/9906967 learned_addr learned my addr 172.21.15.98:0/9906967 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92dc00640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9280b1c20 0x7fa92813f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92e600640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa928152990 0x7fa9281492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92d200640 1 -- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa928152990 msgr2=0x7fa9281492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92d200640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa928152990 0x7fa9281492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92d200640 1 -- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9280b1c20 msgr2=0x7fa92813f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92d200640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9280b1c20 0x7fa92813f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92d200640 1 -- 172.21.15.98:0/9906967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9281408e0 con 0x7fa9280c8460 2024-05-27T00:15:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92e600640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa928152990 0x7fa9281492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:45.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa92d200640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9280c8460 0x7fa92813ed20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa91401a0d0 tx=0x7fa914019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa925600640 1 -- 172.21.15.98:0/9906967 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140ba070 con 0x7fa9280c8460 2024-05-27T00:15:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa925600640 1 -- 172.21.15.98:0/9906967 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9140b43f0 con 0x7fa9280c8460 2024-05-27T00:15:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa930600640 1 -- 172.21.15.98:0/9906967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9280c6260 con 0x7fa9280c8460 2024-05-27T00:15:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa925600640 1 -- 172.21.15.98:0/9906967 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140b0040 con 0x7fa9280c8460 2024-05-27T00:15:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa930600640 1 -- 172.21.15.98:0/9906967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa928140320 con 0x7fa9280c8460 2024-05-27T00:15:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa930600640 1 -- 172.21.15.98:0/9906967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa928075cb0 con 0x7fa9280c8460 2024-05-27T00:15:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.589+0000 7fa925600640 1 -- 172.21.15.98:0/9906967 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9140bd070 con 0x7fa9280c8460 2024-05-27T00:15:45.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.593+0000 7fa925600640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9080678c0 0x7fa908069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.593+0000 7fa925600640 1 -- 172.21.15.98:0/9906967 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa91411ec10 con 0x7fa9280c8460 2024-05-27T00:15:45.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.593+0000 7fa92dc00640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9080678c0 0x7fa908069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.593+0000 7fa92dc00640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9080678c0 0x7fa908069d80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa91c001e00 tx=0x7fa91c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:45.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.597+0000 7fa925600640 1 -- 172.21.15.98:0/9906967 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa9280c6260 con 0x7fa9280c8460 2024-05-27T00:15:45.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.753+0000 7fa930600640 1 -- 172.21.15.98:0/9906967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa928158860 con 0x7fa9280c8460 2024-05-27T00:15:45.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.753+0000 7fa925600640 1 -- 172.21.15.98:0/9906967 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fa914016090 con 0x7fa9280c8460 2024-05-27T00:15:45.758 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 -- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9080678c0 msgr2=0x7fa908069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9080678c0 0x7fa908069d80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa91c001e00 tx=0x7fa91c009040 comp rx=0 tx=0).stop 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 -- 172.21.15.98:0/9906967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9280c8460 msgr2=0x7fa92813ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9280c8460 0x7fa92813ed20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa91401a0d0 tx=0x7fa914019240 comp rx=0 tx=0).stop 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 -- 172.21.15.98:0/9906967 shutdown_connections 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa928152990 0x7fa9281492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9280c8460 0x7fa92813ed20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9080678c0 0x7fa908069d80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 --2- 172.21.15.98:0/9906967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9280b1c20 0x7fa92813f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 -- 172.21.15.98:0/9906967 >> 172.21.15.98:0/9906967 conn(0x7fa9280cbf00 msgr2=0x7fa9280716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:45.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 -- 172.21.15.98:0/9906967 shutdown_connections 2024-05-27T00:15:45.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.757+0000 7fa930600640 1 -- 172.21.15.98:0/9906967 wait complete. 2024-05-27T00:15:45.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T00:15:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.913+0000 7fb7c6400640 1 Processor -- start 2024-05-27T00:15:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.913+0000 7fb7c6400640 1 -- start start 2024-05-27T00:15:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.913+0000 7fb7c6400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb7c0152990 0x7fb7c006e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.913+0000 7fb7c6400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c006ecc0 0x7fb7c00c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.913+0000 7fb7c6400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7c015bc40 con 0x7fb7c00b1c20 2024-05-27T00:15:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.913+0000 7fb7c6400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb7c015c850 con 0x7fb7c0152990 2024-05-27T00:15:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.913+0000 7fb7c6400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb7c015bf20 con 0x7fb7c006ecc0 2024-05-27T00:15:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.917+0000 7fb7bf400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb7c0152990 0x7fb7c006e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.917+0000 7fb7b6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c006ecc0 0x7fb7c00c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.917+0000 7fb7bfe00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fb7c00b1c20 0x7fb7c0158640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:58188/0 (socket says 172.21.15.98:58188) 2024-05-27T00:15:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.917+0000 7fb7bfe00640 1 -- 172.21.15.98:0/3170127116 learned_addr learned my addr 172.21.15.98:0/3170127116 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.917+0000 7fb7bf400640 1 -- 172.21.15.98:0/3170127116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c006ecc0 msgr2=0x7fb7c00c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.917+0000 7fb7bf400640 1 --2- 172.21.15.98:0/3170127116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c006ecc0 0x7fb7c00c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.917+0000 7fb7bf400640 1 -- 172.21.15.98:0/3170127116 >> v1:172.21.15.204:6789/0 conn(0x7fb7c00b1c20 legacy=0x7fb7c0158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.917+0000 7fb7bf400640 1 -- 172.21.15.98:0/3170127116 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c015c200 con 0x7fb7c0152990 2024-05-27T00:15:45.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.917+0000 7fb7bf400640 1 --2- 172.21.15.98:0/3170127116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7c0152990 0x7fb7c006e780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00097c0 tx=0x7fb7b001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30ccd258ead90e15 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:45.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.917+0000 7fb7bea00640 1 -- 172.21.15.98:0/3170127116 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b00b0070 con 0x7fb7c0152990 2024-05-27T00:15:45.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7bea00640 1 -- 172.21.15.98:0/3170127116 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b00ab600 con 0x7fb7c0152990 2024-05-27T00:15:45.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7bea00640 1 -- 172.21.15.98:0/3170127116 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b00ab9d0 con 0x7fb7c0152990 2024-05-27T00:15:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 -- 172.21.15.98:0/3170127116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7c0152990 msgr2=0x7fb7c006e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 --2- 172.21.15.98:0/3170127116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7c0152990 0x7fb7c006e780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00097c0 tx=0x7fb7b001c4a0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 -- 172.21.15.98:0/3170127116 shutdown_connections 2024-05-27T00:15:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 --2- 172.21.15.98:0/3170127116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c006ecc0 0x7fb7c00c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 --2- 172.21.15.98:0/3170127116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7c0152990 0x7fb7c006e780 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 -- 172.21.15.98:0/3170127116 >> 172.21.15.98:0/3170127116 conn(0x7fb7c00cbf00 msgr2=0x7fb7c00cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 -- 172.21.15.98:0/3170127116 shutdown_connections 2024-05-27T00:15:45.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 -- 172.21.15.98:0/3170127116 wait complete. 2024-05-27T00:15:45.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 Processor -- start 2024-05-27T00:15:45.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 -- start start 2024-05-27T00:15:45.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c006ecc0 0x7fb7c01444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c00b1c20 0x7fb7c0159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7c0152990 0x7fb7c0159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01659e0 con 0x7fb7c006ecc0 2024-05-27T00:15:45.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c0165860 con 0x7fb7c0152990 2024-05-27T00:15:45.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7c6400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb7c0165b60 con 0x7fb7c00b1c20 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7bf400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c006ecc0 0x7fb7c01444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7bf400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c006ecc0 0x7fb7c01444a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53974/0 (socket says 172.21.15.98:53974) 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7bf400640 1 -- 172.21.15.98:0/2852584677 learned_addr learned my addr 172.21.15.98:0/2852584677 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7b6a00640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c00b1c20 0x7fb7c0159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.921+0000 7fb7bfe00640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7c0152990 0x7fb7c0159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7bf400640 1 -- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c00b1c20 msgr2=0x7fb7c0159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7bf400640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c00b1c20 0x7fb7c0159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7bf400640 1 -- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7c0152990 msgr2=0x7fb7c0159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7bf400640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7c0152990 0x7fb7c0159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7bf400640 1 -- 172.21.15.98:0/2852584677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c015aea0 con 0x7fb7c006ecc0 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7bfe00640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7c0152990 0x7fb7c0159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7b6a00640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c00b1c20 0x7fb7c0159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:45.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7bf400640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c006ecc0 0x7fb7c01444a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00097c0 tx=0x7fb7b0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:45.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7b7e00640 1 -- 172.21.15.98:0/2852584677 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b00b0040 con 0x7fb7c006ecc0 2024-05-27T00:15:45.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7b7e00640 1 -- 172.21.15.98:0/2852584677 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b00ba070 con 0x7fb7c006ecc0 2024-05-27T00:15:45.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7b7e00640 1 -- 172.21.15.98:0/2852584677 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b00b4830 con 0x7fb7c006ecc0 2024-05-27T00:15:45.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7c6400640 1 -- 172.21.15.98:0/2852584677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c01f3bf0 con 0x7fb7c006ecc0 2024-05-27T00:15:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7c6400640 1 -- 172.21.15.98:0/2852584677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c01449e0 con 0x7fb7c006ecc0 2024-05-27T00:15:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7c6400640 1 -- 172.21.15.98:0/2852584677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c0075cb0 con 0x7fb7c006ecc0 2024-05-27T00:15:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7b7e00640 1 -- 172.21.15.98:0/2852584677 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb7b00b49d0 con 0x7fb7c006ecc0 2024-05-27T00:15:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7b7e00640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb79c0678c0 0x7fb79c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:45.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7b7e00640 1 -- 172.21.15.98:0/2852584677 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fb7b011fc70 con 0x7fb7c006ecc0 2024-05-27T00:15:45.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.925+0000 7fb7b6a00640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb79c0678c0 0x7fb79c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:45.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.929+0000 7fb7b6a00640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb79c0678c0 0x7fb79c069d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00cbbb0 tx=0x7fb7a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:45.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:45.929+0000 7fb7b7e00640 1 -- 172.21.15.98:0/2852584677 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb7c01f3bf0 con 0x7fb7c006ecc0 2024-05-27T00:15:46.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.085+0000 7fb7c6400640 1 -- 172.21.15.98:0/2852584677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb7c0145160 con 0x7fb7c006ecc0 2024-05-27T00:15:46.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.085+0000 7fb7b7e00640 1 -- 172.21.15.98:0/2852584677 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fb7c015aea0 con 0x7fb7c006ecc0 2024-05-27T00:15:46.088 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-27T00:15:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.085+0000 7fb7c6400640 1 -- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb79c0678c0 msgr2=0x7fb79c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.085+0000 7fb7c6400640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb79c0678c0 0x7fb79c069d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00cbbb0 tx=0x7fb7a8008040 comp rx=0 tx=0).stop 2024-05-27T00:15:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.085+0000 7fb7c6400640 1 -- 172.21.15.98:0/2852584677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c006ecc0 msgr2=0x7fb7c01444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.085+0000 7fb7c6400640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c006ecc0 0x7fb7c01444a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00097c0 tx=0x7fb7b0019240 comp rx=0 tx=0).stop 2024-05-27T00:15:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.089+0000 7fb7c6400640 1 -- 172.21.15.98:0/2852584677 shutdown_connections 2024-05-27T00:15:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.089+0000 7fb7c6400640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb79c0678c0 0x7fb79c069d80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.089+0000 7fb7c6400640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7c0152990 0x7fb7c0159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.089+0000 7fb7c6400640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7c00b1c20 0x7fb7c0159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.089+0000 7fb7c6400640 1 --2- 172.21.15.98:0/2852584677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c006ecc0 0x7fb7c01444a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.089+0000 7fb7c6400640 1 -- 172.21.15.98:0/2852584677 >> 172.21.15.98:0/2852584677 conn(0x7fb7c00cbf00 msgr2=0x7fb7c00cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.089+0000 7fb7c6400640 1 -- 172.21.15.98:0/2852584677 shutdown_connections 2024-05-27T00:15:46.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.089+0000 7fb7c6400640 1 -- 172.21.15.98:0/2852584677 wait complete. 2024-05-27T00:15:46.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T00:15:46.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.245+0000 7f7806600640 1 Processor -- start 2024-05-27T00:15:46.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.245+0000 7f7806600640 1 -- start start 2024-05-27T00:15:46.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f7806600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f78000b0060 0x7f78000c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f7806600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780015eb90 con 0x7f780015a800 2024-05-27T00:15:46.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f7806600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78000c33e0 con 0x7f78000c40a0 2024-05-27T00:15:46.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f7806600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f780015ac90 con 0x7f78000b0060 2024-05-27T00:15:46.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f77ffe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f780015a800 0x7f780015e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45842/0 (socket says 172.21.15.98:45842) 2024-05-27T00:15:46.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f77ffe00640 1 -- 172.21.15.98:0/633104468 learned_addr learned my addr 172.21.15.98:0/633104468 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f77ff400640 1 --2- 172.21.15.98:0/633104468 >> v2:172.21.15.204:3300/0 conn(0x7f78000b0060 0x7f78000c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f77fea00640 1 -- 172.21.15.98:0/633104468 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2121263680 0 0) 0x7f780015eb90 con 0x7f780015a800 2024-05-27T00:15:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f77fea00640 1 -- 172.21.15.98:0/633104468 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77e4003530 con 0x7f780015a800 2024-05-27T00:15:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f77ff400640 1 -- 172.21.15.98:0/633104468 >> v1:172.21.15.204:6789/0 conn(0x7f78000c40a0 legacy=0x7f780015dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f77ff400640 1 -- 172.21.15.98:0/633104468 >> v1:172.21.15.98:6789/0 conn(0x7f780015a800 legacy=0x7f780015e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.249+0000 7f77ff400640 1 -- 172.21.15.98:0/633104468 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78000c36c0 con 0x7f78000b0060 2024-05-27T00:15:46.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f77ff400640 1 --2- 172.21.15.98:0/633104468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78000b0060 0x7f78000c4f80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f77f00097f0 tx=0x7f77f001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f2d8c8cf2177b73 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:46.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f77fea00640 1 -- 172.21.15.98:0/633104468 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f00b0070 con 0x7f78000b0060 2024-05-27T00:15:46.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f77fea00640 1 -- 172.21.15.98:0/633104468 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f00ab600 con 0x7f78000b0060 2024-05-27T00:15:46.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f77fea00640 1 -- 172.21.15.98:0/633104468 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f00ab9b0 con 0x7f78000b0060 2024-05-27T00:15:46.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f7806600640 1 -- 172.21.15.98:0/633104468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78000b0060 msgr2=0x7f78000c4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f7806600640 1 --2- 172.21.15.98:0/633104468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78000b0060 0x7f78000c4f80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f77f00097f0 tx=0x7f77f001c410 comp rx=0 tx=0).stop 2024-05-27T00:15:46.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f7806600640 1 -- 172.21.15.98:0/633104468 shutdown_connections 2024-05-27T00:15:46.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f7806600640 1 --2- 172.21.15.98:0/633104468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78000b0060 0x7f78000c4f80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f7806600640 1 -- 172.21.15.98:0/633104468 >> 172.21.15.98:0/633104468 conn(0x7f78000ce170 msgr2=0x7f78000ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:46.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f7806600640 1 -- 172.21.15.98:0/633104468 shutdown_connections 2024-05-27T00:15:46.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f7806600640 1 -- 172.21.15.98:0/633104468 wait complete. 2024-05-27T00:15:46.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f7806600640 1 Processor -- start 2024-05-27T00:15:46.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.253+0000 7f7806600640 1 -- start start 2024-05-27T00:15:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f7806600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78000b0060 0x7f7800153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f7806600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78000c40a0 0x7f78000b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f7806600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f780015a800 0x7f78000b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f7806600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78000ae8c0 con 0x7f780015a800 2024-05-27T00:15:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f7806600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78000c4af0 con 0x7f78000b0060 2024-05-27T00:15:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f7806600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f78000aea40 con 0x7f78000c40a0 2024-05-27T00:15:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f780015a800 0x7f78000b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f780015a800 0x7f78000b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53984/0 (socket says 172.21.15.98:53984) 2024-05-27T00:15:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77ffe00640 1 -- 172.21.15.98:0/2083492677 learned_addr learned my addr 172.21.15.98:0/2083492677 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:46.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77ff400640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78000b0060 0x7f7800153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77f6a00640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78000c40a0 0x7f78000b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77ffe00640 1 -- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78000c40a0 msgr2=0x7f78000b2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77ffe00640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78000c40a0 0x7f78000b2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77ffe00640 1 -- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78000b0060 msgr2=0x7f7800153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77ffe00640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78000b0060 0x7f7800153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77ffe00640 1 -- 172.21.15.98:0/2083492677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78001f42d0 con 0x7f780015a800 2024-05-27T00:15:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77f6a00640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78000c40a0 0x7f78000b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:46.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77ffe00640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f780015a800 0x7f78000b3450 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f77f800ca20 tx=0x7f77f800cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:46.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77f7e00640 1 -- 172.21.15.98:0/2083492677 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8017070 con 0x7f780015a800 2024-05-27T00:15:46.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77f7e00640 1 -- 172.21.15.98:0/2083492677 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f800abd0 con 0x7f780015a800 2024-05-27T00:15:46.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f7806600640 1 -- 172.21.15.98:0/2083492677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78001f4540 con 0x7f780015a800 2024-05-27T00:15:46.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77f7e00640 1 -- 172.21.15.98:0/2083492677 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8012400 con 0x7f780015a800 2024-05-27T00:15:46.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f7806600640 1 -- 172.21.15.98:0/2083492677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78000b29f0 con 0x7f780015a800 2024-05-27T00:15:46.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f7806600640 1 -- 172.21.15.98:0/2083492677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7800075cb0 con 0x7f780015a800 2024-05-27T00:15:46.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.257+0000 7f77f7e00640 1 -- 172.21.15.98:0/2083492677 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f77f801a070 con 0x7f780015a800 2024-05-27T00:15:46.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.261+0000 7f77f7e00640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f77dc0678c0 0x7f77dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.261+0000 7f77f7e00640 1 -- 172.21.15.98:0/2083492677 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f77f8085980 con 0x7f780015a800 2024-05-27T00:15:46.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.261+0000 7f77ff400640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f77dc0678c0 0x7f77dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.261+0000 7f77ff400640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f77dc0678c0 0x7f77dc069d80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77f00ad000 tx=0x7f77f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:46.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.265+0000 7f77f7e00640 1 -- 172.21.15.98:0/2083492677 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f77f8055e70 con 0x7f780015a800 2024-05-27T00:15:46.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.417+0000 7f7806600640 1 -- 172.21.15.98:0/2083492677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78000ca3a0 con 0x7f780015a800 2024-05-27T00:15:46.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f77f7e00640 1 -- 172.21.15.98:0/2083492677 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f77f805a170 con 0x7f780015a800 2024-05-27T00:15:46.423 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-27T00:15:46.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 -- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f77dc0678c0 msgr2=0x7f77dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f77dc0678c0 0x7f77dc069d80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77f00ad000 tx=0x7f77f0008040 comp rx=0 tx=0).stop 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 -- 172.21.15.98:0/2083492677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f780015a800 msgr2=0x7f78000b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f780015a800 0x7f78000b3450 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f77f800ca20 tx=0x7f77f800cee0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 -- 172.21.15.98:0/2083492677 shutdown_connections 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f780015a800 0x7f78000b3450 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78000c40a0 0x7f78000b2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f77dc0678c0 0x7f77dc069d80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 --2- 172.21.15.98:0/2083492677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78000b0060 0x7f7800153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 -- 172.21.15.98:0/2083492677 >> 172.21.15.98:0/2083492677 conn(0x7f78000ce170 msgr2=0x7f7800159850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 -- 172.21.15.98:0/2083492677 shutdown_connections 2024-05-27T00:15:46.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.421+0000 7f7806600640 1 -- 172.21.15.98:0/2083492677 wait complete. 2024-05-27T00:15:46.444 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-05-27T00:15:46.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:15:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f458da00640 1 Processor -- start 2024-05-27T00:15:46.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f458da00640 1 -- start start 2024-05-27T00:15:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f458da00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 0x7f45880c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f458da00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4588071010 con 0x7f45880c36c0 2024-05-27T00:15:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f458da00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45880c88c0 con 0x7f45880b3260 2024-05-27T00:15:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f458da00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f45880c8b80 con 0x7f4588070660 2024-05-27T00:15:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f458d000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f45880c36c0 0x7f45880c9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45854/0 (socket says 172.21.15.98:45854) 2024-05-27T00:15:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f458d000640 1 -- 172.21.15.98:0/1836583217 learned_addr learned my addr 172.21.15.98:0/1836583217 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:46.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f457f400640 1 --2- 172.21.15.98:0/1836583217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 0x7f45880c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f4587400640 1 -- 172.21.15.98:0/1836583217 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 579604493 0 0) 0x7f45800014a0 con 0x7f45880c36c0 2024-05-27T00:15:46.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f4587400640 1 -- 172.21.15.98:0/1836583217 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f456c003530 con 0x7f45880c36c0 2024-05-27T00:15:46.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f4587400640 1 -- 172.21.15.98:0/1836583217 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3271716747 0 0) 0x7f45880c88c0 con 0x7f45880b3260 2024-05-27T00:15:46.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f4587400640 1 -- 172.21.15.98:0/1836583217 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45800014a0 con 0x7f45880b3260 2024-05-27T00:15:46.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f4587400640 1 -- 172.21.15.98:0/1836583217 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3495132281 0 0) 0x7f456c003530 con 0x7f45880c36c0 2024-05-27T00:15:46.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f457f400640 1 -- 172.21.15.98:0/1836583217 >> v1:172.21.15.204:6789/0 conn(0x7f45880b3260 legacy=0x7f45880c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f457f400640 1 -- 172.21.15.98:0/1836583217 >> v1:172.21.15.98:6789/0 conn(0x7f45880c36c0 legacy=0x7f45880c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.581+0000 7f457f400640 1 -- 172.21.15.98:0/1836583217 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45880c8e60 con 0x7f4588070660 2024-05-27T00:15:46.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f457f400640 1 --2- 172.21.15.98:0/1836583217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 0x7f45880c3180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f45700097f0 tx=0x7f457001ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ef2409dc919e5d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:46.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f4587400640 1 -- 172.21.15.98:0/1836583217 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45700b5070 con 0x7f4588070660 2024-05-27T00:15:46.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f4587400640 1 -- 172.21.15.98:0/1836583217 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45700a8670 con 0x7f4588070660 2024-05-27T00:15:46.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f4587400640 1 -- 172.21.15.98:0/1836583217 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580001bd0 con 0x7f4588070660 2024-05-27T00:15:46.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f458da00640 1 -- 172.21.15.98:0/1836583217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 msgr2=0x7f45880c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f458da00640 1 --2- 172.21.15.98:0/1836583217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 0x7f45880c3180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f45700097f0 tx=0x7f457001ebf0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f458da00640 1 -- 172.21.15.98:0/1836583217 shutdown_connections 2024-05-27T00:15:46.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f458da00640 1 --2- 172.21.15.98:0/1836583217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 0x7f45880c3180 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f458da00640 1 -- 172.21.15.98:0/1836583217 >> 172.21.15.98:0/1836583217 conn(0x7f45880cdeb0 msgr2=0x7f45880ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:46.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f458da00640 1 -- 172.21.15.98:0/1836583217 shutdown_connections 2024-05-27T00:15:46.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.585+0000 7f458da00640 1 -- 172.21.15.98:0/1836583217 wait complete. 2024-05-27T00:15:46.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 Processor -- start 2024-05-27T00:15:46.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 -- start start 2024-05-27T00:15:46.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 0x7f458806b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45880c36c0 0x7f45881e4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45881f19c0 0x7f45880b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45880c9940 con 0x7f45881f19c0 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f45880c9420 con 0x7f45880c36c0 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f45880c9ac0 con 0x7f4588070660 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45881f19c0 0x7f45880b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45881f19c0 0x7f45880b22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54000/0 (socket says 172.21.15.98:54000) 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458d000640 1 -- 172.21.15.98:0/4097538352 learned_addr learned my addr 172.21.15.98:0/4097538352 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f4587e00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 0x7f458806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f457f400640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45880c36c0 0x7f45881e4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458d000640 1 -- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 msgr2=0x7f458806b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458d000640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 0x7f458806b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458d000640 1 -- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45880c36c0 msgr2=0x7f45881e4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458d000640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45880c36c0 0x7f45881e4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458d000640 1 -- 172.21.15.98:0/4097538352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45881e65d0 con 0x7f45881f19c0 2024-05-27T00:15:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f4587e00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 0x7f458806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:46.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458d000640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45881f19c0 0x7f45880b22a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4580001b60 tx=0x7f458000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:46.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f4584c00640 1 -- 172.21.15.98:0/4097538352 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580012400 con 0x7f45881f19c0 2024-05-27T00:15:46.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f4584c00640 1 -- 172.21.15.98:0/4097538352 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45800125c0 con 0x7f45881f19c0 2024-05-27T00:15:46.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f4584c00640 1 -- 172.21.15.98:0/4097538352 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45800129b0 con 0x7f45881f19c0 2024-05-27T00:15:46.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 -- 172.21.15.98:0/4097538352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45881e68b0 con 0x7f45881f19c0 2024-05-27T00:15:46.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 -- 172.21.15.98:0/4097538352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45881e49f0 con 0x7f45881f19c0 2024-05-27T00:15:46.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.589+0000 7f458da00640 1 -- 172.21.15.98:0/4097538352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45881e4bf0 con 0x7f45881f19c0 2024-05-27T00:15:46.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.593+0000 7f4584c00640 1 -- 172.21.15.98:0/4097538352 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4580012b50 con 0x7f45881f19c0 2024-05-27T00:15:46.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.593+0000 7f4584c00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45640678f0 0x7f4564069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.593+0000 7f4584c00640 1 -- 172.21.15.98:0/4097538352 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f4580086670 con 0x7f45881f19c0 2024-05-27T00:15:46.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.593+0000 7f4587e00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45640678f0 0x7f4564069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.593+0000 7f4587e00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45640678f0 0x7f4564069db0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f45780011f0 tx=0x7f4578009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:46.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.597+0000 7f4584c00640 1 -- 172.21.15.98:0/4097538352 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f45881e4bf0 con 0x7f45881f19c0 2024-05-27T00:15:46.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.785+0000 7f458da00640 1 -- 172.21.15.98:0/4097538352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4588075c40 con 0x7f45881f19c0 2024-05-27T00:15:46.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.785+0000 7f4584c00640 1 -- 172.21.15.98:0/4097538352 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f4588071010 con 0x7f45881f19c0 2024-05-27T00:15:46.790 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:46.790 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:15:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:15:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 -- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45640678f0 msgr2=0x7f4564069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45640678f0 0x7f4564069db0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f45780011f0 tx=0x7f4578009040 comp rx=0 tx=0).stop 2024-05-27T00:15:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 -- 172.21.15.98:0/4097538352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45881f19c0 msgr2=0x7f45880b22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45881f19c0 0x7f45880b22a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4580001b60 tx=0x7f458000a430 comp rx=0 tx=0).stop 2024-05-27T00:15:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 -- 172.21.15.98:0/4097538352 shutdown_connections 2024-05-27T00:15:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45881f19c0 0x7f45880b22a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45640678f0 0x7f4564069db0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45880c36c0 0x7f45881e4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 --2- 172.21.15.98:0/4097538352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4588070660 0x7f458806b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 -- 172.21.15.98:0/4097538352 >> 172.21.15.98:0/4097538352 conn(0x7f45880cdeb0 msgr2=0x7f45880ca690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:46.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 -- 172.21.15.98:0/4097538352 shutdown_connections 2024-05-27T00:15:46.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.789+0000 7f458da00640 1 -- 172.21.15.98:0/4097538352 wait complete. 2024-05-27T00:15:46.810 INFO:tasks.mds_thrash:Ready to start thrashing 2024-05-27T00:15:46.810 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-05-27T00:15:46.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:15:46.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.941+0000 7f064c000640 1 Processor -- start 2024-05-27T00:15:46.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.941+0000 7f064c000640 1 -- start start 2024-05-27T00:15:46.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f064c000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f06440737f0 0x7f0644166150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f064c000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f064406cb30 con 0x7f064406b970 2024-05-27T00:15:46.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f064c000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06440c7b20 con 0x7f0644158080 2024-05-27T00:15:46.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f064c000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f06440c9820 con 0x7f06440737f0 2024-05-27T00:15:46.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0648c00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f064406b970 0x7f06440730e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45860/0 (socket says 172.21.15.98:45860) 2024-05-27T00:15:46.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0648c00640 1 -- 172.21.15.98:0/2828819298 learned_addr learned my addr 172.21.15.98:0/2828819298 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:46.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f064a000640 1 --2- 172.21.15.98:0/2828819298 >> v2:172.21.15.204:3300/0 conn(0x7f06440737f0 0x7f0644166150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3354001539 0 0) 0x7f064406cb30 con 0x7f064406b970 2024-05-27T00:15:46.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f062c003530 con 0x7f064406b970 2024-05-27T00:15:46.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2034946041 0 0) 0x7f062c003530 con 0x7f064406b970 2024-05-27T00:15:46.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 >> v2:172.21.15.204:3300/0 conn(0x7f06440737f0 msgr2=0x7f0644166150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:46.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 --2- 172.21.15.98:0/2828819298 >> v2:172.21.15.204:3300/0 conn(0x7f06440737f0 0x7f0644166150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 >> v1:172.21.15.204:6789/0 conn(0x7f0644158080 legacy=0x7f06440c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064406c570 con 0x7f064406b970 2024-05-27T00:15:46.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f0630001680 con 0x7f064406b970 2024-05-27T00:15:46.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 >> v1:172.21.15.98:6789/0 conn(0x7f064406b970 legacy=0x7f06440730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f062c0039c0 0x7f062c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f062c0043c0 0x7f062c004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f062c005090 0x7f062c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06440c9820 con 0x7f062c0043c0 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f062c00bff0 con 0x7f062c0039c0 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f062c00c1f0 con 0x7f062c005090 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0648c00640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f062c0043c0 0x7f062c004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0649600640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f062c0039c0 0x7f062c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f064a000640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f062c005090 0x7f062c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0648c00640 1 -- 172.21.15.98:0/2828819298 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f062c005090 msgr2=0x7f062c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0648c00640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f062c005090 0x7f062c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0648c00640 1 -- 172.21.15.98:0/2828819298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f062c0039c0 msgr2=0x7f062c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0648c00640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f062c0039c0 0x7f062c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f0648c00640 1 -- 172.21.15.98:0/2828819298 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062c003530 con 0x7f062c0043c0 2024-05-27T00:15:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.945+0000 7f064a000640 1 -- 172.21.15.98:0/2828819298 reap_dead start 2024-05-27T00:15:46.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f0648c00640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f062c0043c0 0x7f062c004890 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f063000c240 tx=0x7f063001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162b35db050cfab2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:46.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630007d40 con 0x7f062c0043c0 2024-05-27T00:15:46.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06300a9020 con 0x7f062c0043c0 2024-05-27T00:15:46.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f0643e00640 1 -- 172.21.15.98:0/2828819298 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630005b00 con 0x7f062c0043c0 2024-05-27T00:15:46.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f064c000640 1 -- 172.21.15.98:0/2828819298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f062c0043c0 msgr2=0x7f062c004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f064c000640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f062c0043c0 0x7f062c004890 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f063000c240 tx=0x7f063001cbe0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f064c000640 1 -- 172.21.15.98:0/2828819298 shutdown_connections 2024-05-27T00:15:46.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f064c000640 1 --2- 172.21.15.98:0/2828819298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f062c0043c0 0x7f062c004890 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f064c000640 1 -- 172.21.15.98:0/2828819298 >> 172.21.15.98:0/2828819298 conn(0x7f064406fa70 msgr2=0x7f064406fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:46.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f064c000640 1 -- 172.21.15.98:0/2828819298 shutdown_connections 2024-05-27T00:15:46.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f064c000640 1 -- 172.21.15.98:0/2828819298 wait complete. 2024-05-27T00:15:46.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f064c000640 1 Processor -- start 2024-05-27T00:15:46.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.949+0000 7f064c000640 1 -- start start 2024-05-27T00:15:46.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f064c000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f062c0043c0 0x7f06441edfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f064c000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064415b1c0 0x7f0644158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f064c000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06441f37c0 0x7f064414ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f064c000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06440c9430 con 0x7f064415b1c0 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f064c000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f06440c7e00 con 0x7f06441f37c0 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f064c000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f06440c95b0 con 0x7f062c0043c0 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0648c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064415b1c0 0x7f0644158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0648c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064415b1c0 0x7f0644158e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54010/0 (socket says 172.21.15.98:54010) 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0648c00640 1 -- 172.21.15.98:0/3881749485 learned_addr learned my addr 172.21.15.98:0/3881749485 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0649600640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f062c0043c0 0x7f06441edfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f064a000640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06441f37c0 0x7f064414ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0648c00640 1 -- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f062c0043c0 msgr2=0x7f06441edfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0648c00640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f062c0043c0 0x7f06441edfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0648c00640 1 -- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06441f37c0 msgr2=0x7f064414ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0648c00640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06441f37c0 0x7f064414ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0648c00640 1 -- 172.21.15.98:0/3881749485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06441f4100 con 0x7f064415b1c0 2024-05-27T00:15:46.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0648c00640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064415b1c0 0x7f0644158e50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f063001ca10 tx=0x7f062c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0641600640 1 -- 172.21.15.98:0/3881749485 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630007d40 con 0x7f064415b1c0 2024-05-27T00:15:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0641600640 1 -- 172.21.15.98:0/3881749485 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06300049f0 con 0x7f064415b1c0 2024-05-27T00:15:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0641600640 1 -- 172.21.15.98:0/3881749485 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630001c80 con 0x7f064415b1c0 2024-05-27T00:15:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f064c000640 1 -- 172.21.15.98:0/3881749485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f064415a400 con 0x7f064415b1c0 2024-05-27T00:15:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f064c000640 1 -- 172.21.15.98:0/3881749485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f064415a9c0 con 0x7f064415b1c0 2024-05-27T00:15:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f064c000640 1 -- 172.21.15.98:0/3881749485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06440cce00 con 0x7f064415b1c0 2024-05-27T00:15:46.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.953+0000 7f0641600640 1 -- 172.21.15.98:0/3881749485 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f06300bd060 con 0x7f064415b1c0 2024-05-27T00:15:46.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.957+0000 7f0641600640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f06240678c0 0x7f0624069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:46.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.957+0000 7f0641600640 1 -- 172.21.15.98:0/3881749485 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f063011ef90 con 0x7f064415b1c0 2024-05-27T00:15:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.957+0000 7f0649600640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f06240678c0 0x7f0624069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:46.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.957+0000 7f0649600640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f06240678c0 0x7f0624069d80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0644069d30 tx=0x7f063800a940 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:46.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:46.961+0000 7f0641600640 1 -- 172.21.15.98:0/3881749485 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f06300168f0 con 0x7f064415b1c0 2024-05-27T00:15:47.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.113+0000 7f064c000640 1 -- 172.21.15.98:0/3881749485 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0644075cb0 con 0x7f06240678c0 2024-05-27T00:15:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.133+0000 7f0641600640 1 -- 172.21.15.98:0/3881749485 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251852 (secure 0 0 0) 0x7f0644075cb0 con 0x7f06240678c0 2024-05-27T00:15:47.137 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:15:47.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 -- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f06240678c0 msgr2=0x7f0624069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f06240678c0 0x7f0624069d80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0644069d30 tx=0x7f063800a940 comp rx=0 tx=0).stop 2024-05-27T00:15:47.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 -- 172.21.15.98:0/3881749485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064415b1c0 msgr2=0x7f0644158e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064415b1c0 0x7f0644158e50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f063001ca10 tx=0x7f062c005840 comp rx=0 tx=0).stop 2024-05-27T00:15:47.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 -- 172.21.15.98:0/3881749485 shutdown_connections 2024-05-27T00:15:47.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f06240678c0 0x7f0624069d80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06441f37c0 0x7f064414ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064415b1c0 0x7f0644158e50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 --2- 172.21.15.98:0/3881749485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f062c0043c0 0x7f06441edfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 -- 172.21.15.98:0/3881749485 >> 172.21.15.98:0/3881749485 conn(0x7f064406fa70 msgr2=0x7f06440c3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:47.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 -- 172.21.15.98:0/3881749485 shutdown_connections 2024-05-27T00:15:47.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.137+0000 7f064c000640 1 -- 172.21.15.98:0/3881749485 wait complete. 2024-05-27T00:15:47.157 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-05-27T00:15:45.139848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219112,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755608,"statfs":{"total":773094113280,"available":772869742592,"internally_reserved":0,"allocated":3055616,"data_stored":1816820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004704"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748978+0000","last_change":"2024-05-27T00:15:25.640698+0000","last_active":"2024-05-27T00:15:26.748978+0000","last_peered":"2024-05-27T00:15:26.748978+0000","last_clean":"2024-05-27T00:15:26.748978+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:15:26.748978+0000","last_undegraded":"2024-05-27T00:15:26.748978+0000","last_fullsized":"2024-05-27T00:15:26.748978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:47:27.030637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583615+0000","last_change":"2024-05-27T00:15:25.598672+0000","last_active":"2024-05-27T00:15:26.583615+0000","last_peered":"2024-05-27T00:15:26.583615+0000","last_clean":"2024-05-27T00:15:26.583615+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:15:26.583615+0000","last_undegraded":"2024-05-27T00:15:26.583615+0000","last_fullsized":"2024-05-27T00:15:26.583615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745467+0000","last_change":"2024-05-27T00:15:25.593316+0000","last_active":"2024-05-27T00:15:26.745467+0000","last_peered":"2024-05-27T00:15:26.745467+0000","last_clean":"2024-05-27T00:15:26.745467+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:15:26.745467+0000","last_undegraded":"2024-05-27T00:15:26.745467+0000","last_fullsized":"2024-05-27T00:15:26.745467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587216+0000","last_change":"2024-05-27T00:15:25.620237+0000","last_active":"2024-05-27T00:15:26.587216+0000","last_peered":"2024-05-27T00:15:26.587216+0000","last_clean":"2024-05-27T00:15:26.587216+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:15:26.587216+0000","last_undegraded":"2024-05-27T00:15:26.587216+0000","last_fullsized":"2024-05-27T00:15:26.587216+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:17:53.053461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748475+0000","last_change":"2024-05-27T00:15:25.640146+0000","last_active":"2024-05-27T00:15:26.748475+0000","last_peered":"2024-05-27T00:15:26.748475+0000","last_clean":"2024-05-27T00:15:26.748475+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:15:26.748475+0000","last_undegraded":"2024-05-27T00:15:26.748475+0000","last_fullsized":"2024-05-27T00:15:26.748475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:41:46.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589129+0000","last_change":"2024-05-27T00:15:25.627128+0000","last_active":"2024-05-27T00:15:26.589129+0000","last_peered":"2024-05-27T00:15:26.589129+0000","last_clean":"2024-05-27T00:15:26.589129+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:15:26.589129+0000","last_undegraded":"2024-05-27T00:15:26.589129+0000","last_fullsized":"2024-05-27T00:15:26.589129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:47:27.030637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746743+0000","last_change":"2024-05-27T00:15:25.630446+0000","last_active":"2024-05-27T00:15:26.746743+0000","last_peered":"2024-05-27T00:15:26.746743+0000","last_clean":"2024-05-27T00:15:26.746743+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:15:26.746743+0000","last_undegraded":"2024-05-27T00:15:26.746743+0000","last_fullsized":"2024-05-27T00:15:26.746743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:59.804368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744980+0000","last_change":"2024-05-27T00:15:25.612157+0000","last_active":"2024-05-27T00:15:26.744980+0000","last_peered":"2024-05-27T00:15:26.744980+0000","last_clean":"2024-05-27T00:15:26.744980+0000","last_became_active":"2024-05-27T00:15:25.611884+0000","last_became_peered":"2024-05-27T00:15:25.611884+0000","last_unstale":"2024-05-27T00:15:26.744980+0000","last_undegraded":"2024-05-27T00:15:26.744980+0000","last_fullsized":"2024-05-27T00:15:26.744980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:12:28.505840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749289+0000","last_change":"2024-05-27T00:15:25.643312+0000","last_active":"2024-05-27T00:15:26.749289+0000","last_peered":"2024-05-27T00:15:26.749289+0000","last_clean":"2024-05-27T00:15:26.749289+0000","last_became_active":"2024-05-27T00:15:25.642663+0000","last_became_peered":"2024-05-27T00:15:25.642663+0000","last_unstale":"2024-05-27T00:15:26.749289+0000","last_undegraded":"2024-05-27T00:15:26.749289+0000","last_fullsized":"2024-05-27T00:15:26.749289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:06:27.232620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747471+0000","last_change":"2024-05-27T00:15:25.629892+0000","last_active":"2024-05-27T00:15:26.747471+0000","last_peered":"2024-05-27T00:15:26.747471+0000","last_clean":"2024-05-27T00:15:26.747471+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:15:26.747471+0000","last_undegraded":"2024-05-27T00:15:26.747471+0000","last_fullsized":"2024-05-27T00:15:26.747471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747909+0000","last_change":"2024-05-27T00:15:25.614887+0000","last_active":"2024-05-27T00:15:26.747909+0000","last_peered":"2024-05-27T00:15:26.747909+0000","last_clean":"2024-05-27T00:15:26.747909+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:15:26.747909+0000","last_undegraded":"2024-05-27T00:15:26.747909+0000","last_fullsized":"2024-05-27T00:15:26.747909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588218+0000","last_change":"2024-05-27T00:15:25.623864+0000","last_active":"2024-05-27T00:15:26.588218+0000","last_peered":"2024-05-27T00:15:26.588218+0000","last_clean":"2024-05-27T00:15:26.588218+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:15:26.588218+0000","last_undegraded":"2024-05-27T00:15:26.588218+0000","last_fullsized":"2024-05-27T00:15:26.588218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746025+0000","last_change":"2024-05-27T00:15:25.615495+0000","last_active":"2024-05-27T00:15:26.746025+0000","last_peered":"2024-05-27T00:15:26.746025+0000","last_clean":"2024-05-27T00:15:26.746025+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:15:26.746025+0000","last_undegraded":"2024-05-27T00:15:26.746025+0000","last_fullsized":"2024-05-27T00:15:26.746025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:42:53.882749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745273+0000","last_change":"2024-05-27T00:15:25.594508+0000","last_active":"2024-05-27T00:15:26.745273+0000","last_peered":"2024-05-27T00:15:26.745273+0000","last_clean":"2024-05-27T00:15:26.745273+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:15:26.745273+0000","last_undegraded":"2024-05-27T00:15:26.745273+0000","last_fullsized":"2024-05-27T00:15:26.745273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583449+0000","last_change":"2024-05-27T00:15:25.611818+0000","last_active":"2024-05-27T00:15:26.583449+0000","last_peered":"2024-05-27T00:15:26.583449+0000","last_clean":"2024-05-27T00:15:26.583449+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:15:26.583449+0000","last_undegraded":"2024-05-27T00:15:26.583449+0000","last_fullsized":"2024-05-27T00:15:26.583449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586675+0000","last_change":"2024-05-27T00:15:25.611365+0000","last_active":"2024-05-27T00:15:26.586675+0000","last_peered":"2024-05-27T00:15:26.586675+0000","last_clean":"2024-05-27T00:15:26.586675+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:15:26.586675+0000","last_undegraded":"2024-05-27T00:15:26.586675+0000","last_fullsized":"2024-05-27T00:15:26.586675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.590092+0000","last_change":"2024-05-27T00:15:25.632120+0000","last_active":"2024-05-27T00:15:26.590092+0000","last_peered":"2024-05-27T00:15:26.590092+0000","last_clean":"2024-05-27T00:15:26.590092+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:15:26.590092+0000","last_undegraded":"2024-05-27T00:15:26.590092+0000","last_fullsized":"2024-05-27T00:15:26.590092+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:06:27.232620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745618+0000","last_change":"2024-05-27T00:15:25.615868+0000","last_active":"2024-05-27T00:15:26.745618+0000","last_peered":"2024-05-27T00:15:26.745618+0000","last_clean":"2024-05-27T00:15:26.745618+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:15:26.745618+0000","last_undegraded":"2024-05-27T00:15:26.745618+0000","last_fullsized":"2024-05-27T00:15:26.745618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749314+0000","last_change":"2024-05-27T00:15:25.629101+0000","last_active":"2024-05-27T00:15:26.749314+0000","last_peered":"2024-05-27T00:15:26.749314+0000","last_clean":"2024-05-27T00:15:26.749314+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:15:26.749314+0000","last_undegraded":"2024-05-27T00:15:26.749314+0000","last_fullsized":"2024-05-27T00:15:26.749314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:26.893574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747537+0000","last_change":"2024-05-27T00:15:25.619351+0000","last_active":"2024-05-27T00:15:26.747537+0000","last_peered":"2024-05-27T00:15:26.747537+0000","last_clean":"2024-05-27T00:15:26.747537+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:15:26.747537+0000","last_undegraded":"2024-05-27T00:15:26.747537+0000","last_fullsized":"2024-05-27T00:15:26.747537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746054+0000","last_change":"2024-05-27T00:15:25.616436+0000","last_active":"2024-05-27T00:15:26.746054+0000","last_peered":"2024-05-27T00:15:26.746054+0000","last_clean":"2024-05-27T00:15:26.746054+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:15:26.746054+0000","last_undegraded":"2024-05-27T00:15:26.746054+0000","last_fullsized":"2024-05-27T00:15:26.746054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586830+0000","last_change":"2024-05-27T00:15:25.629373+0000","last_active":"2024-05-27T00:15:26.586830+0000","last_peered":"2024-05-27T00:15:26.586830+0000","last_clean":"2024-05-27T00:15:26.586830+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:15:26.586830+0000","last_undegraded":"2024-05-27T00:15:26.586830+0000","last_fullsized":"2024-05-27T00:15:26.586830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:02:15.012827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745308+0000","last_change":"2024-05-27T00:15:25.610707+0000","last_active":"2024-05-27T00:15:26.745308+0000","last_peered":"2024-05-27T00:15:26.745308+0000","last_clean":"2024-05-27T00:15:26.745308+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:15:26.745308+0000","last_undegraded":"2024-05-27T00:15:26.745308+0000","last_fullsized":"2024-05-27T00:15:26.745308+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:09:16.698319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587101+0000","last_change":"2024-05-27T00:15:25.613013+0000","last_active":"2024-05-27T00:15:26.587101+0000","last_peered":"2024-05-27T00:15:26.587101+0000","last_clean":"2024-05-27T00:15:26.587101+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:15:26.587101+0000","last_undegraded":"2024-05-27T00:15:26.587101+0000","last_fullsized":"2024-05-27T00:15:26.587101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745910+0000","last_change":"2024-05-27T00:15:25.610070+0000","last_active":"2024-05-27T00:15:26.745910+0000","last_peered":"2024-05-27T00:15:26.745910+0000","last_clean":"2024-05-27T00:15:26.745910+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:15:26.745910+0000","last_undegraded":"2024-05-27T00:15:26.745910+0000","last_fullsized":"2024-05-27T00:15:26.745910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:02:17.514654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747817+0000","last_change":"2024-05-27T00:15:25.633470+0000","last_active":"2024-05-27T00:15:26.747817+0000","last_peered":"2024-05-27T00:15:26.747817+0000","last_clean":"2024-05-27T00:15:26.747817+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:15:26.747817+0000","last_undegraded":"2024-05-27T00:15:26.747817+0000","last_fullsized":"2024-05-27T00:15:26.747817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583597+0000","last_change":"2024-05-27T00:15:25.597615+0000","last_active":"2024-05-27T00:15:26.583597+0000","last_peered":"2024-05-27T00:15:26.583597+0000","last_clean":"2024-05-27T00:15:26.583597+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:15:26.583597+0000","last_undegraded":"2024-05-27T00:15:26.583597+0000","last_fullsized":"2024-05-27T00:15:26.583597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746038+0000","last_change":"2024-05-27T00:15:25.610933+0000","last_active":"2024-05-27T00:15:26.746038+0000","last_peered":"2024-05-27T00:15:26.746038+0000","last_clean":"2024-05-27T00:15:26.746038+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:15:26.746038+0000","last_undegraded":"2024-05-27T00:15:26.746038+0000","last_fullsized":"2024-05-27T00:15:26.746038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:45:25.888432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746185+0000","last_change":"2024-05-27T00:15:25.598227+0000","last_active":"2024-05-27T00:15:26.746185+0000","last_peered":"2024-05-27T00:15:26.746185+0000","last_clean":"2024-05-27T00:15:26.746185+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:15:26.746185+0000","last_undegraded":"2024-05-27T00:15:26.746185+0000","last_fullsized":"2024-05-27T00:15:26.746185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588189+0000","last_change":"2024-05-27T00:15:25.620621+0000","last_active":"2024-05-27T00:15:26.588189+0000","last_peered":"2024-05-27T00:15:26.588189+0000","last_clean":"2024-05-27T00:15:26.588189+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:15:26.588189+0000","last_undegraded":"2024-05-27T00:15:26.588189+0000","last_fullsized":"2024-05-27T00:15:26.588189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:22:30.152810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744910+0000","last_change":"2024-05-27T00:15:25.605839+0000","last_active":"2024-05-27T00:15:26.744910+0000","last_peered":"2024-05-27T00:15:26.744910+0000","last_clean":"2024-05-27T00:15:26.744910+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:15:26.744910+0000","last_undegraded":"2024-05-27T00:15:26.744910+0000","last_fullsized":"2024-05-27T00:15:26.744910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746666+0000","last_change":"2024-05-27T00:15:25.607188+0000","last_active":"2024-05-27T00:15:26.746666+0000","last_peered":"2024-05-27T00:15:26.746666+0000","last_clean":"2024-05-27T00:15:26.746666+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:15:26.746666+0000","last_undegraded":"2024-05-27T00:15:26.746666+0000","last_fullsized":"2024-05-27T00:15:26.746666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587251+0000","last_change":"2024-05-27T00:15:25.615579+0000","last_active":"2024-05-27T00:15:26.587251+0000","last_peered":"2024-05-27T00:15:26.587251+0000","last_clean":"2024-05-27T00:15:26.587251+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:15:26.587251+0000","last_undegraded":"2024-05-27T00:15:26.587251+0000","last_fullsized":"2024-05-27T00:15:26.587251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:53.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746392+0000","last_change":"2024-05-27T00:15:24.611108+0000","last_active":"2024-05-27T00:15:26.746392+0000","last_peered":"2024-05-27T00:15:26.746392+0000","last_clean":"2024-05-27T00:15:26.746392+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:15:26.746392+0000","last_undegraded":"2024-05-27T00:15:26.746392+0000","last_fullsized":"2024-05-27T00:15:26.746392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:11.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746332+0000","last_change":"2024-05-27T00:15:25.614251+0000","last_active":"2024-05-27T00:15:26.746332+0000","last_peered":"2024-05-27T00:15:26.746332+0000","last_clean":"2024-05-27T00:15:26.746332+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:15:26.746332+0000","last_undegraded":"2024-05-27T00:15:26.746332+0000","last_fullsized":"2024-05-27T00:15:26.746332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747555+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:15:26.747555+0000","last_peered":"2024-05-27T00:15:26.747555+0000","last_clean":"2024-05-27T00:15:26.747555+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:15:26.747555+0000","last_undegraded":"2024-05-27T00:15:26.747555+0000","last_fullsized":"2024-05-27T00:15:26.747555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745607+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:15:26.745607+0000","last_peered":"2024-05-27T00:15:26.745607+0000","last_clean":"2024-05-27T00:15:26.745607+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:15:26.745607+0000","last_undegraded":"2024-05-27T00:15:26.745607+0000","last_fullsized":"2024-05-27T00:15:26.745607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:46:08.645927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745132+0000","last_change":"2024-05-27T00:15:25.596844+0000","last_active":"2024-05-27T00:15:26.745132+0000","last_peered":"2024-05-27T00:15:26.745132+0000","last_clean":"2024-05-27T00:15:26.745132+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:15:26.745132+0000","last_undegraded":"2024-05-27T00:15:26.745132+0000","last_fullsized":"2024-05-27T00:15:26.745132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:17:40.503615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587344+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:15:26.587344+0000","last_peered":"2024-05-27T00:15:26.587344+0000","last_clean":"2024-05-27T00:15:26.587344+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:15:26.587344+0000","last_undegraded":"2024-05-27T00:15:26.587344+0000","last_fullsized":"2024-05-27T00:15:26.587344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583235+0000","last_change":"2024-05-27T00:15:25.593149+0000","last_active":"2024-05-27T00:15:26.583235+0000","last_peered":"2024-05-27T00:15:26.583235+0000","last_clean":"2024-05-27T00:15:26.583235+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:15:26.583235+0000","last_undegraded":"2024-05-27T00:15:26.583235+0000","last_fullsized":"2024-05-27T00:15:26.583235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746974+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:15:26.746974+0000","last_peered":"2024-05-27T00:15:26.746974+0000","last_clean":"2024-05-27T00:15:26.746974+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:15:26.746974+0000","last_undegraded":"2024-05-27T00:15:26.746974+0000","last_fullsized":"2024-05-27T00:15:26.746974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589140+0000","last_change":"2024-05-27T00:15:25.618650+0000","last_active":"2024-05-27T00:15:26.589140+0000","last_peered":"2024-05-27T00:15:26.589140+0000","last_clean":"2024-05-27T00:15:26.589140+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:15:26.589140+0000","last_undegraded":"2024-05-27T00:15:26.589140+0000","last_fullsized":"2024-05-27T00:15:26.589140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:21.935526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589356+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:15:26.589356+0000","last_peered":"2024-05-27T00:15:26.589356+0000","last_clean":"2024-05-27T00:15:26.589356+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:15:26.589356+0000","last_undegraded":"2024-05-27T00:15:26.589356+0000","last_fullsized":"2024-05-27T00:15:26.589356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746448+0000","last_change":"2024-05-27T00:15:25.613142+0000","last_active":"2024-05-27T00:15:26.746448+0000","last_peered":"2024-05-27T00:15:26.746448+0000","last_clean":"2024-05-27T00:15:26.746448+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:15:26.746448+0000","last_undegraded":"2024-05-27T00:15:26.746448+0000","last_fullsized":"2024-05-27T00:15:26.746448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.620068+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:15:30.620068+0000","last_peered":"2024-05-27T00:15:30.620068+0000","last_clean":"2024-05-27T00:15:30.620068+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:15:30.620068+0000","last_undegraded":"2024-05-27T00:15:30.620068+0000","last_fullsized":"2024-05-27T00:15:30.620068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588245+0000","last_change":"2024-05-27T00:15:25.622383+0000","last_active":"2024-05-27T00:15:26.588245+0000","last_peered":"2024-05-27T00:15:26.588245+0000","last_clean":"2024-05-27T00:15:26.588245+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:15:26.588245+0000","last_undegraded":"2024-05-27T00:15:26.588245+0000","last_fullsized":"2024-05-27T00:15:26.588245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.624481+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:15:30.624481+0000","last_peered":"2024-05-27T00:15:30.624481+0000","last_clean":"2024-05-27T00:15:30.624481+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:15:30.624481+0000","last_undegraded":"2024-05-27T00:15:30.624481+0000","last_fullsized":"2024-05-27T00:15:30.624481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748511+0000","last_change":"2024-05-27T00:15:25.628646+0000","last_active":"2024-05-27T00:15:26.748511+0000","last_peered":"2024-05-27T00:15:26.748511+0000","last_clean":"2024-05-27T00:15:26.748511+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:15:26.748511+0000","last_undegraded":"2024-05-27T00:15:26.748511+0000","last_fullsized":"2024-05-27T00:15:26.748511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:22:30.152810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619790+0000","last_change":"2024-05-27T00:15:24.605986+0000","last_active":"2024-05-27T00:15:30.619790+0000","last_peered":"2024-05-27T00:15:30.619790+0000","last_clean":"2024-05-27T00:15:30.619790+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:15:30.619790+0000","last_undegraded":"2024-05-27T00:15:30.619790+0000","last_fullsized":"2024-05-27T00:15:30.619790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745414+0000","last_change":"2024-05-27T00:15:25.624136+0000","last_active":"2024-05-27T00:15:26.745414+0000","last_peered":"2024-05-27T00:15:26.745414+0000","last_clean":"2024-05-27T00:15:26.745414+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:15:26.745414+0000","last_undegraded":"2024-05-27T00:15:26.745414+0000","last_fullsized":"2024-05-27T00:15:26.745414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588435+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:15:26.588435+0000","last_peered":"2024-05-27T00:15:26.588435+0000","last_clean":"2024-05-27T00:15:26.588435+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:15:26.588435+0000","last_undegraded":"2024-05-27T00:15:26.588435+0000","last_fullsized":"2024-05-27T00:15:26.588435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748720+0000","last_change":"2024-05-27T00:15:25.641965+0000","last_active":"2024-05-27T00:15:26.748720+0000","last_peered":"2024-05-27T00:15:26.748720+0000","last_clean":"2024-05-27T00:15:26.748720+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:15:26.748720+0000","last_undegraded":"2024-05-27T00:15:26.748720+0000","last_fullsized":"2024-05-27T00:15:26.748720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:17.513529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746779+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:15:26.746779+0000","last_peered":"2024-05-27T00:15:26.746779+0000","last_clean":"2024-05-27T00:15:26.746779+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:15:26.746779+0000","last_undegraded":"2024-05-27T00:15:26.746779+0000","last_fullsized":"2024-05-27T00:15:26.746779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746732+0000","last_change":"2024-05-27T00:15:25.608549+0000","last_active":"2024-05-27T00:15:26.746732+0000","last_peered":"2024-05-27T00:15:26.746732+0000","last_clean":"2024-05-27T00:15:26.746732+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:15:26.746732+0000","last_undegraded":"2024-05-27T00:15:26.746732+0000","last_fullsized":"2024-05-27T00:15:26.746732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619615+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:15:30.619615+0000","last_peered":"2024-05-27T00:15:30.619615+0000","last_clean":"2024-05-27T00:15:30.619615+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:15:30.619615+0000","last_undegraded":"2024-05-27T00:15:30.619615+0000","last_fullsized":"2024-05-27T00:15:30.619615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747206+0000","last_change":"2024-05-27T00:15:25.638558+0000","last_active":"2024-05-27T00:15:26.747206+0000","last_peered":"2024-05-27T00:15:26.747206+0000","last_clean":"2024-05-27T00:15:26.747206+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:15:26.747206+0000","last_undegraded":"2024-05-27T00:15:26.747206+0000","last_fullsized":"2024-05-27T00:15:26.747206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:02:15.012827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587664+0000","last_change":"2024-05-27T00:15:24.601924+0000","last_active":"2024-05-27T00:15:26.587664+0000","last_peered":"2024-05-27T00:15:26.587664+0000","last_clean":"2024-05-27T00:15:26.587664+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:15:26.587664+0000","last_undegraded":"2024-05-27T00:15:26.587664+0000","last_fullsized":"2024-05-27T00:15:26.587664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:32.185839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749041+0000","last_change":"2024-05-27T00:15:25.639140+0000","last_active":"2024-05-27T00:15:26.749041+0000","last_peered":"2024-05-27T00:15:26.749041+0000","last_clean":"2024-05-27T00:15:26.749041+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:15:26.749041+0000","last_undegraded":"2024-05-27T00:15:26.749041+0000","last_fullsized":"2024-05-27T00:15:26.749041+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:21.935526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619020+0000","last_change":"2024-05-27T00:15:24.600878+0000","last_active":"2024-05-27T00:15:30.619020+0000","last_peered":"2024-05-27T00:15:30.619020+0000","last_clean":"2024-05-27T00:15:30.619020+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:15:30.619020+0000","last_undegraded":"2024-05-27T00:15:30.619020+0000","last_fullsized":"2024-05-27T00:15:30.619020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749435+0000","last_change":"2024-05-27T00:15:25.644249+0000","last_active":"2024-05-27T00:15:26.749435+0000","last_peered":"2024-05-27T00:15:26.749435+0000","last_clean":"2024-05-27T00:15:26.749435+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:15:26.749435+0000","last_undegraded":"2024-05-27T00:15:26.749435+0000","last_fullsized":"2024-05-27T00:15:26.749435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:59:25.042662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583715+0000","last_change":"2024-05-27T00:15:24.603452+0000","last_active":"2024-05-27T00:15:26.583715+0000","last_peered":"2024-05-27T00:15:26.583715+0000","last_clean":"2024-05-27T00:15:26.583715+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:15:26.583715+0000","last_undegraded":"2024-05-27T00:15:26.583715+0000","last_fullsized":"2024-05-27T00:15:26.583715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587865+0000","last_change":"2024-05-27T00:15:25.619341+0000","last_active":"2024-05-27T00:15:26.587865+0000","last_peered":"2024-05-27T00:15:26.587865+0000","last_clean":"2024-05-27T00:15:26.587865+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:15:26.587865+0000","last_undegraded":"2024-05-27T00:15:26.587865+0000","last_fullsized":"2024-05-27T00:15:26.587865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:41:46.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745117+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:15:26.745117+0000","last_peered":"2024-05-27T00:15:26.745117+0000","last_clean":"2024-05-27T00:15:26.745117+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:15:26.745117+0000","last_undegraded":"2024-05-27T00:15:26.745117+0000","last_fullsized":"2024-05-27T00:15:26.745117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748893+0000","last_change":"2024-05-27T00:15:25.639462+0000","last_active":"2024-05-27T00:15:26.748893+0000","last_peered":"2024-05-27T00:15:26.748893+0000","last_clean":"2024-05-27T00:15:26.748893+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:15:26.748893+0000","last_undegraded":"2024-05-27T00:15:26.748893+0000","last_fullsized":"2024-05-27T00:15:26.748893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:52:55.175945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746823+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:15:26.746823+0000","last_peered":"2024-05-27T00:15:26.746823+0000","last_clean":"2024-05-27T00:15:26.746823+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:15:26.746823+0000","last_undegraded":"2024-05-27T00:15:26.746823+0000","last_fullsized":"2024-05-27T00:15:26.746823+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586683+0000","last_change":"2024-05-27T00:15:24.603781+0000","last_active":"2024-05-27T00:15:26.586683+0000","last_peered":"2024-05-27T00:15:26.586683+0000","last_clean":"2024-05-27T00:15:26.586683+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:15:26.586683+0000","last_undegraded":"2024-05-27T00:15:26.586683+0000","last_fullsized":"2024-05-27T00:15:26.586683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746108+0000","last_change":"2024-05-27T00:15:25.631978+0000","last_active":"2024-05-27T00:15:26.746108+0000","last_peered":"2024-05-27T00:15:26.746108+0000","last_clean":"2024-05-27T00:15:26.746108+0000","last_became_active":"2024-05-27T00:15:25.631697+0000","last_became_peered":"2024-05-27T00:15:25.631697+0000","last_unstale":"2024-05-27T00:15:26.746108+0000","last_undegraded":"2024-05-27T00:15:26.746108+0000","last_fullsized":"2024-05-27T00:15:26.746108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:05.141408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746536+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:15:26.746536+0000","last_peered":"2024-05-27T00:15:26.746536+0000","last_clean":"2024-05-27T00:15:26.746536+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:15:26.746536+0000","last_undegraded":"2024-05-27T00:15:26.746536+0000","last_fullsized":"2024-05-27T00:15:26.746536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746253+0000","last_change":"2024-05-27T00:15:25.610627+0000","last_active":"2024-05-27T00:15:26.746253+0000","last_peered":"2024-05-27T00:15:26.746253+0000","last_clean":"2024-05-27T00:15:26.746253+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:15:26.746253+0000","last_undegraded":"2024-05-27T00:15:26.746253+0000","last_fullsized":"2024-05-27T00:15:26.746253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744898+0000","last_change":"2024-05-27T00:15:24.598182+0000","last_active":"2024-05-27T00:15:26.744898+0000","last_peered":"2024-05-27T00:15:26.744898+0000","last_clean":"2024-05-27T00:15:26.744898+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:15:26.744898+0000","last_undegraded":"2024-05-27T00:15:26.744898+0000","last_fullsized":"2024-05-27T00:15:26.744898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583243+0000","last_change":"2024-05-27T00:15:25.591449+0000","last_active":"2024-05-27T00:15:26.583243+0000","last_peered":"2024-05-27T00:15:26.583243+0000","last_clean":"2024-05-27T00:15:26.583243+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:15:26.583243+0000","last_undegraded":"2024-05-27T00:15:26.583243+0000","last_fullsized":"2024-05-27T00:15:26.583243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588693+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:15:26.588693+0000","last_peered":"2024-05-27T00:15:26.588693+0000","last_clean":"2024-05-27T00:15:26.588693+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:15:26.588693+0000","last_undegraded":"2024-05-27T00:15:26.588693+0000","last_fullsized":"2024-05-27T00:15:26.588693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587675+0000","last_change":"2024-05-27T00:15:25.614073+0000","last_active":"2024-05-27T00:15:26.587675+0000","last_peered":"2024-05-27T00:15:26.587675+0000","last_clean":"2024-05-27T00:15:26.587675+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:15:26.587675+0000","last_undegraded":"2024-05-27T00:15:26.587675+0000","last_fullsized":"2024-05-27T00:15:26.587675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583569+0000","last_change":"2024-05-27T00:15:24.603518+0000","last_active":"2024-05-27T00:15:26.583569+0000","last_peered":"2024-05-27T00:15:26.583569+0000","last_clean":"2024-05-27T00:15:26.583569+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:15:26.583569+0000","last_undegraded":"2024-05-27T00:15:26.583569+0000","last_fullsized":"2024-05-27T00:15:26.583569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588810+0000","last_change":"2024-05-27T00:15:25.621519+0000","last_active":"2024-05-27T00:15:26.588810+0000","last_peered":"2024-05-27T00:15:26.588810+0000","last_clean":"2024-05-27T00:15:26.588810+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:15:26.588810+0000","last_undegraded":"2024-05-27T00:15:26.588810+0000","last_fullsized":"2024-05-27T00:15:26.588810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:52:55.175945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749299+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:15:26.749299+0000","last_peered":"2024-05-27T00:15:26.749299+0000","last_clean":"2024-05-27T00:15:26.749299+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:15:26.749299+0000","last_undegraded":"2024-05-27T00:15:26.749299+0000","last_fullsized":"2024-05-27T00:15:26.749299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583460+0000","last_change":"2024-05-27T00:15:25.593180+0000","last_active":"2024-05-27T00:15:26.583460+0000","last_peered":"2024-05-27T00:15:26.583460+0000","last_clean":"2024-05-27T00:15:26.583460+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:15:26.583460+0000","last_undegraded":"2024-05-27T00:15:26.583460+0000","last_fullsized":"2024-05-27T00:15:26.583460+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583701+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:15:26.583701+0000","last_peered":"2024-05-27T00:15:26.583701+0000","last_clean":"2024-05-27T00:15:26.583701+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:15:26.583701+0000","last_undegraded":"2024-05-27T00:15:26.583701+0000","last_fullsized":"2024-05-27T00:15:26.583701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745915+0000","last_change":"2024-05-27T00:15:25.631919+0000","last_active":"2024-05-27T00:15:26.745915+0000","last_peered":"2024-05-27T00:15:26.745915+0000","last_clean":"2024-05-27T00:15:26.745915+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:15:26.745915+0000","last_undegraded":"2024-05-27T00:15:26.745915+0000","last_fullsized":"2024-05-27T00:15:26.745915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:24:31.995603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.620892+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:15:30.620892+0000","last_peered":"2024-05-27T00:15:30.620892+0000","last_clean":"2024-05-27T00:15:30.620892+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:15:30.620892+0000","last_undegraded":"2024-05-27T00:15:30.620892+0000","last_fullsized":"2024-05-27T00:15:30.620892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745629+0000","last_change":"2024-05-27T00:15:25.596792+0000","last_active":"2024-05-27T00:15:26.745629+0000","last_peered":"2024-05-27T00:15:26.745629+0000","last_clean":"2024-05-27T00:15:26.745629+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:15:26.745629+0000","last_undegraded":"2024-05-27T00:15:26.745629+0000","last_fullsized":"2024-05-27T00:15:26.745629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747720+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:15:26.747720+0000","last_peered":"2024-05-27T00:15:26.747720+0000","last_clean":"2024-05-27T00:15:26.747720+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:15:26.747720+0000","last_undegraded":"2024-05-27T00:15:26.747720+0000","last_fullsized":"2024-05-27T00:15:26.747720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587779+0000","last_change":"2024-05-27T00:15:25.610697+0000","last_active":"2024-05-27T00:15:26.587779+0000","last_peered":"2024-05-27T00:15:26.587779+0000","last_clean":"2024-05-27T00:15:26.587779+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:15:26.587779+0000","last_undegraded":"2024-05-27T00:15:26.587779+0000","last_fullsized":"2024-05-27T00:15:26.587779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:14.969827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583375+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:15:26.583375+0000","last_peered":"2024-05-27T00:15:26.583375+0000","last_clean":"2024-05-27T00:15:26.583375+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:15:26.583375+0000","last_undegraded":"2024-05-27T00:15:26.583375+0000","last_fullsized":"2024-05-27T00:15:26.583375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745282+0000","last_change":"2024-05-27T00:15:25.597164+0000","last_active":"2024-05-27T00:15:26.745282+0000","last_peered":"2024-05-27T00:15:26.745282+0000","last_clean":"2024-05-27T00:15:26.745282+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:15:26.745282+0000","last_undegraded":"2024-05-27T00:15:26.745282+0000","last_fullsized":"2024-05-27T00:15:26.745282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:04.655872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583422+0000","last_change":"2024-05-27T00:15:24.598238+0000","last_active":"2024-05-27T00:15:26.583422+0000","last_peered":"2024-05-27T00:15:26.583422+0000","last_clean":"2024-05-27T00:15:26.583422+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:15:26.583422+0000","last_undegraded":"2024-05-27T00:15:26.583422+0000","last_fullsized":"2024-05-27T00:15:26.583422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746235+0000","last_change":"2024-05-27T00:15:25.601140+0000","last_active":"2024-05-27T00:15:26.746235+0000","last_peered":"2024-05-27T00:15:26.746235+0000","last_clean":"2024-05-27T00:15:26.746235+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:15:26.746235+0000","last_undegraded":"2024-05-27T00:15:26.746235+0000","last_fullsized":"2024-05-27T00:15:26.746235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:42:05.334730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.621275+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:15:30.621275+0000","last_peered":"2024-05-27T00:15:30.621275+0000","last_clean":"2024-05-27T00:15:30.621275+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:15:30.621275+0000","last_undegraded":"2024-05-27T00:15:30.621275+0000","last_fullsized":"2024-05-27T00:15:30.621275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747292+0000","last_change":"2024-05-27T00:15:25.619694+0000","last_active":"2024-05-27T00:15:26.747292+0000","last_peered":"2024-05-27T00:15:26.747292+0000","last_clean":"2024-05-27T00:15:26.747292+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:15:26.747292+0000","last_undegraded":"2024-05-27T00:15:26.747292+0000","last_fullsized":"2024-05-27T00:15:26.747292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588751+0000","last_change":"2024-05-27T00:15:25.627578+0000","last_active":"2024-05-27T00:15:26.588751+0000","last_peered":"2024-05-27T00:15:26.588751+0000","last_clean":"2024-05-27T00:15:26.588751+0000","last_became_active":"2024-05-27T00:15:25.622421+0000","last_became_peered":"2024-05-27T00:15:25.622421+0000","last_unstale":"2024-05-27T00:15:26.588751+0000","last_undegraded":"2024-05-27T00:15:26.588751+0000","last_fullsized":"2024-05-27T00:15:26.588751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:17.513529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619639+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:15:30.619639+0000","last_peered":"2024-05-27T00:15:30.619639+0000","last_clean":"2024-05-27T00:15:30.619639+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:15:30.619639+0000","last_undegraded":"2024-05-27T00:15:30.619639+0000","last_fullsized":"2024-05-27T00:15:30.619639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587486+0000","last_change":"2024-05-27T00:15:25.623207+0000","last_active":"2024-05-27T00:15:26.587486+0000","last_peered":"2024-05-27T00:15:26.587486+0000","last_clean":"2024-05-27T00:15:26.587486+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:15:26.587486+0000","last_undegraded":"2024-05-27T00:15:26.587486+0000","last_fullsized":"2024-05-27T00:15:26.587486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:59.804368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.618376+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:15:30.618376+0000","last_peered":"2024-05-27T00:15:30.618376+0000","last_clean":"2024-05-27T00:15:30.618376+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:15:30.618376+0000","last_undegraded":"2024-05-27T00:15:30.618376+0000","last_fullsized":"2024-05-27T00:15:30.618376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746305+0000","last_change":"2024-05-27T00:15:25.613018+0000","last_active":"2024-05-27T00:15:26.746305+0000","last_peered":"2024-05-27T00:15:26.746305+0000","last_clean":"2024-05-27T00:15:26.746305+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:15:26.746305+0000","last_undegraded":"2024-05-27T00:15:26.746305+0000","last_fullsized":"2024-05-27T00:15:26.746305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745698+0000","last_change":"2024-05-27T00:15:24.600571+0000","last_active":"2024-05-27T00:15:26.745698+0000","last_peered":"2024-05-27T00:15:26.745698+0000","last_clean":"2024-05-27T00:15:26.745698+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:15:26.745698+0000","last_undegraded":"2024-05-27T00:15:26.745698+0000","last_fullsized":"2024-05-27T00:15:26.745698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:44:21.645223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745724+0000","last_change":"2024-05-27T00:15:25.615649+0000","last_active":"2024-05-27T00:15:26.745724+0000","last_peered":"2024-05-27T00:15:26.745724+0000","last_clean":"2024-05-27T00:15:26.745724+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:15:26.745724+0000","last_undegraded":"2024-05-27T00:15:26.745724+0000","last_fullsized":"2024-05-27T00:15:26.745724+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:16:16.309939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.621025+0000","last_change":"2024-05-27T00:15:24.615066+0000","last_active":"2024-05-27T00:15:30.621025+0000","last_peered":"2024-05-27T00:15:30.621025+0000","last_clean":"2024-05-27T00:15:30.621025+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:15:30.621025+0000","last_undegraded":"2024-05-27T00:15:30.621025+0000","last_fullsized":"2024-05-27T00:15:30.621025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":669858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":79034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542154,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":80520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":78908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542154,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":79068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542153,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":669914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:15:47.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:15:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.333+0000 7fc4b2200640 1 Processor -- start 2024-05-27T00:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.333+0000 7fc4b2200640 1 -- start start 2024-05-27T00:15:47.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4b2200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac0731f0 0x7fc4ac0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4b2200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4ac0c17d0 con 0x7fc4ac0735d0 2024-05-27T00:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4b2200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4ac070e10 con 0x7fc4ac0c79f0 2024-05-27T00:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4b2200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ac071990 con 0x7fc4ac0731f0 2024-05-27T00:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4a2a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc4ac0735d0 0x7fc4ac151430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45868/0 (socket says 172.21.15.98:45868) 2024-05-27T00:15:47.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4a2a00640 1 -- 172.21.15.98:0/1875479992 learned_addr learned my addr 172.21.15.98:0/1875479992 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4ab400640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac0731f0 0x7fc4ac0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3213249447 0 0) 0x7fc4ac0c17d0 con 0x7fc4ac0735d0 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc490003530 con 0x7fc4ac0735d0 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1032609711 0 0) 0x7fc490003530 con 0x7fc4ac0735d0 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac0731f0 msgr2=0x7fc4ac0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac0731f0 0x7fc4ac0c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 >> v1:172.21.15.204:6789/0 conn(0x7fc4ac0c79f0 legacy=0x7fc4ac157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ac0710f0 con 0x7fc4ac0735d0 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fc494001680 con 0x7fc4ac0735d0 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 >> v1:172.21.15.98:6789/0 conn(0x7fc4ac0735d0 legacy=0x7fc4ac151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4900039f0 0x7fc490003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4900043a0 0x7fc490004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac0731f0 0x7fc4900050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac071990 con 0x7fc4900043a0 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc49000b9d0 con 0x7fc4900039f0 2024-05-27T00:15:47.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc49000bbd0 con 0x7fc4ac0731f0 2024-05-27T00:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4a2a00640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4900043a0 0x7fc490004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4ab400640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4900039f0 0x7fc490003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4abe00640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac0731f0 0x7fc4900050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4a2a00640 1 -- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac0731f0 msgr2=0x7fc4900050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4a2a00640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac0731f0 0x7fc4900050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4a2a00640 1 -- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4900039f0 msgr2=0x7fc490003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4a2a00640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4900039f0 0x7fc490003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4a2a00640 1 -- 172.21.15.98:0/1875479992 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc490003530 con 0x7fc4900043a0 2024-05-27T00:15:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.337+0000 7fc4abe00640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac0731f0 0x7fc4900050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4a2a00640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4900043a0 0x7fc490004870 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0739f0 tx=0x7fc49401f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b700313601c8f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494007cf0 con 0x7fc4900043a0 2024-05-27T00:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc494005b00 con 0x7fc4900043a0 2024-05-27T00:15:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4aaa00640 1 -- 172.21.15.98:0/1875479992 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494007070 con 0x7fc4900043a0 2024-05-27T00:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4b2200640 1 -- 172.21.15.98:0/1875479992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4900043a0 msgr2=0x7fc490004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4b2200640 1 --2- 172.21.15.98:0/1875479992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4900043a0 0x7fc490004870 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0739f0 tx=0x7fc49401f350 comp rx=0 tx=0).stop 2024-05-27T00:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4abe00640 1 -- 172.21.15.98:0/1875479992 reap_dead start 2024-05-27T00:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4b2200640 1 -- 172.21.15.98:0/1875479992 shutdown_connections 2024-05-27T00:15:47.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4b2200640 1 -- 172.21.15.98:0/1875479992 >> 172.21.15.98:0/1875479992 conn(0x7fc4ac067530 msgr2=0x7fc4ac06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4b2200640 1 -- 172.21.15.98:0/1875479992 shutdown_connections 2024-05-27T00:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4b2200640 1 -- 172.21.15.98:0/1875479992 wait complete. 2024-05-27T00:15:47.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4b2200640 1 Processor -- start 2024-05-27T00:15:47.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.341+0000 7fc4b2200640 1 -- start start 2024-05-27T00:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4b2200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4900043a0 0x7fc4ac0c1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4b2200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac148670 0x7fc4ac1546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4b2200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac152fd0 0x7fc4ac153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4b2200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac0c8400 con 0x7fc4ac148670 2024-05-27T00:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4b2200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac0716b0 con 0x7fc4900043a0 2024-05-27T00:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4b2200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ac151b40 con 0x7fc4ac152fd0 2024-05-27T00:15:47.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a2a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac148670 0x7fc4ac1546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a2a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac148670 0x7fc4ac1546e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54022/0 (socket says 172.21.15.98:54022) 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a2a00640 1 -- 172.21.15.98:0/2773300131 learned_addr learned my addr 172.21.15.98:0/2773300131 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4ab400640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4900043a0 0x7fc4ac0c1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4abe00640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac152fd0 0x7fc4ac153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a2a00640 1 -- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac152fd0 msgr2=0x7fc4ac153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a2a00640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac152fd0 0x7fc4ac153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a2a00640 1 -- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4900043a0 msgr2=0x7fc4ac0c1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a2a00640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4900043a0 0x7fc4ac0c1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a2a00640 1 -- 172.21.15.98:0/2773300131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ac151fa0 con 0x7fc4ac148670 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4ab400640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4900043a0 0x7fc4ac0c1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a2a00640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac148670 0x7fc4ac1546e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc494001900 tx=0x7fc490005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a3e00640 1 -- 172.21.15.98:0/2773300131 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494005cb0 con 0x7fc4ac148670 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a3e00640 1 -- 172.21.15.98:0/2773300131 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc494005e50 con 0x7fc4ac148670 2024-05-27T00:15:47.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4a3e00640 1 -- 172.21.15.98:0/2773300131 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494007290 con 0x7fc4ac148670 2024-05-27T00:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4b2200640 1 -- 172.21.15.98:0/2773300131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ac0c20f0 con 0x7fc4ac148670 2024-05-27T00:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4b2200640 1 -- 172.21.15.98:0/2773300131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ac1524f0 con 0x7fc4ac148670 2024-05-27T00:15:47.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.345+0000 7fc4b2200640 1 -- 172.21.15.98:0/2773300131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4ac075cb0 con 0x7fc4ac148670 2024-05-27T00:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.349+0000 7fc4a3e00640 1 -- 172.21.15.98:0/2773300131 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc4940c2060 con 0x7fc4ac148670 2024-05-27T00:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.349+0000 7fc4a3e00640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4880678c0 0x7fc488069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.349+0000 7fc4a3e00640 1 -- 172.21.15.98:0/2773300131 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc494122fc0 con 0x7fc4ac148670 2024-05-27T00:15:47.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.349+0000 7fc4ab400640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4880678c0 0x7fc488069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.349+0000 7fc4ab400640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4880678c0 0x7fc488069d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009be0 tx=0x7fc49c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:47.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.353+0000 7fc4a3e00640 1 -- 172.21.15.98:0/2773300131 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc494019060 con 0x7fc4ac148670 2024-05-27T00:15:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.505+0000 7fc4b2200640 1 -- 172.21.15.98:0/2773300131 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc4ac06ece0 con 0x7fc4880678c0 2024-05-27T00:15:47.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.521+0000 7fc4a3e00640 1 -- 172.21.15.98:0/2773300131 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251854 (secure 0 0 0) 0x7fc4ac06ece0 con 0x7fc4880678c0 2024-05-27T00:15:47.528 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:47.529 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:15:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 -- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4880678c0 msgr2=0x7fc488069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4880678c0 0x7fc488069d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009be0 tx=0x7fc49c008040 comp rx=0 tx=0).stop 2024-05-27T00:15:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 -- 172.21.15.98:0/2773300131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac148670 msgr2=0x7fc4ac1546e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac148670 0x7fc4ac1546e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc494001900 tx=0x7fc490005840 comp rx=0 tx=0).stop 2024-05-27T00:15:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 -- 172.21.15.98:0/2773300131 shutdown_connections 2024-05-27T00:15:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4ac152fd0 0x7fc4ac153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4ac148670 0x7fc4ac1546e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4880678c0 0x7fc488069d80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 --2- 172.21.15.98:0/2773300131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4900043a0 0x7fc4ac0c1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 -- 172.21.15.98:0/2773300131 >> 172.21.15.98:0/2773300131 conn(0x7fc4ac067530 msgr2=0x7fc4ac0c62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:47.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 -- 172.21.15.98:0/2773300131 shutdown_connections 2024-05-27T00:15:47.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.525+0000 7fc4b2200640 1 -- 172.21.15.98:0/2773300131 wait complete. 2024-05-27T00:15:47.548 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-05-27T00:15:47.140432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593032,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":205,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219112,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755608,"statfs":{"total":773094113280,"available":772869742592,"internally_reserved":0,"allocated":3055616,"data_stored":1817328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":254,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004211"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748978+0000","last_change":"2024-05-27T00:15:25.640698+0000","last_active":"2024-05-27T00:15:26.748978+0000","last_peered":"2024-05-27T00:15:26.748978+0000","last_clean":"2024-05-27T00:15:26.748978+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:15:26.748978+0000","last_undegraded":"2024-05-27T00:15:26.748978+0000","last_fullsized":"2024-05-27T00:15:26.748978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:47:27.030637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583615+0000","last_change":"2024-05-27T00:15:25.598672+0000","last_active":"2024-05-27T00:15:26.583615+0000","last_peered":"2024-05-27T00:15:26.583615+0000","last_clean":"2024-05-27T00:15:26.583615+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:15:26.583615+0000","last_undegraded":"2024-05-27T00:15:26.583615+0000","last_fullsized":"2024-05-27T00:15:26.583615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745467+0000","last_change":"2024-05-27T00:15:25.593316+0000","last_active":"2024-05-27T00:15:26.745467+0000","last_peered":"2024-05-27T00:15:26.745467+0000","last_clean":"2024-05-27T00:15:26.745467+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:15:26.745467+0000","last_undegraded":"2024-05-27T00:15:26.745467+0000","last_fullsized":"2024-05-27T00:15:26.745467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587216+0000","last_change":"2024-05-27T00:15:25.620237+0000","last_active":"2024-05-27T00:15:26.587216+0000","last_peered":"2024-05-27T00:15:26.587216+0000","last_clean":"2024-05-27T00:15:26.587216+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:15:26.587216+0000","last_undegraded":"2024-05-27T00:15:26.587216+0000","last_fullsized":"2024-05-27T00:15:26.587216+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:17:53.053461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748475+0000","last_change":"2024-05-27T00:15:25.640146+0000","last_active":"2024-05-27T00:15:26.748475+0000","last_peered":"2024-05-27T00:15:26.748475+0000","last_clean":"2024-05-27T00:15:26.748475+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:15:26.748475+0000","last_undegraded":"2024-05-27T00:15:26.748475+0000","last_fullsized":"2024-05-27T00:15:26.748475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:41:46.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589129+0000","last_change":"2024-05-27T00:15:25.627128+0000","last_active":"2024-05-27T00:15:26.589129+0000","last_peered":"2024-05-27T00:15:26.589129+0000","last_clean":"2024-05-27T00:15:26.589129+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:15:26.589129+0000","last_undegraded":"2024-05-27T00:15:26.589129+0000","last_fullsized":"2024-05-27T00:15:26.589129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:47:27.030637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746743+0000","last_change":"2024-05-27T00:15:25.630446+0000","last_active":"2024-05-27T00:15:26.746743+0000","last_peered":"2024-05-27T00:15:26.746743+0000","last_clean":"2024-05-27T00:15:26.746743+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:15:26.746743+0000","last_undegraded":"2024-05-27T00:15:26.746743+0000","last_fullsized":"2024-05-27T00:15:26.746743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:59.804368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744980+0000","last_change":"2024-05-27T00:15:25.612157+0000","last_active":"2024-05-27T00:15:26.744980+0000","last_peered":"2024-05-27T00:15:26.744980+0000","last_clean":"2024-05-27T00:15:26.744980+0000","last_became_active":"2024-05-27T00:15:25.611884+0000","last_became_peered":"2024-05-27T00:15:25.611884+0000","last_unstale":"2024-05-27T00:15:26.744980+0000","last_undegraded":"2024-05-27T00:15:26.744980+0000","last_fullsized":"2024-05-27T00:15:26.744980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:12:28.505840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749289+0000","last_change":"2024-05-27T00:15:25.643312+0000","last_active":"2024-05-27T00:15:26.749289+0000","last_peered":"2024-05-27T00:15:26.749289+0000","last_clean":"2024-05-27T00:15:26.749289+0000","last_became_active":"2024-05-27T00:15:25.642663+0000","last_became_peered":"2024-05-27T00:15:25.642663+0000","last_unstale":"2024-05-27T00:15:26.749289+0000","last_undegraded":"2024-05-27T00:15:26.749289+0000","last_fullsized":"2024-05-27T00:15:26.749289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:06:27.232620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747471+0000","last_change":"2024-05-27T00:15:25.629892+0000","last_active":"2024-05-27T00:15:26.747471+0000","last_peered":"2024-05-27T00:15:26.747471+0000","last_clean":"2024-05-27T00:15:26.747471+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:15:26.747471+0000","last_undegraded":"2024-05-27T00:15:26.747471+0000","last_fullsized":"2024-05-27T00:15:26.747471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747909+0000","last_change":"2024-05-27T00:15:25.614887+0000","last_active":"2024-05-27T00:15:26.747909+0000","last_peered":"2024-05-27T00:15:26.747909+0000","last_clean":"2024-05-27T00:15:26.747909+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:15:26.747909+0000","last_undegraded":"2024-05-27T00:15:26.747909+0000","last_fullsized":"2024-05-27T00:15:26.747909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588218+0000","last_change":"2024-05-27T00:15:25.623864+0000","last_active":"2024-05-27T00:15:26.588218+0000","last_peered":"2024-05-27T00:15:26.588218+0000","last_clean":"2024-05-27T00:15:26.588218+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:15:26.588218+0000","last_undegraded":"2024-05-27T00:15:26.588218+0000","last_fullsized":"2024-05-27T00:15:26.588218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746025+0000","last_change":"2024-05-27T00:15:25.615495+0000","last_active":"2024-05-27T00:15:26.746025+0000","last_peered":"2024-05-27T00:15:26.746025+0000","last_clean":"2024-05-27T00:15:26.746025+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:15:26.746025+0000","last_undegraded":"2024-05-27T00:15:26.746025+0000","last_fullsized":"2024-05-27T00:15:26.746025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:42:53.882749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745273+0000","last_change":"2024-05-27T00:15:25.594508+0000","last_active":"2024-05-27T00:15:26.745273+0000","last_peered":"2024-05-27T00:15:26.745273+0000","last_clean":"2024-05-27T00:15:26.745273+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:15:26.745273+0000","last_undegraded":"2024-05-27T00:15:26.745273+0000","last_fullsized":"2024-05-27T00:15:26.745273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583449+0000","last_change":"2024-05-27T00:15:25.611818+0000","last_active":"2024-05-27T00:15:26.583449+0000","last_peered":"2024-05-27T00:15:26.583449+0000","last_clean":"2024-05-27T00:15:26.583449+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:15:26.583449+0000","last_undegraded":"2024-05-27T00:15:26.583449+0000","last_fullsized":"2024-05-27T00:15:26.583449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586675+0000","last_change":"2024-05-27T00:15:25.611365+0000","last_active":"2024-05-27T00:15:26.586675+0000","last_peered":"2024-05-27T00:15:26.586675+0000","last_clean":"2024-05-27T00:15:26.586675+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:15:26.586675+0000","last_undegraded":"2024-05-27T00:15:26.586675+0000","last_fullsized":"2024-05-27T00:15:26.586675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.590092+0000","last_change":"2024-05-27T00:15:25.632120+0000","last_active":"2024-05-27T00:15:26.590092+0000","last_peered":"2024-05-27T00:15:26.590092+0000","last_clean":"2024-05-27T00:15:26.590092+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:15:26.590092+0000","last_undegraded":"2024-05-27T00:15:26.590092+0000","last_fullsized":"2024-05-27T00:15:26.590092+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:06:27.232620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745618+0000","last_change":"2024-05-27T00:15:25.615868+0000","last_active":"2024-05-27T00:15:26.745618+0000","last_peered":"2024-05-27T00:15:26.745618+0000","last_clean":"2024-05-27T00:15:26.745618+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:15:26.745618+0000","last_undegraded":"2024-05-27T00:15:26.745618+0000","last_fullsized":"2024-05-27T00:15:26.745618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749314+0000","last_change":"2024-05-27T00:15:25.629101+0000","last_active":"2024-05-27T00:15:26.749314+0000","last_peered":"2024-05-27T00:15:26.749314+0000","last_clean":"2024-05-27T00:15:26.749314+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:15:26.749314+0000","last_undegraded":"2024-05-27T00:15:26.749314+0000","last_fullsized":"2024-05-27T00:15:26.749314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:26.893574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747537+0000","last_change":"2024-05-27T00:15:25.619351+0000","last_active":"2024-05-27T00:15:26.747537+0000","last_peered":"2024-05-27T00:15:26.747537+0000","last_clean":"2024-05-27T00:15:26.747537+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:15:26.747537+0000","last_undegraded":"2024-05-27T00:15:26.747537+0000","last_fullsized":"2024-05-27T00:15:26.747537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746054+0000","last_change":"2024-05-27T00:15:25.616436+0000","last_active":"2024-05-27T00:15:26.746054+0000","last_peered":"2024-05-27T00:15:26.746054+0000","last_clean":"2024-05-27T00:15:26.746054+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:15:26.746054+0000","last_undegraded":"2024-05-27T00:15:26.746054+0000","last_fullsized":"2024-05-27T00:15:26.746054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586830+0000","last_change":"2024-05-27T00:15:25.629373+0000","last_active":"2024-05-27T00:15:26.586830+0000","last_peered":"2024-05-27T00:15:26.586830+0000","last_clean":"2024-05-27T00:15:26.586830+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:15:26.586830+0000","last_undegraded":"2024-05-27T00:15:26.586830+0000","last_fullsized":"2024-05-27T00:15:26.586830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:02:15.012827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745308+0000","last_change":"2024-05-27T00:15:25.610707+0000","last_active":"2024-05-27T00:15:26.745308+0000","last_peered":"2024-05-27T00:15:26.745308+0000","last_clean":"2024-05-27T00:15:26.745308+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:15:26.745308+0000","last_undegraded":"2024-05-27T00:15:26.745308+0000","last_fullsized":"2024-05-27T00:15:26.745308+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:09:16.698319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587101+0000","last_change":"2024-05-27T00:15:25.613013+0000","last_active":"2024-05-27T00:15:26.587101+0000","last_peered":"2024-05-27T00:15:26.587101+0000","last_clean":"2024-05-27T00:15:26.587101+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:15:26.587101+0000","last_undegraded":"2024-05-27T00:15:26.587101+0000","last_fullsized":"2024-05-27T00:15:26.587101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745910+0000","last_change":"2024-05-27T00:15:25.610070+0000","last_active":"2024-05-27T00:15:26.745910+0000","last_peered":"2024-05-27T00:15:26.745910+0000","last_clean":"2024-05-27T00:15:26.745910+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:15:26.745910+0000","last_undegraded":"2024-05-27T00:15:26.745910+0000","last_fullsized":"2024-05-27T00:15:26.745910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:02:17.514654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747817+0000","last_change":"2024-05-27T00:15:25.633470+0000","last_active":"2024-05-27T00:15:26.747817+0000","last_peered":"2024-05-27T00:15:26.747817+0000","last_clean":"2024-05-27T00:15:26.747817+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:15:26.747817+0000","last_undegraded":"2024-05-27T00:15:26.747817+0000","last_fullsized":"2024-05-27T00:15:26.747817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583597+0000","last_change":"2024-05-27T00:15:25.597615+0000","last_active":"2024-05-27T00:15:26.583597+0000","last_peered":"2024-05-27T00:15:26.583597+0000","last_clean":"2024-05-27T00:15:26.583597+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:15:26.583597+0000","last_undegraded":"2024-05-27T00:15:26.583597+0000","last_fullsized":"2024-05-27T00:15:26.583597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746038+0000","last_change":"2024-05-27T00:15:25.610933+0000","last_active":"2024-05-27T00:15:26.746038+0000","last_peered":"2024-05-27T00:15:26.746038+0000","last_clean":"2024-05-27T00:15:26.746038+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:15:26.746038+0000","last_undegraded":"2024-05-27T00:15:26.746038+0000","last_fullsized":"2024-05-27T00:15:26.746038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:45:25.888432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746185+0000","last_change":"2024-05-27T00:15:25.598227+0000","last_active":"2024-05-27T00:15:26.746185+0000","last_peered":"2024-05-27T00:15:26.746185+0000","last_clean":"2024-05-27T00:15:26.746185+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:15:26.746185+0000","last_undegraded":"2024-05-27T00:15:26.746185+0000","last_fullsized":"2024-05-27T00:15:26.746185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588189+0000","last_change":"2024-05-27T00:15:25.620621+0000","last_active":"2024-05-27T00:15:26.588189+0000","last_peered":"2024-05-27T00:15:26.588189+0000","last_clean":"2024-05-27T00:15:26.588189+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:15:26.588189+0000","last_undegraded":"2024-05-27T00:15:26.588189+0000","last_fullsized":"2024-05-27T00:15:26.588189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:22:30.152810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744910+0000","last_change":"2024-05-27T00:15:25.605839+0000","last_active":"2024-05-27T00:15:26.744910+0000","last_peered":"2024-05-27T00:15:26.744910+0000","last_clean":"2024-05-27T00:15:26.744910+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:15:26.744910+0000","last_undegraded":"2024-05-27T00:15:26.744910+0000","last_fullsized":"2024-05-27T00:15:26.744910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746666+0000","last_change":"2024-05-27T00:15:25.607188+0000","last_active":"2024-05-27T00:15:26.746666+0000","last_peered":"2024-05-27T00:15:26.746666+0000","last_clean":"2024-05-27T00:15:26.746666+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:15:26.746666+0000","last_undegraded":"2024-05-27T00:15:26.746666+0000","last_fullsized":"2024-05-27T00:15:26.746666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587251+0000","last_change":"2024-05-27T00:15:25.615579+0000","last_active":"2024-05-27T00:15:26.587251+0000","last_peered":"2024-05-27T00:15:26.587251+0000","last_clean":"2024-05-27T00:15:26.587251+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:15:26.587251+0000","last_undegraded":"2024-05-27T00:15:26.587251+0000","last_fullsized":"2024-05-27T00:15:26.587251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:53.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746392+0000","last_change":"2024-05-27T00:15:24.611108+0000","last_active":"2024-05-27T00:15:26.746392+0000","last_peered":"2024-05-27T00:15:26.746392+0000","last_clean":"2024-05-27T00:15:26.746392+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:15:26.746392+0000","last_undegraded":"2024-05-27T00:15:26.746392+0000","last_fullsized":"2024-05-27T00:15:26.746392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:11.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746332+0000","last_change":"2024-05-27T00:15:25.614251+0000","last_active":"2024-05-27T00:15:26.746332+0000","last_peered":"2024-05-27T00:15:26.746332+0000","last_clean":"2024-05-27T00:15:26.746332+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:15:26.746332+0000","last_undegraded":"2024-05-27T00:15:26.746332+0000","last_fullsized":"2024-05-27T00:15:26.746332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747555+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:15:26.747555+0000","last_peered":"2024-05-27T00:15:26.747555+0000","last_clean":"2024-05-27T00:15:26.747555+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:15:26.747555+0000","last_undegraded":"2024-05-27T00:15:26.747555+0000","last_fullsized":"2024-05-27T00:15:26.747555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745607+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:15:26.745607+0000","last_peered":"2024-05-27T00:15:26.745607+0000","last_clean":"2024-05-27T00:15:26.745607+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:15:26.745607+0000","last_undegraded":"2024-05-27T00:15:26.745607+0000","last_fullsized":"2024-05-27T00:15:26.745607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:46:08.645927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745132+0000","last_change":"2024-05-27T00:15:25.596844+0000","last_active":"2024-05-27T00:15:26.745132+0000","last_peered":"2024-05-27T00:15:26.745132+0000","last_clean":"2024-05-27T00:15:26.745132+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:15:26.745132+0000","last_undegraded":"2024-05-27T00:15:26.745132+0000","last_fullsized":"2024-05-27T00:15:26.745132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:17:40.503615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587344+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:15:26.587344+0000","last_peered":"2024-05-27T00:15:26.587344+0000","last_clean":"2024-05-27T00:15:26.587344+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:15:26.587344+0000","last_undegraded":"2024-05-27T00:15:26.587344+0000","last_fullsized":"2024-05-27T00:15:26.587344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583235+0000","last_change":"2024-05-27T00:15:25.593149+0000","last_active":"2024-05-27T00:15:26.583235+0000","last_peered":"2024-05-27T00:15:26.583235+0000","last_clean":"2024-05-27T00:15:26.583235+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:15:26.583235+0000","last_undegraded":"2024-05-27T00:15:26.583235+0000","last_fullsized":"2024-05-27T00:15:26.583235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746974+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:15:26.746974+0000","last_peered":"2024-05-27T00:15:26.746974+0000","last_clean":"2024-05-27T00:15:26.746974+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:15:26.746974+0000","last_undegraded":"2024-05-27T00:15:26.746974+0000","last_fullsized":"2024-05-27T00:15:26.746974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589140+0000","last_change":"2024-05-27T00:15:25.618650+0000","last_active":"2024-05-27T00:15:26.589140+0000","last_peered":"2024-05-27T00:15:26.589140+0000","last_clean":"2024-05-27T00:15:26.589140+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:15:26.589140+0000","last_undegraded":"2024-05-27T00:15:26.589140+0000","last_fullsized":"2024-05-27T00:15:26.589140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:21.935526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589356+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:15:26.589356+0000","last_peered":"2024-05-27T00:15:26.589356+0000","last_clean":"2024-05-27T00:15:26.589356+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:15:26.589356+0000","last_undegraded":"2024-05-27T00:15:26.589356+0000","last_fullsized":"2024-05-27T00:15:26.589356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746448+0000","last_change":"2024-05-27T00:15:25.613142+0000","last_active":"2024-05-27T00:15:26.746448+0000","last_peered":"2024-05-27T00:15:26.746448+0000","last_clean":"2024-05-27T00:15:26.746448+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:15:26.746448+0000","last_undegraded":"2024-05-27T00:15:26.746448+0000","last_fullsized":"2024-05-27T00:15:26.746448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.620068+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:15:30.620068+0000","last_peered":"2024-05-27T00:15:30.620068+0000","last_clean":"2024-05-27T00:15:30.620068+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:15:30.620068+0000","last_undegraded":"2024-05-27T00:15:30.620068+0000","last_fullsized":"2024-05-27T00:15:30.620068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588245+0000","last_change":"2024-05-27T00:15:25.622383+0000","last_active":"2024-05-27T00:15:26.588245+0000","last_peered":"2024-05-27T00:15:26.588245+0000","last_clean":"2024-05-27T00:15:26.588245+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:15:26.588245+0000","last_undegraded":"2024-05-27T00:15:26.588245+0000","last_fullsized":"2024-05-27T00:15:26.588245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.627016+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:15:30.627016+0000","last_peered":"2024-05-27T00:15:30.627016+0000","last_clean":"2024-05-27T00:15:30.627016+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:15:30.627016+0000","last_undegraded":"2024-05-27T00:15:30.627016+0000","last_fullsized":"2024-05-27T00:15:30.627016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1866,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748511+0000","last_change":"2024-05-27T00:15:25.628646+0000","last_active":"2024-05-27T00:15:26.748511+0000","last_peered":"2024-05-27T00:15:26.748511+0000","last_clean":"2024-05-27T00:15:26.748511+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:15:26.748511+0000","last_undegraded":"2024-05-27T00:15:26.748511+0000","last_fullsized":"2024-05-27T00:15:26.748511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:22:30.152810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619790+0000","last_change":"2024-05-27T00:15:24.605986+0000","last_active":"2024-05-27T00:15:30.619790+0000","last_peered":"2024-05-27T00:15:30.619790+0000","last_clean":"2024-05-27T00:15:30.619790+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:15:30.619790+0000","last_undegraded":"2024-05-27T00:15:30.619790+0000","last_fullsized":"2024-05-27T00:15:30.619790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745414+0000","last_change":"2024-05-27T00:15:25.624136+0000","last_active":"2024-05-27T00:15:26.745414+0000","last_peered":"2024-05-27T00:15:26.745414+0000","last_clean":"2024-05-27T00:15:26.745414+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:15:26.745414+0000","last_undegraded":"2024-05-27T00:15:26.745414+0000","last_fullsized":"2024-05-27T00:15:26.745414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588435+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:15:26.588435+0000","last_peered":"2024-05-27T00:15:26.588435+0000","last_clean":"2024-05-27T00:15:26.588435+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:15:26.588435+0000","last_undegraded":"2024-05-27T00:15:26.588435+0000","last_fullsized":"2024-05-27T00:15:26.588435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748720+0000","last_change":"2024-05-27T00:15:25.641965+0000","last_active":"2024-05-27T00:15:26.748720+0000","last_peered":"2024-05-27T00:15:26.748720+0000","last_clean":"2024-05-27T00:15:26.748720+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:15:26.748720+0000","last_undegraded":"2024-05-27T00:15:26.748720+0000","last_fullsized":"2024-05-27T00:15:26.748720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:17.513529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746779+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:15:26.746779+0000","last_peered":"2024-05-27T00:15:26.746779+0000","last_clean":"2024-05-27T00:15:26.746779+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:15:26.746779+0000","last_undegraded":"2024-05-27T00:15:26.746779+0000","last_fullsized":"2024-05-27T00:15:26.746779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746732+0000","last_change":"2024-05-27T00:15:25.608549+0000","last_active":"2024-05-27T00:15:26.746732+0000","last_peered":"2024-05-27T00:15:26.746732+0000","last_clean":"2024-05-27T00:15:26.746732+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:15:26.746732+0000","last_undegraded":"2024-05-27T00:15:26.746732+0000","last_fullsized":"2024-05-27T00:15:26.746732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619615+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:15:30.619615+0000","last_peered":"2024-05-27T00:15:30.619615+0000","last_clean":"2024-05-27T00:15:30.619615+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:15:30.619615+0000","last_undegraded":"2024-05-27T00:15:30.619615+0000","last_fullsized":"2024-05-27T00:15:30.619615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747206+0000","last_change":"2024-05-27T00:15:25.638558+0000","last_active":"2024-05-27T00:15:26.747206+0000","last_peered":"2024-05-27T00:15:26.747206+0000","last_clean":"2024-05-27T00:15:26.747206+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:15:26.747206+0000","last_undegraded":"2024-05-27T00:15:26.747206+0000","last_fullsized":"2024-05-27T00:15:26.747206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:02:15.012827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587664+0000","last_change":"2024-05-27T00:15:24.601924+0000","last_active":"2024-05-27T00:15:26.587664+0000","last_peered":"2024-05-27T00:15:26.587664+0000","last_clean":"2024-05-27T00:15:26.587664+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:15:26.587664+0000","last_undegraded":"2024-05-27T00:15:26.587664+0000","last_fullsized":"2024-05-27T00:15:26.587664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:32.185839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749041+0000","last_change":"2024-05-27T00:15:25.639140+0000","last_active":"2024-05-27T00:15:26.749041+0000","last_peered":"2024-05-27T00:15:26.749041+0000","last_clean":"2024-05-27T00:15:26.749041+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:15:26.749041+0000","last_undegraded":"2024-05-27T00:15:26.749041+0000","last_fullsized":"2024-05-27T00:15:26.749041+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:21.935526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619020+0000","last_change":"2024-05-27T00:15:24.600878+0000","last_active":"2024-05-27T00:15:30.619020+0000","last_peered":"2024-05-27T00:15:30.619020+0000","last_clean":"2024-05-27T00:15:30.619020+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:15:30.619020+0000","last_undegraded":"2024-05-27T00:15:30.619020+0000","last_fullsized":"2024-05-27T00:15:30.619020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749435+0000","last_change":"2024-05-27T00:15:25.644249+0000","last_active":"2024-05-27T00:15:26.749435+0000","last_peered":"2024-05-27T00:15:26.749435+0000","last_clean":"2024-05-27T00:15:26.749435+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:15:26.749435+0000","last_undegraded":"2024-05-27T00:15:26.749435+0000","last_fullsized":"2024-05-27T00:15:26.749435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:59:25.042662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583715+0000","last_change":"2024-05-27T00:15:24.603452+0000","last_active":"2024-05-27T00:15:26.583715+0000","last_peered":"2024-05-27T00:15:26.583715+0000","last_clean":"2024-05-27T00:15:26.583715+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:15:26.583715+0000","last_undegraded":"2024-05-27T00:15:26.583715+0000","last_fullsized":"2024-05-27T00:15:26.583715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587865+0000","last_change":"2024-05-27T00:15:25.619341+0000","last_active":"2024-05-27T00:15:26.587865+0000","last_peered":"2024-05-27T00:15:26.587865+0000","last_clean":"2024-05-27T00:15:26.587865+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:15:26.587865+0000","last_undegraded":"2024-05-27T00:15:26.587865+0000","last_fullsized":"2024-05-27T00:15:26.587865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:41:46.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745117+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:15:26.745117+0000","last_peered":"2024-05-27T00:15:26.745117+0000","last_clean":"2024-05-27T00:15:26.745117+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:15:26.745117+0000","last_undegraded":"2024-05-27T00:15:26.745117+0000","last_fullsized":"2024-05-27T00:15:26.745117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748893+0000","last_change":"2024-05-27T00:15:25.639462+0000","last_active":"2024-05-27T00:15:26.748893+0000","last_peered":"2024-05-27T00:15:26.748893+0000","last_clean":"2024-05-27T00:15:26.748893+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:15:26.748893+0000","last_undegraded":"2024-05-27T00:15:26.748893+0000","last_fullsized":"2024-05-27T00:15:26.748893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:52:55.175945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746823+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:15:26.746823+0000","last_peered":"2024-05-27T00:15:26.746823+0000","last_clean":"2024-05-27T00:15:26.746823+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:15:26.746823+0000","last_undegraded":"2024-05-27T00:15:26.746823+0000","last_fullsized":"2024-05-27T00:15:26.746823+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586683+0000","last_change":"2024-05-27T00:15:24.603781+0000","last_active":"2024-05-27T00:15:26.586683+0000","last_peered":"2024-05-27T00:15:26.586683+0000","last_clean":"2024-05-27T00:15:26.586683+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:15:26.586683+0000","last_undegraded":"2024-05-27T00:15:26.586683+0000","last_fullsized":"2024-05-27T00:15:26.586683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746108+0000","last_change":"2024-05-27T00:15:25.631978+0000","last_active":"2024-05-27T00:15:26.746108+0000","last_peered":"2024-05-27T00:15:26.746108+0000","last_clean":"2024-05-27T00:15:26.746108+0000","last_became_active":"2024-05-27T00:15:25.631697+0000","last_became_peered":"2024-05-27T00:15:25.631697+0000","last_unstale":"2024-05-27T00:15:26.746108+0000","last_undegraded":"2024-05-27T00:15:26.746108+0000","last_fullsized":"2024-05-27T00:15:26.746108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:05.141408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746536+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:15:26.746536+0000","last_peered":"2024-05-27T00:15:26.746536+0000","last_clean":"2024-05-27T00:15:26.746536+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:15:26.746536+0000","last_undegraded":"2024-05-27T00:15:26.746536+0000","last_fullsized":"2024-05-27T00:15:26.746536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746253+0000","last_change":"2024-05-27T00:15:25.610627+0000","last_active":"2024-05-27T00:15:26.746253+0000","last_peered":"2024-05-27T00:15:26.746253+0000","last_clean":"2024-05-27T00:15:26.746253+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:15:26.746253+0000","last_undegraded":"2024-05-27T00:15:26.746253+0000","last_fullsized":"2024-05-27T00:15:26.746253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744898+0000","last_change":"2024-05-27T00:15:24.598182+0000","last_active":"2024-05-27T00:15:26.744898+0000","last_peered":"2024-05-27T00:15:26.744898+0000","last_clean":"2024-05-27T00:15:26.744898+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:15:26.744898+0000","last_undegraded":"2024-05-27T00:15:26.744898+0000","last_fullsized":"2024-05-27T00:15:26.744898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583243+0000","last_change":"2024-05-27T00:15:25.591449+0000","last_active":"2024-05-27T00:15:26.583243+0000","last_peered":"2024-05-27T00:15:26.583243+0000","last_clean":"2024-05-27T00:15:26.583243+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:15:26.583243+0000","last_undegraded":"2024-05-27T00:15:26.583243+0000","last_fullsized":"2024-05-27T00:15:26.583243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588693+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:15:26.588693+0000","last_peered":"2024-05-27T00:15:26.588693+0000","last_clean":"2024-05-27T00:15:26.588693+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:15:26.588693+0000","last_undegraded":"2024-05-27T00:15:26.588693+0000","last_fullsized":"2024-05-27T00:15:26.588693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587675+0000","last_change":"2024-05-27T00:15:25.614073+0000","last_active":"2024-05-27T00:15:26.587675+0000","last_peered":"2024-05-27T00:15:26.587675+0000","last_clean":"2024-05-27T00:15:26.587675+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:15:26.587675+0000","last_undegraded":"2024-05-27T00:15:26.587675+0000","last_fullsized":"2024-05-27T00:15:26.587675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583569+0000","last_change":"2024-05-27T00:15:24.603518+0000","last_active":"2024-05-27T00:15:26.583569+0000","last_peered":"2024-05-27T00:15:26.583569+0000","last_clean":"2024-05-27T00:15:26.583569+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:15:26.583569+0000","last_undegraded":"2024-05-27T00:15:26.583569+0000","last_fullsized":"2024-05-27T00:15:26.583569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588810+0000","last_change":"2024-05-27T00:15:25.621519+0000","last_active":"2024-05-27T00:15:26.588810+0000","last_peered":"2024-05-27T00:15:26.588810+0000","last_clean":"2024-05-27T00:15:26.588810+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:15:26.588810+0000","last_undegraded":"2024-05-27T00:15:26.588810+0000","last_fullsized":"2024-05-27T00:15:26.588810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:52:55.175945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749299+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:15:26.749299+0000","last_peered":"2024-05-27T00:15:26.749299+0000","last_clean":"2024-05-27T00:15:26.749299+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:15:26.749299+0000","last_undegraded":"2024-05-27T00:15:26.749299+0000","last_fullsized":"2024-05-27T00:15:26.749299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583460+0000","last_change":"2024-05-27T00:15:25.593180+0000","last_active":"2024-05-27T00:15:26.583460+0000","last_peered":"2024-05-27T00:15:26.583460+0000","last_clean":"2024-05-27T00:15:26.583460+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:15:26.583460+0000","last_undegraded":"2024-05-27T00:15:26.583460+0000","last_fullsized":"2024-05-27T00:15:26.583460+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583701+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:15:26.583701+0000","last_peered":"2024-05-27T00:15:26.583701+0000","last_clean":"2024-05-27T00:15:26.583701+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:15:26.583701+0000","last_undegraded":"2024-05-27T00:15:26.583701+0000","last_fullsized":"2024-05-27T00:15:26.583701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745915+0000","last_change":"2024-05-27T00:15:25.631919+0000","last_active":"2024-05-27T00:15:26.745915+0000","last_peered":"2024-05-27T00:15:26.745915+0000","last_clean":"2024-05-27T00:15:26.745915+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:15:26.745915+0000","last_undegraded":"2024-05-27T00:15:26.745915+0000","last_fullsized":"2024-05-27T00:15:26.745915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:24:31.995603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.620892+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:15:30.620892+0000","last_peered":"2024-05-27T00:15:30.620892+0000","last_clean":"2024-05-27T00:15:30.620892+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:15:30.620892+0000","last_undegraded":"2024-05-27T00:15:30.620892+0000","last_fullsized":"2024-05-27T00:15:30.620892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745629+0000","last_change":"2024-05-27T00:15:25.596792+0000","last_active":"2024-05-27T00:15:26.745629+0000","last_peered":"2024-05-27T00:15:26.745629+0000","last_clean":"2024-05-27T00:15:26.745629+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:15:26.745629+0000","last_undegraded":"2024-05-27T00:15:26.745629+0000","last_fullsized":"2024-05-27T00:15:26.745629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747720+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:15:26.747720+0000","last_peered":"2024-05-27T00:15:26.747720+0000","last_clean":"2024-05-27T00:15:26.747720+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:15:26.747720+0000","last_undegraded":"2024-05-27T00:15:26.747720+0000","last_fullsized":"2024-05-27T00:15:26.747720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587779+0000","last_change":"2024-05-27T00:15:25.610697+0000","last_active":"2024-05-27T00:15:26.587779+0000","last_peered":"2024-05-27T00:15:26.587779+0000","last_clean":"2024-05-27T00:15:26.587779+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:15:26.587779+0000","last_undegraded":"2024-05-27T00:15:26.587779+0000","last_fullsized":"2024-05-27T00:15:26.587779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:14.969827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583375+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:15:26.583375+0000","last_peered":"2024-05-27T00:15:26.583375+0000","last_clean":"2024-05-27T00:15:26.583375+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:15:26.583375+0000","last_undegraded":"2024-05-27T00:15:26.583375+0000","last_fullsized":"2024-05-27T00:15:26.583375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745282+0000","last_change":"2024-05-27T00:15:25.597164+0000","last_active":"2024-05-27T00:15:26.745282+0000","last_peered":"2024-05-27T00:15:26.745282+0000","last_clean":"2024-05-27T00:15:26.745282+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:15:26.745282+0000","last_undegraded":"2024-05-27T00:15:26.745282+0000","last_fullsized":"2024-05-27T00:15:26.745282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:04.655872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583422+0000","last_change":"2024-05-27T00:15:24.598238+0000","last_active":"2024-05-27T00:15:26.583422+0000","last_peered":"2024-05-27T00:15:26.583422+0000","last_clean":"2024-05-27T00:15:26.583422+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:15:26.583422+0000","last_undegraded":"2024-05-27T00:15:26.583422+0000","last_fullsized":"2024-05-27T00:15:26.583422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746235+0000","last_change":"2024-05-27T00:15:25.601140+0000","last_active":"2024-05-27T00:15:26.746235+0000","last_peered":"2024-05-27T00:15:26.746235+0000","last_clean":"2024-05-27T00:15:26.746235+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:15:26.746235+0000","last_undegraded":"2024-05-27T00:15:26.746235+0000","last_fullsized":"2024-05-27T00:15:26.746235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:42:05.334730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.621275+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:15:30.621275+0000","last_peered":"2024-05-27T00:15:30.621275+0000","last_clean":"2024-05-27T00:15:30.621275+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:15:30.621275+0000","last_undegraded":"2024-05-27T00:15:30.621275+0000","last_fullsized":"2024-05-27T00:15:30.621275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747292+0000","last_change":"2024-05-27T00:15:25.619694+0000","last_active":"2024-05-27T00:15:26.747292+0000","last_peered":"2024-05-27T00:15:26.747292+0000","last_clean":"2024-05-27T00:15:26.747292+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:15:26.747292+0000","last_undegraded":"2024-05-27T00:15:26.747292+0000","last_fullsized":"2024-05-27T00:15:26.747292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588751+0000","last_change":"2024-05-27T00:15:25.627578+0000","last_active":"2024-05-27T00:15:26.588751+0000","last_peered":"2024-05-27T00:15:26.588751+0000","last_clean":"2024-05-27T00:15:26.588751+0000","last_became_active":"2024-05-27T00:15:25.622421+0000","last_became_peered":"2024-05-27T00:15:25.622421+0000","last_unstale":"2024-05-27T00:15:26.588751+0000","last_undegraded":"2024-05-27T00:15:26.588751+0000","last_fullsized":"2024-05-27T00:15:26.588751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:17.513529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619639+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:15:30.619639+0000","last_peered":"2024-05-27T00:15:30.619639+0000","last_clean":"2024-05-27T00:15:30.619639+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:15:30.619639+0000","last_undegraded":"2024-05-27T00:15:30.619639+0000","last_fullsized":"2024-05-27T00:15:30.619639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587486+0000","last_change":"2024-05-27T00:15:25.623207+0000","last_active":"2024-05-27T00:15:26.587486+0000","last_peered":"2024-05-27T00:15:26.587486+0000","last_clean":"2024-05-27T00:15:26.587486+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:15:26.587486+0000","last_undegraded":"2024-05-27T00:15:26.587486+0000","last_fullsized":"2024-05-27T00:15:26.587486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:59.804368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.618376+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:15:30.618376+0000","last_peered":"2024-05-27T00:15:30.618376+0000","last_clean":"2024-05-27T00:15:30.618376+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:15:30.618376+0000","last_undegraded":"2024-05-27T00:15:30.618376+0000","last_fullsized":"2024-05-27T00:15:30.618376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746305+0000","last_change":"2024-05-27T00:15:25.613018+0000","last_active":"2024-05-27T00:15:26.746305+0000","last_peered":"2024-05-27T00:15:26.746305+0000","last_clean":"2024-05-27T00:15:26.746305+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:15:26.746305+0000","last_undegraded":"2024-05-27T00:15:26.746305+0000","last_fullsized":"2024-05-27T00:15:26.746305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745698+0000","last_change":"2024-05-27T00:15:24.600571+0000","last_active":"2024-05-27T00:15:26.745698+0000","last_peered":"2024-05-27T00:15:26.745698+0000","last_clean":"2024-05-27T00:15:26.745698+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:15:26.745698+0000","last_undegraded":"2024-05-27T00:15:26.745698+0000","last_fullsized":"2024-05-27T00:15:26.745698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:44:21.645223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745724+0000","last_change":"2024-05-27T00:15:25.615649+0000","last_active":"2024-05-27T00:15:26.745724+0000","last_peered":"2024-05-27T00:15:26.745724+0000","last_clean":"2024-05-27T00:15:26.745724+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:15:26.745724+0000","last_undegraded":"2024-05-27T00:15:26.745724+0000","last_fullsized":"2024-05-27T00:15:26.745724+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:16:16.309939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.621025+0000","last_change":"2024-05-27T00:15:24.615066+0000","last_active":"2024-05-27T00:15:30.621025+0000","last_peered":"2024-05-27T00:15:30.621025+0000","last_clean":"2024-05-27T00:15:30.621025+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:15:30.621025+0000","last_undegraded":"2024-05-27T00:15:30.621025+0000","last_fullsized":"2024-05-27T00:15:30.621025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2664,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":669858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542155,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":80774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542156,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":79034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":80774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":78908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542155,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":79068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":669914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:15:47.552 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-05-27T00:15:47.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:15:47.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.721+0000 7fda27800640 1 Processor -- start 2024-05-27T00:15:47.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.721+0000 7fda27800640 1 -- start start 2024-05-27T00:15:47.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda27800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fda2006b970 0x7fda200730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda27800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda2006cb30 con 0x7fda20073620 2024-05-27T00:15:47.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda27800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda200c7b20 con 0x7fda20158080 2024-05-27T00:15:47.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda27800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fda200c9820 con 0x7fda2006b970 2024-05-27T00:15:47.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda25800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fda20073620 0x7fda20165140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45882/0 (socket says 172.21.15.98:45882) 2024-05-27T00:15:47.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda25800640 1 -- 172.21.15.98:0/1186008389 learned_addr learned my addr 172.21.15.98:0/1186008389 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:47.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda17e00640 1 --2- 172.21.15.98:0/1186008389 >> v2:172.21.15.204:3300/0 conn(0x7fda2006b970 0x7fda200730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda1fe00640 1 -- 172.21.15.98:0/1186008389 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 445885881 0 0) 0x7fda2006cb30 con 0x7fda20073620 2024-05-27T00:15:47.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda1fe00640 1 -- 172.21.15.98:0/1186008389 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda04003530 con 0x7fda20073620 2024-05-27T00:15:47.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda17e00640 1 -- 172.21.15.98:0/1186008389 >> v1:172.21.15.204:6789/0 conn(0x7fda20158080 legacy=0x7fda200c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda17e00640 1 -- 172.21.15.98:0/1186008389 >> v1:172.21.15.98:6789/0 conn(0x7fda20073620 legacy=0x7fda20165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.725+0000 7fda17e00640 1 -- 172.21.15.98:0/1186008389 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2006c570 con 0x7fda2006b970 2024-05-27T00:15:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda17e00640 1 --2- 172.21.15.98:0/1186008389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda2006b970 0x7fda200730e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fda080097c0 tx=0x7fda0801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c49902e6a0ba2447 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda1fe00640 1 -- 172.21.15.98:0/1186008389 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080b0070 con 0x7fda2006b970 2024-05-27T00:15:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda1fe00640 1 -- 172.21.15.98:0/1186008389 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda080ab600 con 0x7fda2006b970 2024-05-27T00:15:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda1fe00640 1 -- 172.21.15.98:0/1186008389 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080ab9d0 con 0x7fda2006b970 2024-05-27T00:15:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda27800640 1 -- 172.21.15.98:0/1186008389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda2006b970 msgr2=0x7fda200730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda27800640 1 --2- 172.21.15.98:0/1186008389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda2006b970 0x7fda200730e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fda080097c0 tx=0x7fda0801c4a0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda27800640 1 -- 172.21.15.98:0/1186008389 shutdown_connections 2024-05-27T00:15:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda27800640 1 --2- 172.21.15.98:0/1186008389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda2006b970 0x7fda200730e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda27800640 1 -- 172.21.15.98:0/1186008389 >> 172.21.15.98:0/1186008389 conn(0x7fda2006fa70 msgr2=0x7fda2006fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda27800640 1 -- 172.21.15.98:0/1186008389 shutdown_connections 2024-05-27T00:15:47.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.729+0000 7fda27800640 1 -- 172.21.15.98:0/1186008389 wait complete. 2024-05-27T00:15:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 Processor -- start 2024-05-27T00:15:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 -- start start 2024-05-27T00:15:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2006b970 0x7fda2014e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda20073620 0x7fda20155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda20158080 0x7fda20156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda200ca310 con 0x7fda2006b970 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fda200ca190 con 0x7fda20073620 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fda200c4030 con 0x7fda20158080 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda24e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2006b970 0x7fda2014e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda24e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2006b970 0x7fda2014e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54030/0 (socket says 172.21.15.98:54030) 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda24e00640 1 -- 172.21.15.98:0/3213593378 learned_addr learned my addr 172.21.15.98:0/3213593378 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda17e00640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda20073620 0x7fda20155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda25800640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda20158080 0x7fda20156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda24e00640 1 -- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda20158080 msgr2=0x7fda20156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda24e00640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda20158080 0x7fda20156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda24e00640 1 -- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda20073620 msgr2=0x7fda20155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda24e00640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda20073620 0x7fda20155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda24e00640 1 -- 172.21.15.98:0/3213593378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda201f38d0 con 0x7fda2006b970 2024-05-27T00:15:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda25800640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda20158080 0x7fda20156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:15:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda24e00640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2006b970 0x7fda2014e250 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fda1000c530 tx=0x7fda1000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda1d600640 1 -- 172.21.15.98:0/3213593378 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10016020 con 0x7fda2006b970 2024-05-27T00:15:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda1d600640 1 -- 172.21.15.98:0/3213593378 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda10005220 con 0x7fda2006b970 2024-05-27T00:15:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda1d600640 1 -- 172.21.15.98:0/3213593378 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10007030 con 0x7fda2006b970 2024-05-27T00:15:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 -- 172.21.15.98:0/3213593378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda201f3b90 con 0x7fda2006b970 2024-05-27T00:15:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 -- 172.21.15.98:0/3213593378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda201f4150 con 0x7fda2006b970 2024-05-27T00:15:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.733+0000 7fda27800640 1 -- 172.21.15.98:0/3213593378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda20075cb0 con 0x7fda2006b970 2024-05-27T00:15:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.737+0000 7fda1d600640 1 -- 172.21.15.98:0/3213593378 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fda100071d0 con 0x7fda2006b970 2024-05-27T00:15:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.737+0000 7fda1d600640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9fc0678f0 0x7fd9fc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.737+0000 7fda1d600640 1 -- 172.21.15.98:0/3213593378 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fda10085620 con 0x7fda2006b970 2024-05-27T00:15:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.737+0000 7fda17e00640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9fc0678f0 0x7fd9fc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.737+0000 7fda17e00640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9fc0678f0 0x7fd9fc069db0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fda080ad000 tx=0x7fda080a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:47.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.741+0000 7fda1d600640 1 -- 172.21.15.98:0/3213593378 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fda10020020 con 0x7fda2006b970 2024-05-27T00:15:47.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.897+0000 7fda27800640 1 -- 172.21.15.98:0/3213593378 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fda200b1c90 con 0x7fd9fc0678f0 2024-05-27T00:15:47.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.913+0000 7fda1d600640 1 -- 172.21.15.98:0/3213593378 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251854 (secure 0 0 0) 0x7fda200b1c90 con 0x7fd9fc0678f0 2024-05-27T00:15:47.918 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:47.920 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:15:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 -- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9fc0678f0 msgr2=0x7fd9fc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9fc0678f0 0x7fd9fc069db0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fda080ad000 tx=0x7fda080a6000 comp rx=0 tx=0).stop 2024-05-27T00:15:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 -- 172.21.15.98:0/3213593378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2006b970 msgr2=0x7fda2014e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2006b970 0x7fda2014e250 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fda1000c530 tx=0x7fda1000c9f0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 -- 172.21.15.98:0/3213593378 shutdown_connections 2024-05-27T00:15:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda20158080 0x7fda20156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9fc0678f0 0x7fd9fc069db0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda20073620 0x7fda20155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 --2- 172.21.15.98:0/3213593378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda2006b970 0x7fda2014e250 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:47.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 -- 172.21.15.98:0/3213593378 >> 172.21.15.98:0/3213593378 conn(0x7fda2006fa70 msgr2=0x7fda200c9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:47.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 -- 172.21.15.98:0/3213593378 shutdown_connections 2024-05-27T00:15:47.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:47.917+0000 7fda27800640 1 -- 172.21.15.98:0/3213593378 wait complete. 2024-05-27T00:15:47.939 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-05-27T00:15:47.140432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593032,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":205,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219112,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755608,"statfs":{"total":773094113280,"available":772869742592,"internally_reserved":0,"allocated":3055616,"data_stored":1817328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":254,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004211"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748978+0000","last_change":"2024-05-27T00:15:25.640698+0000","last_active":"2024-05-27T00:15:26.748978+0000","last_peered":"2024-05-27T00:15:26.748978+0000","last_clean":"2024-05-27T00:15:26.748978+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:15:26.748978+0000","last_undegraded":"2024-05-27T00:15:26.748978+0000","last_fullsized":"2024-05-27T00:15:26.748978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:47:27.030637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583615+0000","last_change":"2024-05-27T00:15:25.598672+0000","last_active":"2024-05-27T00:15:26.583615+0000","last_peered":"2024-05-27T00:15:26.583615+0000","last_clean":"2024-05-27T00:15:26.583615+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:15:26.583615+0000","last_undegraded":"2024-05-27T00:15:26.583615+0000","last_fullsized":"2024-05-27T00:15:26.583615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745467+0000","last_change":"2024-05-27T00:15:25.593316+0000","last_active":"2024-05-27T00:15:26.745467+0000","last_peered":"2024-05-27T00:15:26.745467+0000","last_clean":"2024-05-27T00:15:26.745467+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:15:26.745467+0000","last_undegraded":"2024-05-27T00:15:26.745467+0000","last_fullsized":"2024-05-27T00:15:26.745467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587216+0000","last_change":"2024-05-27T00:15:25.620237+0000","last_active":"2024-05-27T00:15:26.587216+0000","last_peered":"2024-05-27T00:15:26.587216+0000","last_clean":"2024-05-27T00:15:26.587216+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:15:26.587216+0000","last_undegraded":"2024-05-27T00:15:26.587216+0000","last_fullsized":"2024-05-27T00:15:26.587216+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:17:53.053461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748475+0000","last_change":"2024-05-27T00:15:25.640146+0000","last_active":"2024-05-27T00:15:26.748475+0000","last_peered":"2024-05-27T00:15:26.748475+0000","last_clean":"2024-05-27T00:15:26.748475+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:15:26.748475+0000","last_undegraded":"2024-05-27T00:15:26.748475+0000","last_fullsized":"2024-05-27T00:15:26.748475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:41:46.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589129+0000","last_change":"2024-05-27T00:15:25.627128+0000","last_active":"2024-05-27T00:15:26.589129+0000","last_peered":"2024-05-27T00:15:26.589129+0000","last_clean":"2024-05-27T00:15:26.589129+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:15:26.589129+0000","last_undegraded":"2024-05-27T00:15:26.589129+0000","last_fullsized":"2024-05-27T00:15:26.589129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:47:27.030637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746743+0000","last_change":"2024-05-27T00:15:25.630446+0000","last_active":"2024-05-27T00:15:26.746743+0000","last_peered":"2024-05-27T00:15:26.746743+0000","last_clean":"2024-05-27T00:15:26.746743+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:15:26.746743+0000","last_undegraded":"2024-05-27T00:15:26.746743+0000","last_fullsized":"2024-05-27T00:15:26.746743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:59.804368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744980+0000","last_change":"2024-05-27T00:15:25.612157+0000","last_active":"2024-05-27T00:15:26.744980+0000","last_peered":"2024-05-27T00:15:26.744980+0000","last_clean":"2024-05-27T00:15:26.744980+0000","last_became_active":"2024-05-27T00:15:25.611884+0000","last_became_peered":"2024-05-27T00:15:25.611884+0000","last_unstale":"2024-05-27T00:15:26.744980+0000","last_undegraded":"2024-05-27T00:15:26.744980+0000","last_fullsized":"2024-05-27T00:15:26.744980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:12:28.505840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749289+0000","last_change":"2024-05-27T00:15:25.643312+0000","last_active":"2024-05-27T00:15:26.749289+0000","last_peered":"2024-05-27T00:15:26.749289+0000","last_clean":"2024-05-27T00:15:26.749289+0000","last_became_active":"2024-05-27T00:15:25.642663+0000","last_became_peered":"2024-05-27T00:15:25.642663+0000","last_unstale":"2024-05-27T00:15:26.749289+0000","last_undegraded":"2024-05-27T00:15:26.749289+0000","last_fullsized":"2024-05-27T00:15:26.749289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:06:27.232620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747471+0000","last_change":"2024-05-27T00:15:25.629892+0000","last_active":"2024-05-27T00:15:26.747471+0000","last_peered":"2024-05-27T00:15:26.747471+0000","last_clean":"2024-05-27T00:15:26.747471+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:15:26.747471+0000","last_undegraded":"2024-05-27T00:15:26.747471+0000","last_fullsized":"2024-05-27T00:15:26.747471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747909+0000","last_change":"2024-05-27T00:15:25.614887+0000","last_active":"2024-05-27T00:15:26.747909+0000","last_peered":"2024-05-27T00:15:26.747909+0000","last_clean":"2024-05-27T00:15:26.747909+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:15:26.747909+0000","last_undegraded":"2024-05-27T00:15:26.747909+0000","last_fullsized":"2024-05-27T00:15:26.747909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588218+0000","last_change":"2024-05-27T00:15:25.623864+0000","last_active":"2024-05-27T00:15:26.588218+0000","last_peered":"2024-05-27T00:15:26.588218+0000","last_clean":"2024-05-27T00:15:26.588218+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:15:26.588218+0000","last_undegraded":"2024-05-27T00:15:26.588218+0000","last_fullsized":"2024-05-27T00:15:26.588218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746025+0000","last_change":"2024-05-27T00:15:25.615495+0000","last_active":"2024-05-27T00:15:26.746025+0000","last_peered":"2024-05-27T00:15:26.746025+0000","last_clean":"2024-05-27T00:15:26.746025+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:15:26.746025+0000","last_undegraded":"2024-05-27T00:15:26.746025+0000","last_fullsized":"2024-05-27T00:15:26.746025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:42:53.882749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745273+0000","last_change":"2024-05-27T00:15:25.594508+0000","last_active":"2024-05-27T00:15:26.745273+0000","last_peered":"2024-05-27T00:15:26.745273+0000","last_clean":"2024-05-27T00:15:26.745273+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:15:26.745273+0000","last_undegraded":"2024-05-27T00:15:26.745273+0000","last_fullsized":"2024-05-27T00:15:26.745273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:07:53.407750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583449+0000","last_change":"2024-05-27T00:15:25.611818+0000","last_active":"2024-05-27T00:15:26.583449+0000","last_peered":"2024-05-27T00:15:26.583449+0000","last_clean":"2024-05-27T00:15:26.583449+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:15:26.583449+0000","last_undegraded":"2024-05-27T00:15:26.583449+0000","last_fullsized":"2024-05-27T00:15:26.583449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586675+0000","last_change":"2024-05-27T00:15:25.611365+0000","last_active":"2024-05-27T00:15:26.586675+0000","last_peered":"2024-05-27T00:15:26.586675+0000","last_clean":"2024-05-27T00:15:26.586675+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:15:26.586675+0000","last_undegraded":"2024-05-27T00:15:26.586675+0000","last_fullsized":"2024-05-27T00:15:26.586675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.590092+0000","last_change":"2024-05-27T00:15:25.632120+0000","last_active":"2024-05-27T00:15:26.590092+0000","last_peered":"2024-05-27T00:15:26.590092+0000","last_clean":"2024-05-27T00:15:26.590092+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:15:26.590092+0000","last_undegraded":"2024-05-27T00:15:26.590092+0000","last_fullsized":"2024-05-27T00:15:26.590092+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:06:27.232620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745618+0000","last_change":"2024-05-27T00:15:25.615868+0000","last_active":"2024-05-27T00:15:26.745618+0000","last_peered":"2024-05-27T00:15:26.745618+0000","last_clean":"2024-05-27T00:15:26.745618+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:15:26.745618+0000","last_undegraded":"2024-05-27T00:15:26.745618+0000","last_fullsized":"2024-05-27T00:15:26.745618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749314+0000","last_change":"2024-05-27T00:15:25.629101+0000","last_active":"2024-05-27T00:15:26.749314+0000","last_peered":"2024-05-27T00:15:26.749314+0000","last_clean":"2024-05-27T00:15:26.749314+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:15:26.749314+0000","last_undegraded":"2024-05-27T00:15:26.749314+0000","last_fullsized":"2024-05-27T00:15:26.749314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:26.893574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747537+0000","last_change":"2024-05-27T00:15:25.619351+0000","last_active":"2024-05-27T00:15:26.747537+0000","last_peered":"2024-05-27T00:15:26.747537+0000","last_clean":"2024-05-27T00:15:26.747537+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:15:26.747537+0000","last_undegraded":"2024-05-27T00:15:26.747537+0000","last_fullsized":"2024-05-27T00:15:26.747537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746054+0000","last_change":"2024-05-27T00:15:25.616436+0000","last_active":"2024-05-27T00:15:26.746054+0000","last_peered":"2024-05-27T00:15:26.746054+0000","last_clean":"2024-05-27T00:15:26.746054+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:15:26.746054+0000","last_undegraded":"2024-05-27T00:15:26.746054+0000","last_fullsized":"2024-05-27T00:15:26.746054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586830+0000","last_change":"2024-05-27T00:15:25.629373+0000","last_active":"2024-05-27T00:15:26.586830+0000","last_peered":"2024-05-27T00:15:26.586830+0000","last_clean":"2024-05-27T00:15:26.586830+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:15:26.586830+0000","last_undegraded":"2024-05-27T00:15:26.586830+0000","last_fullsized":"2024-05-27T00:15:26.586830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:02:15.012827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745308+0000","last_change":"2024-05-27T00:15:25.610707+0000","last_active":"2024-05-27T00:15:26.745308+0000","last_peered":"2024-05-27T00:15:26.745308+0000","last_clean":"2024-05-27T00:15:26.745308+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:15:26.745308+0000","last_undegraded":"2024-05-27T00:15:26.745308+0000","last_fullsized":"2024-05-27T00:15:26.745308+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:09:16.698319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587101+0000","last_change":"2024-05-27T00:15:25.613013+0000","last_active":"2024-05-27T00:15:26.587101+0000","last_peered":"2024-05-27T00:15:26.587101+0000","last_clean":"2024-05-27T00:15:26.587101+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:15:26.587101+0000","last_undegraded":"2024-05-27T00:15:26.587101+0000","last_fullsized":"2024-05-27T00:15:26.587101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745910+0000","last_change":"2024-05-27T00:15:25.610070+0000","last_active":"2024-05-27T00:15:26.745910+0000","last_peered":"2024-05-27T00:15:26.745910+0000","last_clean":"2024-05-27T00:15:26.745910+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:15:26.745910+0000","last_undegraded":"2024-05-27T00:15:26.745910+0000","last_fullsized":"2024-05-27T00:15:26.745910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:02:17.514654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747817+0000","last_change":"2024-05-27T00:15:25.633470+0000","last_active":"2024-05-27T00:15:26.747817+0000","last_peered":"2024-05-27T00:15:26.747817+0000","last_clean":"2024-05-27T00:15:26.747817+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:15:26.747817+0000","last_undegraded":"2024-05-27T00:15:26.747817+0000","last_fullsized":"2024-05-27T00:15:26.747817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583597+0000","last_change":"2024-05-27T00:15:25.597615+0000","last_active":"2024-05-27T00:15:26.583597+0000","last_peered":"2024-05-27T00:15:26.583597+0000","last_clean":"2024-05-27T00:15:26.583597+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:15:26.583597+0000","last_undegraded":"2024-05-27T00:15:26.583597+0000","last_fullsized":"2024-05-27T00:15:26.583597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746038+0000","last_change":"2024-05-27T00:15:25.610933+0000","last_active":"2024-05-27T00:15:26.746038+0000","last_peered":"2024-05-27T00:15:26.746038+0000","last_clean":"2024-05-27T00:15:26.746038+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:15:26.746038+0000","last_undegraded":"2024-05-27T00:15:26.746038+0000","last_fullsized":"2024-05-27T00:15:26.746038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:45:25.888432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746185+0000","last_change":"2024-05-27T00:15:25.598227+0000","last_active":"2024-05-27T00:15:26.746185+0000","last_peered":"2024-05-27T00:15:26.746185+0000","last_clean":"2024-05-27T00:15:26.746185+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:15:26.746185+0000","last_undegraded":"2024-05-27T00:15:26.746185+0000","last_fullsized":"2024-05-27T00:15:26.746185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588189+0000","last_change":"2024-05-27T00:15:25.620621+0000","last_active":"2024-05-27T00:15:26.588189+0000","last_peered":"2024-05-27T00:15:26.588189+0000","last_clean":"2024-05-27T00:15:26.588189+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:15:26.588189+0000","last_undegraded":"2024-05-27T00:15:26.588189+0000","last_fullsized":"2024-05-27T00:15:26.588189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:22:30.152810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744910+0000","last_change":"2024-05-27T00:15:25.605839+0000","last_active":"2024-05-27T00:15:26.744910+0000","last_peered":"2024-05-27T00:15:26.744910+0000","last_clean":"2024-05-27T00:15:26.744910+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:15:26.744910+0000","last_undegraded":"2024-05-27T00:15:26.744910+0000","last_fullsized":"2024-05-27T00:15:26.744910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:02.212115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746666+0000","last_change":"2024-05-27T00:15:25.607188+0000","last_active":"2024-05-27T00:15:26.746666+0000","last_peered":"2024-05-27T00:15:26.746666+0000","last_clean":"2024-05-27T00:15:26.746666+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:15:26.746666+0000","last_undegraded":"2024-05-27T00:15:26.746666+0000","last_fullsized":"2024-05-27T00:15:26.746666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587251+0000","last_change":"2024-05-27T00:15:25.615579+0000","last_active":"2024-05-27T00:15:26.587251+0000","last_peered":"2024-05-27T00:15:26.587251+0000","last_clean":"2024-05-27T00:15:26.587251+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:15:26.587251+0000","last_undegraded":"2024-05-27T00:15:26.587251+0000","last_fullsized":"2024-05-27T00:15:26.587251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:53.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746392+0000","last_change":"2024-05-27T00:15:24.611108+0000","last_active":"2024-05-27T00:15:26.746392+0000","last_peered":"2024-05-27T00:15:26.746392+0000","last_clean":"2024-05-27T00:15:26.746392+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:15:26.746392+0000","last_undegraded":"2024-05-27T00:15:26.746392+0000","last_fullsized":"2024-05-27T00:15:26.746392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:11.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746332+0000","last_change":"2024-05-27T00:15:25.614251+0000","last_active":"2024-05-27T00:15:26.746332+0000","last_peered":"2024-05-27T00:15:26.746332+0000","last_clean":"2024-05-27T00:15:26.746332+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:15:26.746332+0000","last_undegraded":"2024-05-27T00:15:26.746332+0000","last_fullsized":"2024-05-27T00:15:26.746332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:18:45.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747555+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:15:26.747555+0000","last_peered":"2024-05-27T00:15:26.747555+0000","last_clean":"2024-05-27T00:15:26.747555+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:15:26.747555+0000","last_undegraded":"2024-05-27T00:15:26.747555+0000","last_fullsized":"2024-05-27T00:15:26.747555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745607+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:15:26.745607+0000","last_peered":"2024-05-27T00:15:26.745607+0000","last_clean":"2024-05-27T00:15:26.745607+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:15:26.745607+0000","last_undegraded":"2024-05-27T00:15:26.745607+0000","last_fullsized":"2024-05-27T00:15:26.745607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:46:08.645927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745132+0000","last_change":"2024-05-27T00:15:25.596844+0000","last_active":"2024-05-27T00:15:26.745132+0000","last_peered":"2024-05-27T00:15:26.745132+0000","last_clean":"2024-05-27T00:15:26.745132+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:15:26.745132+0000","last_undegraded":"2024-05-27T00:15:26.745132+0000","last_fullsized":"2024-05-27T00:15:26.745132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:17:40.503615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587344+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:15:26.587344+0000","last_peered":"2024-05-27T00:15:26.587344+0000","last_clean":"2024-05-27T00:15:26.587344+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:15:26.587344+0000","last_undegraded":"2024-05-27T00:15:26.587344+0000","last_fullsized":"2024-05-27T00:15:26.587344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583235+0000","last_change":"2024-05-27T00:15:25.593149+0000","last_active":"2024-05-27T00:15:26.583235+0000","last_peered":"2024-05-27T00:15:26.583235+0000","last_clean":"2024-05-27T00:15:26.583235+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:15:26.583235+0000","last_undegraded":"2024-05-27T00:15:26.583235+0000","last_fullsized":"2024-05-27T00:15:26.583235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:50.793889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746974+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:15:26.746974+0000","last_peered":"2024-05-27T00:15:26.746974+0000","last_clean":"2024-05-27T00:15:26.746974+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:15:26.746974+0000","last_undegraded":"2024-05-27T00:15:26.746974+0000","last_fullsized":"2024-05-27T00:15:26.746974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589140+0000","last_change":"2024-05-27T00:15:25.618650+0000","last_active":"2024-05-27T00:15:26.589140+0000","last_peered":"2024-05-27T00:15:26.589140+0000","last_clean":"2024-05-27T00:15:26.589140+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:15:26.589140+0000","last_undegraded":"2024-05-27T00:15:26.589140+0000","last_fullsized":"2024-05-27T00:15:26.589140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:21.935526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.589356+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:15:26.589356+0000","last_peered":"2024-05-27T00:15:26.589356+0000","last_clean":"2024-05-27T00:15:26.589356+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:15:26.589356+0000","last_undegraded":"2024-05-27T00:15:26.589356+0000","last_fullsized":"2024-05-27T00:15:26.589356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746448+0000","last_change":"2024-05-27T00:15:25.613142+0000","last_active":"2024-05-27T00:15:26.746448+0000","last_peered":"2024-05-27T00:15:26.746448+0000","last_clean":"2024-05-27T00:15:26.746448+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:15:26.746448+0000","last_undegraded":"2024-05-27T00:15:26.746448+0000","last_fullsized":"2024-05-27T00:15:26.746448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:25.284413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.620068+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:15:30.620068+0000","last_peered":"2024-05-27T00:15:30.620068+0000","last_clean":"2024-05-27T00:15:30.620068+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:15:30.620068+0000","last_undegraded":"2024-05-27T00:15:30.620068+0000","last_fullsized":"2024-05-27T00:15:30.620068+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588245+0000","last_change":"2024-05-27T00:15:25.622383+0000","last_active":"2024-05-27T00:15:26.588245+0000","last_peered":"2024-05-27T00:15:26.588245+0000","last_clean":"2024-05-27T00:15:26.588245+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:15:26.588245+0000","last_undegraded":"2024-05-27T00:15:26.588245+0000","last_fullsized":"2024-05-27T00:15:26.588245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:36.735310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.627016+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:15:30.627016+0000","last_peered":"2024-05-27T00:15:30.627016+0000","last_clean":"2024-05-27T00:15:30.627016+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:15:30.627016+0000","last_undegraded":"2024-05-27T00:15:30.627016+0000","last_fullsized":"2024-05-27T00:15:30.627016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1866,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748511+0000","last_change":"2024-05-27T00:15:25.628646+0000","last_active":"2024-05-27T00:15:26.748511+0000","last_peered":"2024-05-27T00:15:26.748511+0000","last_clean":"2024-05-27T00:15:26.748511+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:15:26.748511+0000","last_undegraded":"2024-05-27T00:15:26.748511+0000","last_fullsized":"2024-05-27T00:15:26.748511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:22:30.152810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619790+0000","last_change":"2024-05-27T00:15:24.605986+0000","last_active":"2024-05-27T00:15:30.619790+0000","last_peered":"2024-05-27T00:15:30.619790+0000","last_clean":"2024-05-27T00:15:30.619790+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:15:30.619790+0000","last_undegraded":"2024-05-27T00:15:30.619790+0000","last_fullsized":"2024-05-27T00:15:30.619790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745414+0000","last_change":"2024-05-27T00:15:25.624136+0000","last_active":"2024-05-27T00:15:26.745414+0000","last_peered":"2024-05-27T00:15:26.745414+0000","last_clean":"2024-05-27T00:15:26.745414+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:15:26.745414+0000","last_undegraded":"2024-05-27T00:15:26.745414+0000","last_fullsized":"2024-05-27T00:15:26.745414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:56:53.529673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588435+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:15:26.588435+0000","last_peered":"2024-05-27T00:15:26.588435+0000","last_clean":"2024-05-27T00:15:26.588435+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:15:26.588435+0000","last_undegraded":"2024-05-27T00:15:26.588435+0000","last_fullsized":"2024-05-27T00:15:26.588435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748720+0000","last_change":"2024-05-27T00:15:25.641965+0000","last_active":"2024-05-27T00:15:26.748720+0000","last_peered":"2024-05-27T00:15:26.748720+0000","last_clean":"2024-05-27T00:15:26.748720+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:15:26.748720+0000","last_undegraded":"2024-05-27T00:15:26.748720+0000","last_fullsized":"2024-05-27T00:15:26.748720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:17.513529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746779+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:15:26.746779+0000","last_peered":"2024-05-27T00:15:26.746779+0000","last_clean":"2024-05-27T00:15:26.746779+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:15:26.746779+0000","last_undegraded":"2024-05-27T00:15:26.746779+0000","last_fullsized":"2024-05-27T00:15:26.746779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746732+0000","last_change":"2024-05-27T00:15:25.608549+0000","last_active":"2024-05-27T00:15:26.746732+0000","last_peered":"2024-05-27T00:15:26.746732+0000","last_clean":"2024-05-27T00:15:26.746732+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:15:26.746732+0000","last_undegraded":"2024-05-27T00:15:26.746732+0000","last_fullsized":"2024-05-27T00:15:26.746732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:38:08.119796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619615+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:15:30.619615+0000","last_peered":"2024-05-27T00:15:30.619615+0000","last_clean":"2024-05-27T00:15:30.619615+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:15:30.619615+0000","last_undegraded":"2024-05-27T00:15:30.619615+0000","last_fullsized":"2024-05-27T00:15:30.619615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747206+0000","last_change":"2024-05-27T00:15:25.638558+0000","last_active":"2024-05-27T00:15:26.747206+0000","last_peered":"2024-05-27T00:15:26.747206+0000","last_clean":"2024-05-27T00:15:26.747206+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:15:26.747206+0000","last_undegraded":"2024-05-27T00:15:26.747206+0000","last_fullsized":"2024-05-27T00:15:26.747206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:02:15.012827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587664+0000","last_change":"2024-05-27T00:15:24.601924+0000","last_active":"2024-05-27T00:15:26.587664+0000","last_peered":"2024-05-27T00:15:26.587664+0000","last_clean":"2024-05-27T00:15:26.587664+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:15:26.587664+0000","last_undegraded":"2024-05-27T00:15:26.587664+0000","last_fullsized":"2024-05-27T00:15:26.587664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:32.185839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749041+0000","last_change":"2024-05-27T00:15:25.639140+0000","last_active":"2024-05-27T00:15:26.749041+0000","last_peered":"2024-05-27T00:15:26.749041+0000","last_clean":"2024-05-27T00:15:26.749041+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:15:26.749041+0000","last_undegraded":"2024-05-27T00:15:26.749041+0000","last_fullsized":"2024-05-27T00:15:26.749041+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:21.935526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619020+0000","last_change":"2024-05-27T00:15:24.600878+0000","last_active":"2024-05-27T00:15:30.619020+0000","last_peered":"2024-05-27T00:15:30.619020+0000","last_clean":"2024-05-27T00:15:30.619020+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:15:30.619020+0000","last_undegraded":"2024-05-27T00:15:30.619020+0000","last_fullsized":"2024-05-27T00:15:30.619020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749435+0000","last_change":"2024-05-27T00:15:25.644249+0000","last_active":"2024-05-27T00:15:26.749435+0000","last_peered":"2024-05-27T00:15:26.749435+0000","last_clean":"2024-05-27T00:15:26.749435+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:15:26.749435+0000","last_undegraded":"2024-05-27T00:15:26.749435+0000","last_fullsized":"2024-05-27T00:15:26.749435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:59:25.042662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583715+0000","last_change":"2024-05-27T00:15:24.603452+0000","last_active":"2024-05-27T00:15:26.583715+0000","last_peered":"2024-05-27T00:15:26.583715+0000","last_clean":"2024-05-27T00:15:26.583715+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:15:26.583715+0000","last_undegraded":"2024-05-27T00:15:26.583715+0000","last_fullsized":"2024-05-27T00:15:26.583715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587865+0000","last_change":"2024-05-27T00:15:25.619341+0000","last_active":"2024-05-27T00:15:26.587865+0000","last_peered":"2024-05-27T00:15:26.587865+0000","last_clean":"2024-05-27T00:15:26.587865+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:15:26.587865+0000","last_undegraded":"2024-05-27T00:15:26.587865+0000","last_fullsized":"2024-05-27T00:15:26.587865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:41:46.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745117+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:15:26.745117+0000","last_peered":"2024-05-27T00:15:26.745117+0000","last_clean":"2024-05-27T00:15:26.745117+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:15:26.745117+0000","last_undegraded":"2024-05-27T00:15:26.745117+0000","last_fullsized":"2024-05-27T00:15:26.745117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.748893+0000","last_change":"2024-05-27T00:15:25.639462+0000","last_active":"2024-05-27T00:15:26.748893+0000","last_peered":"2024-05-27T00:15:26.748893+0000","last_clean":"2024-05-27T00:15:26.748893+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:15:26.748893+0000","last_undegraded":"2024-05-27T00:15:26.748893+0000","last_fullsized":"2024-05-27T00:15:26.748893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:52:55.175945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746823+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:15:26.746823+0000","last_peered":"2024-05-27T00:15:26.746823+0000","last_clean":"2024-05-27T00:15:26.746823+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:15:26.746823+0000","last_undegraded":"2024-05-27T00:15:26.746823+0000","last_fullsized":"2024-05-27T00:15:26.746823+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.586683+0000","last_change":"2024-05-27T00:15:24.603781+0000","last_active":"2024-05-27T00:15:26.586683+0000","last_peered":"2024-05-27T00:15:26.586683+0000","last_clean":"2024-05-27T00:15:26.586683+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:15:26.586683+0000","last_undegraded":"2024-05-27T00:15:26.586683+0000","last_fullsized":"2024-05-27T00:15:26.586683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746108+0000","last_change":"2024-05-27T00:15:25.631978+0000","last_active":"2024-05-27T00:15:26.746108+0000","last_peered":"2024-05-27T00:15:26.746108+0000","last_clean":"2024-05-27T00:15:26.746108+0000","last_became_active":"2024-05-27T00:15:25.631697+0000","last_became_peered":"2024-05-27T00:15:25.631697+0000","last_unstale":"2024-05-27T00:15:26.746108+0000","last_undegraded":"2024-05-27T00:15:26.746108+0000","last_fullsized":"2024-05-27T00:15:26.746108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:05.141408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746536+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:15:26.746536+0000","last_peered":"2024-05-27T00:15:26.746536+0000","last_clean":"2024-05-27T00:15:26.746536+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:15:26.746536+0000","last_undegraded":"2024-05-27T00:15:26.746536+0000","last_fullsized":"2024-05-27T00:15:26.746536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746253+0000","last_change":"2024-05-27T00:15:25.610627+0000","last_active":"2024-05-27T00:15:26.746253+0000","last_peered":"2024-05-27T00:15:26.746253+0000","last_clean":"2024-05-27T00:15:26.746253+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:15:26.746253+0000","last_undegraded":"2024-05-27T00:15:26.746253+0000","last_fullsized":"2024-05-27T00:15:26.746253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:41:12.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.744898+0000","last_change":"2024-05-27T00:15:24.598182+0000","last_active":"2024-05-27T00:15:26.744898+0000","last_peered":"2024-05-27T00:15:26.744898+0000","last_clean":"2024-05-27T00:15:26.744898+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:15:26.744898+0000","last_undegraded":"2024-05-27T00:15:26.744898+0000","last_fullsized":"2024-05-27T00:15:26.744898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583243+0000","last_change":"2024-05-27T00:15:25.591449+0000","last_active":"2024-05-27T00:15:26.583243+0000","last_peered":"2024-05-27T00:15:26.583243+0000","last_clean":"2024-05-27T00:15:26.583243+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:15:26.583243+0000","last_undegraded":"2024-05-27T00:15:26.583243+0000","last_fullsized":"2024-05-27T00:15:26.583243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:47:25.824538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588693+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:15:26.588693+0000","last_peered":"2024-05-27T00:15:26.588693+0000","last_clean":"2024-05-27T00:15:26.588693+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:15:26.588693+0000","last_undegraded":"2024-05-27T00:15:26.588693+0000","last_fullsized":"2024-05-27T00:15:26.588693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587675+0000","last_change":"2024-05-27T00:15:25.614073+0000","last_active":"2024-05-27T00:15:26.587675+0000","last_peered":"2024-05-27T00:15:26.587675+0000","last_clean":"2024-05-27T00:15:26.587675+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:15:26.587675+0000","last_undegraded":"2024-05-27T00:15:26.587675+0000","last_fullsized":"2024-05-27T00:15:26.587675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:55.937424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583569+0000","last_change":"2024-05-27T00:15:24.603518+0000","last_active":"2024-05-27T00:15:26.583569+0000","last_peered":"2024-05-27T00:15:26.583569+0000","last_clean":"2024-05-27T00:15:26.583569+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:15:26.583569+0000","last_undegraded":"2024-05-27T00:15:26.583569+0000","last_fullsized":"2024-05-27T00:15:26.583569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588810+0000","last_change":"2024-05-27T00:15:25.621519+0000","last_active":"2024-05-27T00:15:26.588810+0000","last_peered":"2024-05-27T00:15:26.588810+0000","last_clean":"2024-05-27T00:15:26.588810+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:15:26.588810+0000","last_undegraded":"2024-05-27T00:15:26.588810+0000","last_fullsized":"2024-05-27T00:15:26.588810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:52:55.175945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.749299+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:15:26.749299+0000","last_peered":"2024-05-27T00:15:26.749299+0000","last_clean":"2024-05-27T00:15:26.749299+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:15:26.749299+0000","last_undegraded":"2024-05-27T00:15:26.749299+0000","last_fullsized":"2024-05-27T00:15:26.749299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583460+0000","last_change":"2024-05-27T00:15:25.593180+0000","last_active":"2024-05-27T00:15:26.583460+0000","last_peered":"2024-05-27T00:15:26.583460+0000","last_clean":"2024-05-27T00:15:26.583460+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:15:26.583460+0000","last_undegraded":"2024-05-27T00:15:26.583460+0000","last_fullsized":"2024-05-27T00:15:26.583460+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583701+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:15:26.583701+0000","last_peered":"2024-05-27T00:15:26.583701+0000","last_clean":"2024-05-27T00:15:26.583701+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:15:26.583701+0000","last_undegraded":"2024-05-27T00:15:26.583701+0000","last_fullsized":"2024-05-27T00:15:26.583701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745915+0000","last_change":"2024-05-27T00:15:25.631919+0000","last_active":"2024-05-27T00:15:26.745915+0000","last_peered":"2024-05-27T00:15:26.745915+0000","last_clean":"2024-05-27T00:15:26.745915+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:15:26.745915+0000","last_undegraded":"2024-05-27T00:15:26.745915+0000","last_fullsized":"2024-05-27T00:15:26.745915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:24:31.995603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.620892+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:15:30.620892+0000","last_peered":"2024-05-27T00:15:30.620892+0000","last_clean":"2024-05-27T00:15:30.620892+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:15:30.620892+0000","last_undegraded":"2024-05-27T00:15:30.620892+0000","last_fullsized":"2024-05-27T00:15:30.620892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745629+0000","last_change":"2024-05-27T00:15:25.596792+0000","last_active":"2024-05-27T00:15:26.745629+0000","last_peered":"2024-05-27T00:15:26.745629+0000","last_clean":"2024-05-27T00:15:26.745629+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:15:26.745629+0000","last_undegraded":"2024-05-27T00:15:26.745629+0000","last_fullsized":"2024-05-27T00:15:26.745629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747720+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:15:26.747720+0000","last_peered":"2024-05-27T00:15:26.747720+0000","last_clean":"2024-05-27T00:15:26.747720+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:15:26.747720+0000","last_undegraded":"2024-05-27T00:15:26.747720+0000","last_fullsized":"2024-05-27T00:15:26.747720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587779+0000","last_change":"2024-05-27T00:15:25.610697+0000","last_active":"2024-05-27T00:15:26.587779+0000","last_peered":"2024-05-27T00:15:26.587779+0000","last_clean":"2024-05-27T00:15:26.587779+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:15:26.587779+0000","last_undegraded":"2024-05-27T00:15:26.587779+0000","last_fullsized":"2024-05-27T00:15:26.587779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:14.969827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583375+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:15:26.583375+0000","last_peered":"2024-05-27T00:15:26.583375+0000","last_clean":"2024-05-27T00:15:26.583375+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:15:26.583375+0000","last_undegraded":"2024-05-27T00:15:26.583375+0000","last_fullsized":"2024-05-27T00:15:26.583375+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745282+0000","last_change":"2024-05-27T00:15:25.597164+0000","last_active":"2024-05-27T00:15:26.745282+0000","last_peered":"2024-05-27T00:15:26.745282+0000","last_clean":"2024-05-27T00:15:26.745282+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:15:26.745282+0000","last_undegraded":"2024-05-27T00:15:26.745282+0000","last_fullsized":"2024-05-27T00:15:26.745282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:04.655872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.583422+0000","last_change":"2024-05-27T00:15:24.598238+0000","last_active":"2024-05-27T00:15:26.583422+0000","last_peered":"2024-05-27T00:15:26.583422+0000","last_clean":"2024-05-27T00:15:26.583422+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:15:26.583422+0000","last_undegraded":"2024-05-27T00:15:26.583422+0000","last_fullsized":"2024-05-27T00:15:26.583422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746235+0000","last_change":"2024-05-27T00:15:25.601140+0000","last_active":"2024-05-27T00:15:26.746235+0000","last_peered":"2024-05-27T00:15:26.746235+0000","last_clean":"2024-05-27T00:15:26.746235+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:15:26.746235+0000","last_undegraded":"2024-05-27T00:15:26.746235+0000","last_fullsized":"2024-05-27T00:15:26.746235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:42:05.334730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.621275+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:15:30.621275+0000","last_peered":"2024-05-27T00:15:30.621275+0000","last_clean":"2024-05-27T00:15:30.621275+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:15:30.621275+0000","last_undegraded":"2024-05-27T00:15:30.621275+0000","last_fullsized":"2024-05-27T00:15:30.621275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.747292+0000","last_change":"2024-05-27T00:15:25.619694+0000","last_active":"2024-05-27T00:15:26.747292+0000","last_peered":"2024-05-27T00:15:26.747292+0000","last_clean":"2024-05-27T00:15:26.747292+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:15:26.747292+0000","last_undegraded":"2024-05-27T00:15:26.747292+0000","last_fullsized":"2024-05-27T00:15:26.747292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:40:09.867791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.588751+0000","last_change":"2024-05-27T00:15:25.627578+0000","last_active":"2024-05-27T00:15:26.588751+0000","last_peered":"2024-05-27T00:15:26.588751+0000","last_clean":"2024-05-27T00:15:26.588751+0000","last_became_active":"2024-05-27T00:15:25.622421+0000","last_became_peered":"2024-05-27T00:15:25.622421+0000","last_unstale":"2024-05-27T00:15:26.588751+0000","last_undegraded":"2024-05-27T00:15:26.588751+0000","last_fullsized":"2024-05-27T00:15:26.588751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:17.513529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.619639+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:15:30.619639+0000","last_peered":"2024-05-27T00:15:30.619639+0000","last_clean":"2024-05-27T00:15:30.619639+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:15:30.619639+0000","last_undegraded":"2024-05-27T00:15:30.619639+0000","last_fullsized":"2024-05-27T00:15:30.619639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.587486+0000","last_change":"2024-05-27T00:15:25.623207+0000","last_active":"2024-05-27T00:15:26.587486+0000","last_peered":"2024-05-27T00:15:26.587486+0000","last_clean":"2024-05-27T00:15:26.587486+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:15:26.587486+0000","last_undegraded":"2024-05-27T00:15:26.587486+0000","last_fullsized":"2024-05-27T00:15:26.587486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:59.804368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.618376+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:15:30.618376+0000","last_peered":"2024-05-27T00:15:30.618376+0000","last_clean":"2024-05-27T00:15:30.618376+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:15:30.618376+0000","last_undegraded":"2024-05-27T00:15:30.618376+0000","last_fullsized":"2024-05-27T00:15:30.618376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.746305+0000","last_change":"2024-05-27T00:15:25.613018+0000","last_active":"2024-05-27T00:15:26.746305+0000","last_peered":"2024-05-27T00:15:26.746305+0000","last_clean":"2024-05-27T00:15:26.746305+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:15:26.746305+0000","last_undegraded":"2024-05-27T00:15:26.746305+0000","last_fullsized":"2024-05-27T00:15:26.746305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:53:10.551955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745698+0000","last_change":"2024-05-27T00:15:24.600571+0000","last_active":"2024-05-27T00:15:26.745698+0000","last_peered":"2024-05-27T00:15:26.745698+0000","last_clean":"2024-05-27T00:15:26.745698+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:15:26.745698+0000","last_undegraded":"2024-05-27T00:15:26.745698+0000","last_fullsized":"2024-05-27T00:15:26.745698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:44:21.645223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:26.745724+0000","last_change":"2024-05-27T00:15:25.615649+0000","last_active":"2024-05-27T00:15:26.745724+0000","last_peered":"2024-05-27T00:15:26.745724+0000","last_clean":"2024-05-27T00:15:26.745724+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:15:26.745724+0000","last_undegraded":"2024-05-27T00:15:26.745724+0000","last_fullsized":"2024-05-27T00:15:26.745724+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:16:16.309939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-05-27T00:15:30.621025+0000","last_change":"2024-05-27T00:15:24.615066+0000","last_active":"2024-05-27T00:15:30.621025+0000","last_peered":"2024-05-27T00:15:30.621025+0000","last_clean":"2024-05-27T00:15:30.621025+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:15:30.621025+0000","last_undegraded":"2024-05-27T00:15:30.621025+0000","last_fullsized":"2024-05-27T00:15:30.621025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2664,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":823296,"data_stored":669858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542155,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":80774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542156,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":78998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":79034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":80774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":78908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542155,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":79068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":669914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:15:47.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:48.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2957600640 1 Processor -- start 2024-05-27T00:15:48.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2957600640 1 -- start start 2024-05-27T00:15:48.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2957600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29500cbe10 0x7f29501592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2957600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2950073570 0x7f29501559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2957600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29500c1f60 con 0x7f29500cc1f0 2024-05-27T00:15:48.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2957600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2950159810 con 0x7f29500cbe10 2024-05-27T00:15:48.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2957600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f29500c6dd0 con 0x7f2950073570 2024-05-27T00:15:48.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2954c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29500cbe10 0x7f29501592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2954c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29500cbe10 0x7f29501592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54032/0 (socket says 172.21.15.98:54032) 2024-05-27T00:15:48.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2954c00640 1 -- 172.21.15.98:0/3407649129 learned_addr learned my addr 172.21.15.98:0/3407649129 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:48.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2955600640 1 --2- 172.21.15.98:0/3407649129 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2950073570 0x7f29501559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2954c00640 1 -- 172.21.15.98:0/3407649129 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2950073570 msgr2=0x7f29501559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:48.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2954c00640 1 --2- 172.21.15.98:0/3407649129 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2950073570 0x7f29501559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2954c00640 1 -- 172.21.15.98:0/3407649129 >> v1:172.21.15.204:6789/0 conn(0x7f29500cc1f0 legacy=0x7f29500c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.113+0000 7f2954c00640 1 -- 172.21.15.98:0/3407649129 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29500c70b0 con 0x7f29500cbe10 2024-05-27T00:15:48.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f2954c00640 1 --2- 172.21.15.98:0/3407649129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cbe10 0x7f29501592d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2940009870 tx=0x7f294001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c113d22974491163 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:48.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f294fe00640 1 -- 172.21.15.98:0/3407649129 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29400b5070 con 0x7f29500cbe10 2024-05-27T00:15:48.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f294fe00640 1 -- 172.21.15.98:0/3407649129 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29400a8a50 con 0x7f29500cbe10 2024-05-27T00:15:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f294fe00640 1 -- 172.21.15.98:0/3407649129 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29400a8e20 con 0x7f29500cbe10 2024-05-27T00:15:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f2957600640 1 -- 172.21.15.98:0/3407649129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cbe10 msgr2=0x7f29501592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f2957600640 1 --2- 172.21.15.98:0/3407649129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cbe10 0x7f29501592d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2940009870 tx=0x7f294001ed50 comp rx=0 tx=0).stop 2024-05-27T00:15:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f2957600640 1 -- 172.21.15.98:0/3407649129 shutdown_connections 2024-05-27T00:15:48.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f2957600640 1 --2- 172.21.15.98:0/3407649129 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2950073570 0x7f29501559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f2957600640 1 --2- 172.21.15.98:0/3407649129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cbe10 0x7f29501592d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f2957600640 1 -- 172.21.15.98:0/3407649129 >> 172.21.15.98:0/3407649129 conn(0x7f2950067530 msgr2=0x7f295006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:48.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f2957600640 1 -- 172.21.15.98:0/3407649129 shutdown_connections 2024-05-27T00:15:48.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.117+0000 7f2957600640 1 -- 172.21.15.98:0/3407649129 wait complete. 2024-05-27T00:15:48.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2957600640 1 Processor -- start 2024-05-27T00:15:48.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2957600640 1 -- start start 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2957600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2950073570 0x7f29501e4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2957600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29500cbe10 0x7f29501566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2957600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cc1f0 0x7f29501ee950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2957600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f29500c1330 con 0x7f29500cc1f0 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2957600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f29500c1050 con 0x7f2950073570 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2957600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f29500c79d0 con 0x7f29500cbe10 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2955600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cc1f0 0x7f29501ee950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2955600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cc1f0 0x7f29501ee950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54036/0 (socket says 172.21.15.98:54036) 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2955600640 1 -- 172.21.15.98:0/1003293268 learned_addr learned my addr 172.21.15.98:0/1003293268 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2954c00640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2950073570 0x7f29501e4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2947e00640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29500cbe10 0x7f29501566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2955600640 1 -- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29500cbe10 msgr2=0x7f29501566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2955600640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29500cbe10 0x7f29501566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2955600640 1 -- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2950073570 msgr2=0x7f29501e4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2955600640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2950073570 0x7f29501e4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2955600640 1 -- 172.21.15.98:0/1003293268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29501e2bf0 con 0x7f29500cc1f0 2024-05-27T00:15:48.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2955600640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cc1f0 0x7f29501ee950 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2948007e40 tx=0x7f294800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f294d600640 1 -- 172.21.15.98:0/1003293268 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948014620 con 0x7f29500cc1f0 2024-05-27T00:15:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f294d600640 1 -- 172.21.15.98:0/1003293268 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2948019070 con 0x7f29500cc1f0 2024-05-27T00:15:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2957600640 1 -- 172.21.15.98:0/1003293268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29501585e0 con 0x7f29500cc1f0 2024-05-27T00:15:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f294d600640 1 -- 172.21.15.98:0/1003293268 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948014a20 con 0x7f29500cc1f0 2024-05-27T00:15:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.121+0000 7f2957600640 1 -- 172.21.15.98:0/1003293268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29501588a0 con 0x7f29500cc1f0 2024-05-27T00:15:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.125+0000 7f2957600640 1 -- 172.21.15.98:0/1003293268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2950075cb0 con 0x7f29500cc1f0 2024-05-27T00:15:48.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.125+0000 7f294d600640 1 -- 172.21.15.98:0/1003293268 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2948014bc0 con 0x7f29500cc1f0 2024-05-27T00:15:48.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.125+0000 7f294d600640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f292c0678c0 0x7f292c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.125+0000 7f294d600640 1 -- 172.21.15.98:0/1003293268 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2948086470 con 0x7f29500cc1f0 2024-05-27T00:15:48.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.125+0000 7f2954c00640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f292c0678c0 0x7f292c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.125+0000 7f2954c00640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f292c0678c0 0x7f292c069d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f29400b2000 tx=0x7f2940009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:48.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.129+0000 7f294d600640 1 -- 172.21.15.98:0/1003293268 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f29480568d0 con 0x7f29500cc1f0 2024-05-27T00:15:48.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.285+0000 7f2957600640 1 -- 172.21.15.98:0/1003293268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29501523b0 con 0x7f29500cc1f0 2024-05-27T00:15:48.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.285+0000 7f294d600640 1 -- 172.21.15.98:0/1003293268 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16581 (secure 0 0 0) 0x7f294805abd0 con 0x7f29500cc1f0 2024-05-27T00:15:48.290 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:48.290 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:26.579229+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 -- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f292c0678c0 msgr2=0x7f292c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f292c0678c0 0x7f292c069d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f29400b2000 tx=0x7f2940009040 comp rx=0 tx=0).stop 2024-05-27T00:15:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 -- 172.21.15.98:0/1003293268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cc1f0 msgr2=0x7f29501ee950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cc1f0 0x7f29501ee950 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2948007e40 tx=0x7f294800e4b0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 -- 172.21.15.98:0/1003293268 shutdown_connections 2024-05-27T00:15:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29500cc1f0 0x7f29501ee950 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29500cbe10 0x7f29501566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f292c0678c0 0x7f292c069d80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 --2- 172.21.15.98:0/1003293268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2950073570 0x7f29501e4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 -- 172.21.15.98:0/1003293268 >> 172.21.15.98:0/1003293268 conn(0x7f2950067530 msgr2=0x7f29500c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 -- 172.21.15.98:0/1003293268 shutdown_connections 2024-05-27T00:15:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.289+0000 7f2957600640 1 -- 172.21.15.98:0/1003293268 wait complete. 2024-05-27T00:15:48.311 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T00:15:13.211703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T00:15:23.224464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T00:15:23.950896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T00:15:48.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T00:15:48.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bd7200640 1 Processor -- start 2024-05-27T00:15:48.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bd7200640 1 -- start start 2024-05-27T00:15:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bd7200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1bd00b1e10 0x7f1bd006e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bd7200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1bd00c2b10 0x7f1bd006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bd7200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bd00c8420 con 0x7f1bd01516b0 2024-05-27T00:15:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bd7200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1bd00c1df0 con 0x7f1bd00b1e10 2024-05-27T00:15:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bd7200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f1bd00c3000 con 0x7f1bd00c2b10 2024-05-27T00:15:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bc7400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1bd00b1e10 0x7f1bd006e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bcfe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1bd01516b0 0x7f1bd00c1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45892/0 (socket says 172.21.15.98:45892) 2024-05-27T00:15:48.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bcfe00640 1 -- 172.21.15.98:0/2532829886 learned_addr learned my addr 172.21.15.98:0/2532829886 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bd5200640 1 --2- 172.21.15.98:0/2532829886 >> v2:172.21.15.204:3300/0 conn(0x7f1bd00c2b10 0x7f1bd006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bc7400640 1 -- 172.21.15.98:0/2532829886 >> v2:172.21.15.204:3300/0 conn(0x7f1bd00c2b10 msgr2=0x7f1bd006ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bc7400640 1 --2- 172.21.15.98:0/2532829886 >> v2:172.21.15.204:3300/0 conn(0x7f1bd00c2b10 0x7f1bd006ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bc7400640 1 -- 172.21.15.98:0/2532829886 >> v1:172.21.15.98:6789/0 conn(0x7f1bd01516b0 legacy=0x7f1bd00c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bc7400640 1 -- 172.21.15.98:0/2532829886 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd00c7e60 con 0x7f1bd00b1e10 2024-05-27T00:15:48.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.449+0000 7f1bcf400640 1 -- 172.21.15.98:0/2532829886 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3243412988 0 0) 0x7f1bd00c8420 con 0x7f1bd01516b0 2024-05-27T00:15:48.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bc7400640 1 --2- 172.21.15.98:0/2532829886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00b1e10 0x7f1bd006e780 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0158590 tx=0x7f1bb801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3c7bea06e54157 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:48.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bcf400640 1 -- 172.21.15.98:0/2532829886 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb80b0070 con 0x7f1bd00b1e10 2024-05-27T00:15:48.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bcf400640 1 -- 172.21.15.98:0/2532829886 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bb80ab600 con 0x7f1bd00b1e10 2024-05-27T00:15:48.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bcf400640 1 -- 172.21.15.98:0/2532829886 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb80ab9d0 con 0x7f1bd00b1e10 2024-05-27T00:15:48.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bd7200640 1 -- 172.21.15.98:0/2532829886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00b1e10 msgr2=0x7f1bd006e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bd7200640 1 --2- 172.21.15.98:0/2532829886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00b1e10 0x7f1bd006e780 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0158590 tx=0x7f1bb801c480 comp rx=0 tx=0).stop 2024-05-27T00:15:48.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bd7200640 1 -- 172.21.15.98:0/2532829886 shutdown_connections 2024-05-27T00:15:48.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bd7200640 1 --2- 172.21.15.98:0/2532829886 >> v2:172.21.15.204:3300/0 conn(0x7f1bd00c2b10 0x7f1bd006ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bd7200640 1 --2- 172.21.15.98:0/2532829886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00b1e10 0x7f1bd006e780 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bd7200640 1 -- 172.21.15.98:0/2532829886 >> 172.21.15.98:0/2532829886 conn(0x7f1bd00cbf00 msgr2=0x7f1bd00cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:48.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bd7200640 1 -- 172.21.15.98:0/2532829886 shutdown_connections 2024-05-27T00:15:48.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.453+0000 7f1bd7200640 1 -- 172.21.15.98:0/2532829886 wait complete. 2024-05-27T00:15:48.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd7200640 1 Processor -- start 2024-05-27T00:15:48.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd7200640 1 -- start start 2024-05-27T00:15:48.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bd00b1e10 0x7f1bd0148f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd7200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00c2b10 0x7f1bd0149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd01516b0 0x7f1bd01409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd7200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd0151c10 con 0x7f1bd00c2b10 2024-05-27T00:15:48.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd0151a90 con 0x7f1bd01516b0 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd0164540 con 0x7f1bd00b1e10 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bc7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00c2b10 0x7f1bd0149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bc7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00c2b10 0x7f1bd0149490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54056/0 (socket says 172.21.15.98:54056) 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bc7400640 1 -- 172.21.15.98:0/3360873406 learned_addr learned my addr 172.21.15.98:0/3360873406 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bcfe00640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bd00b1e10 0x7f1bd0148f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd5200640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd01516b0 0x7f1bd01409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bc7400640 1 -- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bd00b1e10 msgr2=0x7f1bd0148f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bc7400640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bd00b1e10 0x7f1bd0148f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bc7400640 1 -- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd01516b0 msgr2=0x7f1bd01409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bc7400640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd01516b0 0x7f1bd01409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bc7400640 1 -- 172.21.15.98:0/3360873406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd0153db0 con 0x7f1bd00c2b10 2024-05-27T00:15:48.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bc7400640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00c2b10 0x7f1bd0149490 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80ae040 tx=0x7f1bb801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:48.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bccc00640 1 -- 172.21.15.98:0/3360873406 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb80bc070 con 0x7f1bd00c2b10 2024-05-27T00:15:48.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bccc00640 1 -- 172.21.15.98:0/3360873406 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bb80b1040 con 0x7f1bd00c2b10 2024-05-27T00:15:48.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd7200640 1 -- 172.21.15.98:0/3360873406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd01473d0 con 0x7f1bd00c2b10 2024-05-27T00:15:48.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bccc00640 1 -- 172.21.15.98:0/3360873406 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb80b0050 con 0x7f1bd00c2b10 2024-05-27T00:15:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.457+0000 7f1bd7200640 1 -- 172.21.15.98:0/3360873406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd0147990 con 0x7f1bd00c2b10 2024-05-27T00:15:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.461+0000 7f1bd7200640 1 -- 172.21.15.98:0/3360873406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd0075cb0 con 0x7f1bd00c2b10 2024-05-27T00:15:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.461+0000 7f1bccc00640 1 -- 172.21.15.98:0/3360873406 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1bb80bf070 con 0x7f1bd00c2b10 2024-05-27T00:15:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.461+0000 7f1bccc00640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1bac0678c0 0x7f1bac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.461+0000 7f1bccc00640 1 -- 172.21.15.98:0/3360873406 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f1bb811f920 con 0x7f1bd00c2b10 2024-05-27T00:15:48.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.461+0000 7f1bcfe00640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1bac0678c0 0x7f1bac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.461+0000 7f1bcfe00640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1bac0678c0 0x7f1bac069d80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0000f70 tx=0x7f1bc0007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:48.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.465+0000 7f1bccc00640 1 -- 172.21.15.98:0/3360873406 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1bb8016070 con 0x7f1bd00c2b10 2024-05-27T00:15:48.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.621+0000 7f1bd7200640 1 -- 172.21.15.98:0/3360873406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1bd0150c00 con 0x7f1bd00c2b10 2024-05-27T00:15:48.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bccc00640 1 -- 172.21.15.98:0/3360873406 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f1bb80f03c0 con 0x7f1bd00c2b10 2024-05-27T00:15:48.627 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-27T00:15:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 -- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1bac0678c0 msgr2=0x7f1bac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1bac0678c0 0x7f1bac069d80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0000f70 tx=0x7f1bc0007b60 comp rx=0 tx=0).stop 2024-05-27T00:15:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 -- 172.21.15.98:0/3360873406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00c2b10 msgr2=0x7f1bd0149490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00c2b10 0x7f1bd0149490 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80ae040 tx=0x7f1bb801acc0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 -- 172.21.15.98:0/3360873406 shutdown_connections 2024-05-27T00:15:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1bac0678c0 0x7f1bac069d80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bd01516b0 0x7f1bd01409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1bd00c2b10 0x7f1bd0149490 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 --2- 172.21.15.98:0/3360873406 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bd00b1e10 0x7f1bd0148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 -- 172.21.15.98:0/3360873406 >> 172.21.15.98:0/3360873406 conn(0x7f1bd00cbf00 msgr2=0x7f1bd0154100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:48.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 -- 172.21.15.98:0/3360873406 shutdown_connections 2024-05-27T00:15:48.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.625+0000 7f1bd7200640 1 -- 172.21.15.98:0/3360873406 wait complete. 2024-05-27T00:15:48.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T00:15:48.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.781+0000 7f70f1400640 1 Processor -- start 2024-05-27T00:15:48.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.781+0000 7f70f1400640 1 -- start start 2024-05-27T00:15:48.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.785+0000 7f70f1400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70ec152990 0x7f70ec06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.785+0000 7f70f1400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70ec15bc40 con 0x7f70ec0b1c20 2024-05-27T00:15:48.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.785+0000 7f70f1400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70ec15bf20 con 0x7f70ec06ecc0 2024-05-27T00:15:48.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.785+0000 7f70f1400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f70ec15c850 con 0x7f70ec152990 2024-05-27T00:15:48.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.785+0000 7f70ea800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70ec152990 0x7f70ec06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.785+0000 7f70eb200640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f70ec0b1c20 0x7f70ec158640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45900/0 (socket says 172.21.15.98:45900) 2024-05-27T00:15:48.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.785+0000 7f70eb200640 1 -- 172.21.15.98:0/2307827983 learned_addr learned my addr 172.21.15.98:0/2307827983 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:48.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.785+0000 7f70ea800640 1 -- 172.21.15.98:0/2307827983 >> v1:172.21.15.204:6789/0 conn(0x7f70ec06ecc0 legacy=0x7f70ec0c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.785+0000 7f70ea800640 1 -- 172.21.15.98:0/2307827983 >> v1:172.21.15.98:6789/0 conn(0x7f70ec0b1c20 legacy=0x7f70ec158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.785+0000 7f70ea800640 1 -- 172.21.15.98:0/2307827983 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ec15c200 con 0x7f70ec152990 2024-05-27T00:15:48.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70ea800640 1 --2- 172.21.15.98:0/2307827983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec152990 0x7f70ec06e780 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0097c0 tx=0x7f70dc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea37e6f74f5ceac8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:48.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70e9e00640 1 -- 172.21.15.98:0/2307827983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0b0070 con 0x7f70ec152990 2024-05-27T00:15:48.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70e9e00640 1 -- 172.21.15.98:0/2307827983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70dc0ab600 con 0x7f70ec152990 2024-05-27T00:15:48.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70e9e00640 1 -- 172.21.15.98:0/2307827983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0ab9d0 con 0x7f70ec152990 2024-05-27T00:15:48.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70f1400640 1 -- 172.21.15.98:0/2307827983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec152990 msgr2=0x7f70ec06e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70f1400640 1 --2- 172.21.15.98:0/2307827983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec152990 0x7f70ec06e780 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0097c0 tx=0x7f70dc01c4a0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70f1400640 1 -- 172.21.15.98:0/2307827983 shutdown_connections 2024-05-27T00:15:48.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70f1400640 1 --2- 172.21.15.98:0/2307827983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec152990 0x7f70ec06e780 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70f1400640 1 -- 172.21.15.98:0/2307827983 >> 172.21.15.98:0/2307827983 conn(0x7f70ec0cbf00 msgr2=0x7f70ec0cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:48.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70f1400640 1 -- 172.21.15.98:0/2307827983 shutdown_connections 2024-05-27T00:15:48.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70f1400640 1 -- 172.21.15.98:0/2307827983 wait complete. 2024-05-27T00:15:48.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70f1400640 1 Processor -- start 2024-05-27T00:15:48.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.789+0000 7f70f1400640 1 -- start start 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70f1400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec06ecc0 0x7f70ec1598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70f1400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0b1c20 0x7f70ec14da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70f1400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70ec152990 0x7f70ec14df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70f1400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec1669f0 con 0x7f70ec06ecc0 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70f1400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec166870 con 0x7f70ec152990 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70f1400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f70ec166b70 con 0x7f70ec0b1c20 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70ea800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec06ecc0 0x7f70ec1598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70ea800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec06ecc0 0x7f70ec1598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54074/0 (socket says 172.21.15.98:54074) 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70ea800640 1 -- 172.21.15.98:0/2907199201 learned_addr learned my addr 172.21.15.98:0/2907199201 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70eb200640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70ec152990 0x7f70ec14df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70e3e00640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0b1c20 0x7f70ec14da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70ea800640 1 -- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0b1c20 msgr2=0x7f70ec14da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70ea800640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0b1c20 0x7f70ec14da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70ea800640 1 -- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70ec152990 msgr2=0x7f70ec14df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70ea800640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70ec152990 0x7f70ec14df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70ea800640 1 -- 172.21.15.98:0/2907199201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ec0c1730 con 0x7f70ec06ecc0 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70eb200640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70ec152990 0x7f70ec14df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70e3e00640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0b1c20 0x7f70ec14da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70ea800640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec06ecc0 0x7f70ec1598d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0190b0 tx=0x7f70dc019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70e2a00640 1 -- 172.21.15.98:0/2907199201 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0b8030 con 0x7f70ec06ecc0 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70e2a00640 1 -- 172.21.15.98:0/2907199201 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70dc0b4410 con 0x7f70ec06ecc0 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70e2a00640 1 -- 172.21.15.98:0/2907199201 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0b0040 con 0x7f70ec06ecc0 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70f1400640 1 -- 172.21.15.98:0/2907199201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70ec0c1170 con 0x7f70ec06ecc0 2024-05-27T00:15:48.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70f1400640 1 -- 172.21.15.98:0/2907199201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70ec15b1a0 con 0x7f70ec06ecc0 2024-05-27T00:15:48.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70f1400640 1 -- 172.21.15.98:0/2907199201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70ec075cb0 con 0x7f70ec06ecc0 2024-05-27T00:15:48.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.793+0000 7f70e2a00640 1 -- 172.21.15.98:0/2907199201 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f70dc0bb070 con 0x7f70ec06ecc0 2024-05-27T00:15:48.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.797+0000 7f70e2a00640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70c80678f0 0x7f70c8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:48.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.797+0000 7f70e2a00640 1 -- 172.21.15.98:0/2907199201 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f70dc11f170 con 0x7f70ec06ecc0 2024-05-27T00:15:48.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.797+0000 7f70e3e00640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70c80678f0 0x7f70c8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:48.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.797+0000 7f70e3e00640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70c80678f0 0x7f70c8069db0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f70ec06e960 tx=0x7f70d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:48.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.801+0000 7f70e2a00640 1 -- 172.21.15.98:0/2907199201 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f70dc0c3020 con 0x7f70ec06ecc0 2024-05-27T00:15:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.953+0000 7f70f1400640 1 -- 172.21.15.98:0/2907199201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f70ec0c5cb0 con 0x7f70ec06ecc0 2024-05-27T00:15:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.953+0000 7f70e2a00640 1 -- 172.21.15.98:0/2907199201 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f70ec0c1730 con 0x7f70ec06ecc0 2024-05-27T00:15:48.957 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-27T00:15:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 -- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70c80678f0 msgr2=0x7f70c8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70c80678f0 0x7f70c8069db0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f70ec06e960 tx=0x7f70d4009040 comp rx=0 tx=0).stop 2024-05-27T00:15:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 -- 172.21.15.98:0/2907199201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec06ecc0 msgr2=0x7f70ec1598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec06ecc0 0x7f70ec1598d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0190b0 tx=0x7f70dc019240 comp rx=0 tx=0).stop 2024-05-27T00:15:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 -- 172.21.15.98:0/2907199201 shutdown_connections 2024-05-27T00:15:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70c80678f0 0x7f70c8069db0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70ec152990 0x7f70ec14df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0b1c20 0x7f70ec14da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 --2- 172.21.15.98:0/2907199201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70ec06ecc0 0x7f70ec1598d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 -- 172.21.15.98:0/2907199201 >> 172.21.15.98:0/2907199201 conn(0x7f70ec0cbf00 msgr2=0x7f70ec072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 -- 172.21.15.98:0/2907199201 shutdown_connections 2024-05-27T00:15:48.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:48.957+0000 7f70f1400640 1 -- 172.21.15.98:0/2907199201 wait complete. 2024-05-27T00:15:48.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T00:15:49.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.113+0000 7fdd63e00640 1 Processor -- start 2024-05-27T00:15:49.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.113+0000 7fdd63e00640 1 -- start start 2024-05-27T00:15:49.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.113+0000 7fdd63e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c4cc0 0x7fdd6415e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.113+0000 7fdd63e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fdd640c3020 0x7fdd640c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.113+0000 7fdd63e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd6415eca0 con 0x7fdd640c7f90 2024-05-27T00:15:49.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.113+0000 7fdd63e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fdd640c56a0 con 0x7fdd640c3020 2024-05-27T00:15:49.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.113+0000 7fdd63e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdd640c5980 con 0x7fdd640c4cc0 2024-05-27T00:15:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd5a000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fdd640c3020 0x7fdd640c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd62a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c4cc0 0x7fdd6415e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd62a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c4cc0 0x7fdd6415e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:46916/0 (socket says 172.21.15.98:46916) 2024-05-27T00:15:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd5a000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fdd640c3020 0x7fdd640c7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:56736/0 (socket says 172.21.15.98:56736) 2024-05-27T00:15:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd62a00640 1 -- 172.21.15.98:0/3294665713 learned_addr learned my addr 172.21.15.98:0/3294665713 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd5a000640 1 -- 172.21.15.98:0/3294665713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c4cc0 msgr2=0x7fdd6415e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd5a000640 1 --2- 172.21.15.98:0/3294665713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c4cc0 0x7fdd6415e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd5a000640 1 -- 172.21.15.98:0/3294665713 >> v1:172.21.15.204:6789/0 conn(0x7fdd640c7f90 legacy=0x7fdd640c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd5a000640 1 -- 172.21.15.98:0/3294665713 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd640c5c60 con 0x7fdd640c3020 2024-05-27T00:15:49.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd5a000640 1 --2- 172.21.15.98:0/3294665713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd640c3020 0x7fdd640c7a50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0097c0 tx=0x7fdd4c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ba3c1c8e98195c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:49.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.117+0000 7fdd62000640 1 -- 172.21.15.98:0/3294665713 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c0b5070 con 0x7fdd640c3020 2024-05-27T00:15:49.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd62000640 1 -- 172.21.15.98:0/3294665713 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd4c0a8990 con 0x7fdd640c3020 2024-05-27T00:15:49.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd62000640 1 -- 172.21.15.98:0/3294665713 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c0a8d60 con 0x7fdd640c3020 2024-05-27T00:15:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 -- 172.21.15.98:0/3294665713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd640c3020 msgr2=0x7fdd640c7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 --2- 172.21.15.98:0/3294665713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd640c3020 0x7fdd640c7a50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0097c0 tx=0x7fdd4c01ec90 comp rx=0 tx=0).stop 2024-05-27T00:15:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 -- 172.21.15.98:0/3294665713 shutdown_connections 2024-05-27T00:15:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 --2- 172.21.15.98:0/3294665713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd640c3020 0x7fdd640c7a50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 --2- 172.21.15.98:0/3294665713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c4cc0 0x7fdd6415e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 -- 172.21.15.98:0/3294665713 >> 172.21.15.98:0/3294665713 conn(0x7fdd640ce000 msgr2=0x7fdd640ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 -- 172.21.15.98:0/3294665713 shutdown_connections 2024-05-27T00:15:49.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 -- 172.21.15.98:0/3294665713 wait complete. 2024-05-27T00:15:49.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 Processor -- start 2024-05-27T00:15:49.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 -- start start 2024-05-27T00:15:49.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd640c3020 0x7fdd641402b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd640c4cc0 0x7fdd641f1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c7f90 0x7fdd641f1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd640c50a0 con 0x7fdd640c4cc0 2024-05-27T00:15:49.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdd640c6220 con 0x7fdd640c3020 2024-05-27T00:15:49.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdd640c5220 con 0x7fdd640c7f90 2024-05-27T00:15:49.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd5a000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd640c4cc0 0x7fdd641f1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd5a000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd640c4cc0 0x7fdd641f1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54082/0 (socket says 172.21.15.98:54082) 2024-05-27T00:15:49.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd5a000640 1 -- 172.21.15.98:0/2855032064 learned_addr learned my addr 172.21.15.98:0/2855032064 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:49.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd62a00640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd640c3020 0x7fdd641402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.121+0000 7fdd63400640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c7f90 0x7fdd641f1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd5a000640 1 -- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c7f90 msgr2=0x7fdd641f1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd5a000640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c7f90 0x7fdd641f1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd5a000640 1 -- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd640c3020 msgr2=0x7fdd641402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd5a000640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd640c3020 0x7fdd641402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd5a000640 1 -- 172.21.15.98:0/2855032064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6413f530 con 0x7fdd640c4cc0 2024-05-27T00:15:49.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd5a000640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd640c4cc0 0x7fdd641f1f00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0b3040 tx=0x7fdd4c01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:49.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd5b400640 1 -- 172.21.15.98:0/2855032064 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c0b5040 con 0x7fdd640c4cc0 2024-05-27T00:15:49.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd5b400640 1 -- 172.21.15.98:0/2855032064 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd4c0c0070 con 0x7fdd640c4cc0 2024-05-27T00:15:49.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd63e00640 1 -- 172.21.15.98:0/2855032064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd64158920 con 0x7fdd640c4cc0 2024-05-27T00:15:49.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd5b400640 1 -- 172.21.15.98:0/2855032064 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c0b08e0 con 0x7fdd640c4cc0 2024-05-27T00:15:49.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd63e00640 1 -- 172.21.15.98:0/2855032064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd6413f7d0 con 0x7fdd640c4cc0 2024-05-27T00:15:49.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd63e00640 1 -- 172.21.15.98:0/2855032064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd64075c40 con 0x7fdd640c4cc0 2024-05-27T00:15:49.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.125+0000 7fdd5b400640 1 -- 172.21.15.98:0/2855032064 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdd4c0b0a80 con 0x7fdd640c4cc0 2024-05-27T00:15:49.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.129+0000 7fdd5b400640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdd400678c0 0x7fdd40069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.129+0000 7fdd5b400640 1 -- 172.21.15.98:0/2855032064 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fdd4c1230a0 con 0x7fdd640c4cc0 2024-05-27T00:15:49.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.129+0000 7fdd62a00640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdd400678c0 0x7fdd40069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.129+0000 7fdd62a00640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdd400678c0 0x7fdd40069d80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009b50 tx=0x7fdd54009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:49.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.133+0000 7fdd5b400640 1 -- 172.21.15.98:0/2855032064 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fdd4c0187f0 con 0x7fdd640c4cc0 2024-05-27T00:15:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.289+0000 7fdd63e00640 1 -- 172.21.15.98:0/2855032064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdd640cb900 con 0x7fdd640c4cc0 2024-05-27T00:15:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.289+0000 7fdd5b400640 1 -- 172.21.15.98:0/2855032064 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7fdd4c0f35a0 con 0x7fdd640c4cc0 2024-05-27T00:15:49.292 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-27T00:15:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.289+0000 7fdd63e00640 1 -- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdd400678c0 msgr2=0x7fdd40069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.289+0000 7fdd63e00640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdd400678c0 0x7fdd40069d80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009b50 tx=0x7fdd54009240 comp rx=0 tx=0).stop 2024-05-27T00:15:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.289+0000 7fdd63e00640 1 -- 172.21.15.98:0/2855032064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd640c4cc0 msgr2=0x7fdd641f1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.289+0000 7fdd63e00640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd640c4cc0 0x7fdd641f1f00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0b3040 tx=0x7fdd4c01ba30 comp rx=0 tx=0).stop 2024-05-27T00:15:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.293+0000 7fdd63e00640 1 -- 172.21.15.98:0/2855032064 shutdown_connections 2024-05-27T00:15:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.293+0000 7fdd63e00640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd640c7f90 0x7fdd641f1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.293+0000 7fdd63e00640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd640c4cc0 0x7fdd641f1f00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.293+0000 7fdd63e00640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdd400678c0 0x7fdd40069d80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.293+0000 7fdd63e00640 1 --2- 172.21.15.98:0/2855032064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd640c3020 0x7fdd641402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.293+0000 7fdd63e00640 1 -- 172.21.15.98:0/2855032064 >> 172.21.15.98:0/2855032064 conn(0x7fdd640ce000 msgr2=0x7fdd640c4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.293+0000 7fdd63e00640 1 -- 172.21.15.98:0/2855032064 shutdown_connections 2024-05-27T00:15:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.293+0000 7fdd63e00640 1 -- 172.21.15.98:0/2855032064 wait complete. 2024-05-27T00:15:49.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:15:49.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd17400640 1 Processor -- start 2024-05-27T00:15:49.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd17400640 1 -- start start 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd17400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efd100cbe10 0x7efd101592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd17400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7efd100cc1f0 0x7efd100c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd17400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd10073570 0x7efd101559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd17400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7efd10159810 con 0x7efd100cbe10 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd17400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7efd100c6dd0 con 0x7efd100cc1f0 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd17400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efd100c70b0 con 0x7efd10073570 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd14a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efd100cbe10 0x7efd101592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd14a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efd100cbe10 0x7efd101592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54084/0 (socket says 172.21.15.98:54084) 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.449+0000 7efd14a00640 1 -- 172.21.15.98:0/3587285619 learned_addr learned my addr 172.21.15.98:0/3587285619 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd07e00640 1 --2- 172.21.15.98:0/3587285619 >> v2:172.21.15.204:3300/0 conn(0x7efd100cc1f0 0x7efd100c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd15400640 1 --2- 172.21.15.98:0/3587285619 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd10073570 0x7efd101559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd14a00640 1 -- 172.21.15.98:0/3587285619 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd10073570 msgr2=0x7efd101559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:49.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd14a00640 1 --2- 172.21.15.98:0/3587285619 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd10073570 0x7efd101559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd14a00640 1 -- 172.21.15.98:0/3587285619 >> v2:172.21.15.204:3300/0 conn(0x7efd100cc1f0 msgr2=0x7efd100c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:15:49.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd14a00640 1 --2- 172.21.15.98:0/3587285619 >> v2:172.21.15.204:3300/0 conn(0x7efd100cc1f0 0x7efd100c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd14a00640 1 -- 172.21.15.98:0/3587285619 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd100c7390 con 0x7efd100cbe10 2024-05-27T00:15:49.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd14a00640 1 --2- 172.21.15.98:0/3587285619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cbe10 0x7efd101592d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efd100cc6d0 tx=0x7efd0001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea5c4344f555b9b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:49.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd0fe00640 1 -- 172.21.15.98:0/3587285619 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd000b5070 con 0x7efd100cbe10 2024-05-27T00:15:49.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd0fe00640 1 -- 172.21.15.98:0/3587285619 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd000a89d0 con 0x7efd100cbe10 2024-05-27T00:15:49.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd0fe00640 1 -- 172.21.15.98:0/3587285619 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd000a8da0 con 0x7efd100cbe10 2024-05-27T00:15:49.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd17400640 1 -- 172.21.15.98:0/3587285619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cbe10 msgr2=0x7efd101592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.453+0000 7efd17400640 1 --2- 172.21.15.98:0/3587285619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cbe10 0x7efd101592d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efd100cc6d0 tx=0x7efd0001ed10 comp rx=0 tx=0).stop 2024-05-27T00:15:49.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 -- 172.21.15.98:0/3587285619 shutdown_connections 2024-05-27T00:15:49.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 --2- 172.21.15.98:0/3587285619 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd10073570 0x7efd101559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 --2- 172.21.15.98:0/3587285619 >> v2:172.21.15.204:3300/0 conn(0x7efd100cc1f0 0x7efd100c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 --2- 172.21.15.98:0/3587285619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cbe10 0x7efd101592d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 -- 172.21.15.98:0/3587285619 >> 172.21.15.98:0/3587285619 conn(0x7efd10067530 msgr2=0x7efd1006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 -- 172.21.15.98:0/3587285619 shutdown_connections 2024-05-27T00:15:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 -- 172.21.15.98:0/3587285619 wait complete. 2024-05-27T00:15:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 Processor -- start 2024-05-27T00:15:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 -- start start 2024-05-27T00:15:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd10073570 0x7efd10072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd100cbe10 0x7efd10154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cc1f0 0x7efd100b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd100c79d0 con 0x7efd100cc1f0 2024-05-27T00:15:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd100c1330 con 0x7efd10073570 2024-05-27T00:15:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efd100c6990 con 0x7efd100cbe10 2024-05-27T00:15:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd15400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cc1f0 0x7efd100b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd15400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cc1f0 0x7efd100b0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54094/0 (socket says 172.21.15.98:54094) 2024-05-27T00:15:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd15400640 1 -- 172.21.15.98:0/3235442755 learned_addr learned my addr 172.21.15.98:0/3235442755 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd14a00640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd10073570 0x7efd10072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd07e00640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd100cbe10 0x7efd10154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd15400640 1 -- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd100cbe10 msgr2=0x7efd10154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd15400640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd100cbe10 0x7efd10154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd15400640 1 -- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd10073570 msgr2=0x7efd10072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd15400640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd10073570 0x7efd10072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd15400640 1 -- 172.21.15.98:0/3235442755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd101f03f0 con 0x7efd100cc1f0 2024-05-27T00:15:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd15400640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cc1f0 0x7efd100b0970 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efd08007e60 tx=0x7efd0800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:49.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd0d600640 1 -- 172.21.15.98:0/3235442755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd08014620 con 0x7efd100cc1f0 2024-05-27T00:15:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd0d600640 1 -- 172.21.15.98:0/3235442755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd08019070 con 0x7efd100cc1f0 2024-05-27T00:15:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd0d600640 1 -- 172.21.15.98:0/3235442755 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd08014a20 con 0x7efd100cc1f0 2024-05-27T00:15:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.457+0000 7efd17400640 1 -- 172.21.15.98:0/3235442755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd101f06d0 con 0x7efd100cc1f0 2024-05-27T00:15:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.461+0000 7efd17400640 1 -- 172.21.15.98:0/3235442755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd101f0c90 con 0x7efd100cc1f0 2024-05-27T00:15:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.461+0000 7efd17400640 1 -- 172.21.15.98:0/3235442755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd10075cb0 con 0x7efd100cc1f0 2024-05-27T00:15:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.461+0000 7efd0d600640 1 -- 172.21.15.98:0/3235442755 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efd08014bc0 con 0x7efd100cc1f0 2024-05-27T00:15:49.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.461+0000 7efd0d600640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efcec0678c0 0x7efcec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.461+0000 7efd0d600640 1 -- 172.21.15.98:0/3235442755 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7efd080865a0 con 0x7efd100cc1f0 2024-05-27T00:15:49.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.461+0000 7efd14a00640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efcec0678c0 0x7efcec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.465+0000 7efd14a00640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efcec0678c0 0x7efcec069d80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efd000b2000 tx=0x7efd00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:49.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.465+0000 7efd0d600640 1 -- 172.21.15.98:0/3235442755 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7efd08056fb0 con 0x7efd100cc1f0 2024-05-27T00:15:49.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 -- 172.21.15.98:0/3235442755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd101f0e60 con 0x7efd100cc1f0 2024-05-27T00:15:49.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd0d600640 1 -- 172.21.15.98:0/3235442755 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7efd080568b0 con 0x7efd100cc1f0 2024-05-27T00:15:49.663 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:49.663 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:15:49.663 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:15:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 -- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efcec0678c0 msgr2=0x7efcec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efcec0678c0 0x7efcec069d80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efd000b2000 tx=0x7efd00009040 comp rx=0 tx=0).stop 2024-05-27T00:15:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 -- 172.21.15.98:0/3235442755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cc1f0 msgr2=0x7efd100b0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cc1f0 0x7efd100b0970 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efd08007e60 tx=0x7efd0800e4b0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 -- 172.21.15.98:0/3235442755 shutdown_connections 2024-05-27T00:15:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd100cc1f0 0x7efd100b0970 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd100cbe10 0x7efd10154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efcec0678c0 0x7efcec069d80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 --2- 172.21.15.98:0/3235442755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd10073570 0x7efd10072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 -- 172.21.15.98:0/3235442755 >> 172.21.15.98:0/3235442755 conn(0x7efd10067530 msgr2=0x7efd100ca3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:49.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 -- 172.21.15.98:0/3235442755 shutdown_connections 2024-05-27T00:15:49.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.661+0000 7efd17400640 1 -- 172.21.15.98:0/3235442755 wait complete. 2024-05-27T00:15:49.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:15:49.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.813+0000 7f9ea0000640 1 Processor -- start 2024-05-27T00:15:49.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.813+0000 7f9ea0000640 1 -- start start 2024-05-27T00:15:49.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.817+0000 7f9ea0000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e98073570 0x7f9e981592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.817+0000 7f9ea0000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e980c1f60 con 0x7f9e980cbe10 2024-05-27T00:15:49.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.817+0000 7f9ea0000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e980c6dd0 con 0x7f9e980cc1f0 2024-05-27T00:15:49.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.817+0000 7f9ea0000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9e98159810 con 0x7f9e98073570 2024-05-27T00:15:49.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.817+0000 7f9e9d600640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9e980cbe10 0x7f9e980c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45916/0 (socket says 172.21.15.98:45916) 2024-05-27T00:15:49.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.817+0000 7f9e9d600640 1 -- 172.21.15.98:0/2769535031 learned_addr learned my addr 172.21.15.98:0/2769535031 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:15:49.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.817+0000 7f9e9e000640 1 --2- 172.21.15.98:0/2769535031 >> v2:172.21.15.98:3300/0 conn(0x7f9e98073570 0x7f9e981592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.817+0000 7f9e9e000640 1 -- 172.21.15.98:0/2769535031 >> v1:172.21.15.204:6789/0 conn(0x7f9e980cc1f0 legacy=0x7f9e981559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.817+0000 7f9e9e000640 1 -- 172.21.15.98:0/2769535031 >> v1:172.21.15.98:6789/0 conn(0x7f9e980cbe10 legacy=0x7f9e980c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.817+0000 7f9e9e000640 1 -- 172.21.15.98:0/2769535031 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e980c70b0 con 0x7f9e98073570 2024-05-27T00:15:49.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9e9e000640 1 --2- 172.21.15.98:0/2769535031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e98073570 0x7f9e981592d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009d90 tx=0x7f9e9001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4b6cc350784a5f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:49.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9e97e00640 1 -- 172.21.15.98:0/2769535031 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e900b4070 con 0x7f9e98073570 2024-05-27T00:15:49.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9e97e00640 1 -- 172.21.15.98:0/2769535031 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e900af600 con 0x7f9e98073570 2024-05-27T00:15:49.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9e97e00640 1 -- 172.21.15.98:0/2769535031 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e900af9d0 con 0x7f9e98073570 2024-05-27T00:15:49.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9ea0000640 1 -- 172.21.15.98:0/2769535031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e98073570 msgr2=0x7f9e981592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9ea0000640 1 --2- 172.21.15.98:0/2769535031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e98073570 0x7f9e981592d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009d90 tx=0x7f9e9001f150 comp rx=0 tx=0).stop 2024-05-27T00:15:49.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9ea0000640 1 -- 172.21.15.98:0/2769535031 shutdown_connections 2024-05-27T00:15:49.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9ea0000640 1 --2- 172.21.15.98:0/2769535031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e98073570 0x7f9e981592d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9ea0000640 1 -- 172.21.15.98:0/2769535031 >> 172.21.15.98:0/2769535031 conn(0x7f9e98067530 msgr2=0x7f9e9806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:49.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9ea0000640 1 -- 172.21.15.98:0/2769535031 shutdown_connections 2024-05-27T00:15:49.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9ea0000640 1 -- 172.21.15.98:0/2769535031 wait complete. 2024-05-27T00:15:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9ea0000640 1 Processor -- start 2024-05-27T00:15:49.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.821+0000 7f9ea0000640 1 -- start start 2024-05-27T00:15:49.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9ea0000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e98073570 0x7f9e9814e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9ea0000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e980cbe10 0x7f9e981de8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9ea0000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e980cc1f0 0x7f9e981567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9ea0000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9e980c1330 con 0x7f9e980cbe10 2024-05-27T00:15:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9ea0000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9e980c1050 con 0x7f9e98073570 2024-05-27T00:15:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9ea0000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9e980c79d0 con 0x7f9e980cc1f0 2024-05-27T00:15:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9cc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e980cbe10 0x7f9e981de8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9cc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e980cbe10 0x7f9e981de8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54112/0 (socket says 172.21.15.98:54112) 2024-05-27T00:15:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9cc00640 1 -- 172.21.15.98:0/2351723324 learned_addr learned my addr 172.21.15.98:0/2351723324 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:15:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9d600640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e98073570 0x7f9e9814e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9e000640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e980cc1f0 0x7f9e981567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9cc00640 1 -- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e980cc1f0 msgr2=0x7f9e981567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9cc00640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e980cc1f0 0x7f9e981567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9cc00640 1 -- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e98073570 msgr2=0x7f9e9814e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9cc00640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e98073570 0x7f9e9814e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9cc00640 1 -- 172.21.15.98:0/2351723324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e981ee540 con 0x7f9e980cbe10 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e9cc00640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e980cbe10 0x7f9e981de8d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400c3a0 tx=0x7f9e8400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e95600640 1 -- 172.21.15.98:0/2351723324 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e840052d0 con 0x7f9e980cbe10 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e95600640 1 -- 172.21.15.98:0/2351723324 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e84005470 con 0x7f9e980cbe10 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e95600640 1 -- 172.21.15.98:0/2351723324 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84007260 con 0x7f9e980cbe10 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9ea0000640 1 -- 172.21.15.98:0/2351723324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e981ee820 con 0x7f9e980cbe10 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9ea0000640 1 -- 172.21.15.98:0/2351723324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e981eede0 con 0x7f9e980cbe10 2024-05-27T00:15:49.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9ea0000640 1 -- 172.21.15.98:0/2351723324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e98075cb0 con 0x7f9e980cbe10 2024-05-27T00:15:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.825+0000 7f9e95600640 1 -- 172.21.15.98:0/2351723324 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9e84007400 con 0x7f9e980cbe10 2024-05-27T00:15:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.829+0000 7f9e95600640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e780678c0 0x7f9e78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:15:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.829+0000 7f9e95600640 1 -- 172.21.15.98:0/2351723324 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f9e84084fe0 con 0x7f9e980cbe10 2024-05-27T00:15:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.829+0000 7f9e9d600640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e780678c0 0x7f9e78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:15:49.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.829+0000 7f9e9d600640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e780678c0 0x7f9e78069d80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c005890 tx=0x7f9e8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:15:49.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.833+0000 7f9e95600640 1 -- 172.21.15.98:0/2351723324 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9e840554d0 con 0x7f9e980cbe10 2024-05-27T00:15:49.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.985+0000 7f9ea0000640 1 -- 172.21.15.98:0/2351723324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e981e4710 con 0x7f9e980cbe10 2024-05-27T00:15:49.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9e95600640 1 -- 172.21.15.98:0/2351723324 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16581 (secure 0 0 0) 0x7f9e840597d0 con 0x7f9e980cbe10 2024-05-27T00:15:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:15:49.991 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:15:26.579229+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:15:49.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 -- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e780678c0 msgr2=0x7f9e78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e780678c0 0x7f9e78069d80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c005890 tx=0x7f9e8c009040 comp rx=0 tx=0).stop 2024-05-27T00:15:49.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 -- 172.21.15.98:0/2351723324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e980cbe10 msgr2=0x7f9e981de8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:15:49.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e980cbe10 0x7f9e981de8d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400c3a0 tx=0x7f9e8400c860 comp rx=0 tx=0).stop 2024-05-27T00:15:49.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 -- 172.21.15.98:0/2351723324 shutdown_connections 2024-05-27T00:15:49.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e980cc1f0 0x7f9e981567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e980cbe10 0x7f9e981de8d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e780678c0 0x7f9e78069d80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 --2- 172.21.15.98:0/2351723324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e98073570 0x7f9e9814e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:15:49.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 -- 172.21.15.98:0/2351723324 >> 172.21.15.98:0/2351723324 conn(0x7f9e98067530 msgr2=0x7f9e980cab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:15:49.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 -- 172.21.15.98:0/2351723324 shutdown_connections 2024-05-27T00:15:49.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:15:49.989+0000 7f9ea0000640 1 -- 172.21.15.98:0/2351723324 wait complete. 2024-05-27T00:15:50.012 DEBUG:tasks.mds_thrash:Yielding 2024-05-27T00:15:50.012 INFO:teuthology.run_tasks:Running task workunit... 2024-05-27T00:15:50.031 INFO:tasks.workunit:Pulling workunits from ref 7d7cb32d1d130135a98121a564d709025c2154b6 2024-05-27T00:15:50.031 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-27T00:15:50.031 INFO:tasks.workunit:timeout=6h 2024-05-27T00:15:50.032 INFO:tasks.workunit:cleanup=True 2024-05-27T00:15:50.032 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-27T00:15:50.032 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-05-27T00:15:50.032 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2024-05-27T00:15:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-27T00:15:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-27T00:15:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 2ch/44d Inode: 1 Links: 2 2024-05-27T00:15:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T00:15:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-05-27T00:15:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-05-27 00:15:30.606495065 +0000 2024-05-27T00:15:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-05-27 00:15:43.321760205 +0000 2024-05-27T00:15:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-05-27 00:15:30.606495065 +0000 2024-05-27T00:15:50.042 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-27T00:15:50.042 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-27T00:15:50.110 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7d7cb32d1d130135a98121a564d709025c2154b6 2024-05-27T00:15:50.163 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-27T00:16:07.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:07.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.177+0000 7fb0d3800640 1 Processor -- start 2024-05-27T00:16:07.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.177+0000 7fb0d3800640 1 -- start start 2024-05-27T00:16:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0d3800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb0cc0cc8f0 0x7fb0cc070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0d3800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb0cc0c8670 0x7fb0cc0c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0d3800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0cc0c1000 con 0x7fb0cc1511d0 2024-05-27T00:16:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0d3800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb0cc0b1dd0 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0d3800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb0cc0c1a20 con 0x7fb0cc0c8670 2024-05-27T00:16:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0cbe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb0cc0cc8f0 0x7fb0cc070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0cbe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb0cc0cc8f0 0x7fb0cc070e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50292/0 (socket says 172.21.15.98:50292) 2024-05-27T00:16:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0cbe00640 1 -- 172.21.15.98:0/203477981 learned_addr learned my addr 172.21.15.98:0/203477981 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0d1800640 1 --2- 172.21.15.98:0/203477981 >> v2:172.21.15.204:3300/0 conn(0x7fb0cc0c8670 0x7fb0cc0c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0cbe00640 1 -- 172.21.15.98:0/203477981 >> v2:172.21.15.204:3300/0 conn(0x7fb0cc0c8670 msgr2=0x7fb0cc0c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0cbe00640 1 --2- 172.21.15.98:0/203477981 >> v2:172.21.15.204:3300/0 conn(0x7fb0cc0c8670 0x7fb0cc0c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0cbe00640 1 -- 172.21.15.98:0/203477981 >> v1:172.21.15.204:6789/0 conn(0x7fb0cc1511d0 legacy=0x7fb0cc0c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.181+0000 7fb0cbe00640 1 -- 172.21.15.98:0/203477981 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0cc1515b0 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0cbe00640 1 --2- 172.21.15.98:0/203477981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 0x7fb0cc070e10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc06e660 tx=0x7fb0b801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6779fcdbd5a91c55 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0cb400640 1 -- 172.21.15.98:0/203477981 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80b0070 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0cb400640 1 -- 172.21.15.98:0/203477981 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b80ab600 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0cb400640 1 -- 172.21.15.98:0/203477981 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80ab9d0 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0d3800640 1 -- 172.21.15.98:0/203477981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 msgr2=0x7fb0cc070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0d3800640 1 --2- 172.21.15.98:0/203477981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 0x7fb0cc070e10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc06e660 tx=0x7fb0b801c490 comp rx=0 tx=0).stop 2024-05-27T00:16:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0d3800640 1 -- 172.21.15.98:0/203477981 shutdown_connections 2024-05-27T00:16:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0d3800640 1 --2- 172.21.15.98:0/203477981 >> v2:172.21.15.204:3300/0 conn(0x7fb0cc0c8670 0x7fb0cc0c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0d3800640 1 --2- 172.21.15.98:0/203477981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 0x7fb0cc070e10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0d3800640 1 -- 172.21.15.98:0/203477981 >> 172.21.15.98:0/203477981 conn(0x7fb0cc067530 msgr2=0x7fb0cc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0d3800640 1 -- 172.21.15.98:0/203477981 shutdown_connections 2024-05-27T00:16:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0d3800640 1 -- 172.21.15.98:0/203477981 wait complete. 2024-05-27T00:16:07.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0d3800640 1 Processor -- start 2024-05-27T00:16:07.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.185+0000 7fb0d3800640 1 -- start start 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d3800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0cc0c8670 0x7fb0cc1f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d3800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 0x7fb0cc149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d3800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0cc1511d0 0x7fb0cc072880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d3800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc0c2150 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d3800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc0b1a60 con 0x7fb0cc1511d0 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d3800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb0cc0c22d0 con 0x7fb0cc0c8670 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0cbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 0x7fb0cc149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0cbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 0x7fb0cc149c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50296/0 (socket says 172.21.15.98:50296) 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0cbe00640 1 -- 172.21.15.98:0/3516492722 learned_addr learned my addr 172.21.15.98:0/3516492722 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d1800640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0cc1511d0 0x7fb0cc072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d0e00640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0cc0c8670 0x7fb0cc1f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0cbe00640 1 -- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0cc0c8670 msgr2=0x7fb0cc1f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0cbe00640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0cc0c8670 0x7fb0cc1f7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0cbe00640 1 -- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0cc1511d0 msgr2=0x7fb0cc072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0cbe00640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0cc1511d0 0x7fb0cc072880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0cbe00640 1 -- 172.21.15.98:0/3516492722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0cc0c4330 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0cbe00640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 0x7fb0cc149c50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80190a0 tx=0x7fb0b8019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:07.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0c8c00640 1 -- 172.21.15.98:0/3516492722 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80b0070 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0c8c00640 1 -- 172.21.15.98:0/3516492722 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b80b1040 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0c8c00640 1 -- 172.21.15.98:0/3516492722 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80aa7f0 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d3800640 1 -- 172.21.15.98:0/3516492722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0cc0c4610 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d3800640 1 -- 172.21.15.98:0/3516492722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0cc0c4bd0 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0d3800640 1 -- 172.21.15.98:0/3516492722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0cc075cb0 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.189+0000 7fb0c8c00640 1 -- 172.21.15.98:0/3516492722 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb0b80aa990 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.193+0000 7fb0c8c00640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0ac0678c0 0x7fb0ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.193+0000 7fb0c8c00640 1 -- 172.21.15.98:0/3516492722 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fb0b811fa60 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.193+0000 7fb0d0e00640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0ac0678c0 0x7fb0ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.193+0000 7fb0d0e00640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0ac0678c0 0x7fb0ac069d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc149be0 tx=0x7fb0bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:07.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.197+0000 7fb0c8c00640 1 -- 172.21.15.98:0/3516492722 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb0b80c3020 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.385+0000 7fb0d3800640 1 -- 172.21.15.98:0/3516492722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0cc0713c0 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.385+0000 7fb0c8c00640 1 -- 172.21.15.98:0/3516492722 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7fb0b8016110 con 0x7fb0cc0cc8f0 2024-05-27T00:16:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:07.390 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:16:07.390 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:16:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 -- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0ac0678c0 msgr2=0x7fb0ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0ac0678c0 0x7fb0ac069d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc149be0 tx=0x7fb0bc009040 comp rx=0 tx=0).stop 2024-05-27T00:16:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 -- 172.21.15.98:0/3516492722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 msgr2=0x7fb0cc149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 0x7fb0cc149c50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80190a0 tx=0x7fb0b8019230 comp rx=0 tx=0).stop 2024-05-27T00:16:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 -- 172.21.15.98:0/3516492722 shutdown_connections 2024-05-27T00:16:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0ac0678c0 0x7fb0ac069d80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0cc1511d0 0x7fb0cc072880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0cc0cc8f0 0x7fb0cc149c50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 --2- 172.21.15.98:0/3516492722 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0cc0c8670 0x7fb0cc1f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 -- 172.21.15.98:0/3516492722 >> 172.21.15.98:0/3516492722 conn(0x7fb0cc067530 msgr2=0x7fb0cc14f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:07.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 -- 172.21.15.98:0/3516492722 shutdown_connections 2024-05-27T00:16:07.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.389+0000 7fb0d3800640 1 -- 172.21.15.98:0/3516492722 wait complete. 2024-05-27T00:16:07.411 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-05-27T00:16:07.411 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-27T00:16:07.411 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:16:07.424 INFO:tasks.ceph.mds.c:Stopped 2024-05-27T00:16:07.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.597+0000 7fbf0c200640 1 Processor -- start 2024-05-27T00:16:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.597+0000 7fbf0c200640 1 -- start start 2024-05-27T00:16:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf0c200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fbf040731f0 0x7fbf040c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf0c200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf040c79f0 0x7fbf04157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf0c200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf040c17d0 con 0x7fbf040735d0 2024-05-27T00:16:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf0c200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fbf04071990 con 0x7fbf040731f0 2024-05-27T00:16:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf0c200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbf04070e10 con 0x7fbf040c79f0 2024-05-27T00:16:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf08e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbf040735d0 0x7fbf04151430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58852/0 (socket says 172.21.15.98:58852) 2024-05-27T00:16:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf08e00640 1 -- 172.21.15.98:0/4005162232 learned_addr learned my addr 172.21.15.98:0/4005162232 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf0a200640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf040c79f0 0x7fbf04157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf09800640 1 --2- 172.21.15.98:0/4005162232 >> v2:172.21.15.204:3300/0 conn(0x7fbf040731f0 0x7fbf040c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769691248 0 0) 0x7fbf040c17d0 con 0x7fbf040735d0 2024-05-27T00:16:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeec003530 con 0x7fbf040735d0 2024-05-27T00:16:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3281068237 0 0) 0x7fbeec003530 con 0x7fbf040735d0 2024-05-27T00:16:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf040c79f0 msgr2=0x7fbf04157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf040c79f0 0x7fbf04157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 >> v2:172.21.15.204:3300/0 conn(0x7fbf040731f0 msgr2=0x7fbf040c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 --2- 172.21.15.98:0/4005162232 >> v2:172.21.15.204:3300/0 conn(0x7fbf040731f0 0x7fbf040c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf040710f0 con 0x7fbf040735d0 2024-05-27T00:16:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fbef0001680 con 0x7fbf040735d0 2024-05-27T00:16:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 >> v1:172.21.15.98:6789/0 conn(0x7fbf040735d0 legacy=0x7fbf04151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf040c79f0 0x7fbeec0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeec003f40 0x7fbeec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeec004d50 0x7fbeec0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf04071990 con 0x7fbeec003f40 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbf04070e10 con 0x7fbeec004d50 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbeec00ba00 con 0x7fbf040c79f0 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf08e00640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeec003f40 0x7fbeec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf09800640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf040c79f0 0x7fbeec0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf0a200640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeec004d50 0x7fbeec0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf08e00640 1 -- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf040c79f0 msgr2=0x7fbeec0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf08e00640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf040c79f0 0x7fbeec0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf08e00640 1 -- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeec004d50 msgr2=0x7fbeec0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf08e00640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeec004d50 0x7fbeec0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf08e00640 1 -- 172.21.15.98:0/4005162232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeec003530 con 0x7fbeec003f40 2024-05-27T00:16:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.601+0000 7fbf09800640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf040c79f0 0x7fbeec0039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:16:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf08e00640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeec003f40 0x7fbeec004810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbef000c290 tx=0x7fbef001cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc486e661d95e7a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef0007d60 con 0x7fbeec003f40 2024-05-27T00:16:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbef00a9020 con 0x7fbeec003f40 2024-05-27T00:16:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf03e00640 1 -- 172.21.15.98:0/4005162232 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef00a5b90 con 0x7fbeec003f40 2024-05-27T00:16:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf0c200640 1 -- 172.21.15.98:0/4005162232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeec003f40 msgr2=0x7fbeec004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf0c200640 1 --2- 172.21.15.98:0/4005162232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbeec003f40 0x7fbeec004810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbef000c290 tx=0x7fbef001cd20 comp rx=0 tx=0).stop 2024-05-27T00:16:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf0a200640 1 -- 172.21.15.98:0/4005162232 reap_dead start 2024-05-27T00:16:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf0c200640 1 -- 172.21.15.98:0/4005162232 shutdown_connections 2024-05-27T00:16:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf0c200640 1 -- 172.21.15.98:0/4005162232 >> 172.21.15.98:0/4005162232 conn(0x7fbf04067530 msgr2=0x7fbf0406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf0c200640 1 -- 172.21.15.98:0/4005162232 shutdown_connections 2024-05-27T00:16:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf0c200640 1 -- 172.21.15.98:0/4005162232 wait complete. 2024-05-27T00:16:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.605+0000 7fbf0c200640 1 Processor -- start 2024-05-27T00:16:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0c200640 1 -- start start 2024-05-27T00:16:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0c200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeec003f40 0x7fbf04156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0c200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf041f2210 0x7fbf0414cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0c200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf04144220 0x7fbf041446d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0c200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf040c8400 con 0x7fbf04144220 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0c200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbf040716b0 con 0x7fbeec003f40 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0c200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbf040c13f0 con 0x7fbf041f2210 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0a200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf04144220 0x7fbf041446d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0a200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf04144220 0x7fbf041446d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50314/0 (socket says 172.21.15.98:50314) 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0a200640 1 -- 172.21.15.98:0/723682088 learned_addr learned my addr 172.21.15.98:0/723682088 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf08e00640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf041f2210 0x7fbf0414cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf09800640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeec003f40 0x7fbf04156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0a200640 1 -- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf041f2210 msgr2=0x7fbf0414cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0a200640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf041f2210 0x7fbf0414cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0a200640 1 -- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeec003f40 msgr2=0x7fbf04156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0a200640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeec003f40 0x7fbf04156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0a200640 1 -- 172.21.15.98:0/723682088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0406f590 con 0x7fbf04144220 2024-05-27T00:16:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0a200640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf04144220 0x7fbf041446d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbf04070bf0 tx=0x7fbefc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf01600640 1 -- 172.21.15.98:0/723682088 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc011e40 con 0x7fbf04144220 2024-05-27T00:16:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf01600640 1 -- 172.21.15.98:0/723682088 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbefc01b070 con 0x7fbf04144220 2024-05-27T00:16:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0c200640 1 -- 172.21.15.98:0/723682088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf04157400 con 0x7fbf04144220 2024-05-27T00:16:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf01600640 1 -- 172.21.15.98:0/723682088 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0039a0 con 0x7fbf04144220 2024-05-27T00:16:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0c200640 1 -- 172.21.15.98:0/723682088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf041ee7d0 con 0x7fbf04144220 2024-05-27T00:16:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.609+0000 7fbf0c200640 1 -- 172.21.15.98:0/723682088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf04075cb0 con 0x7fbf04144220 2024-05-27T00:16:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.613+0000 7fbf01600640 1 -- 172.21.15.98:0/723682088 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbefc003b40 con 0x7fbf04144220 2024-05-27T00:16:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.613+0000 7fbf01600640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbee40678f0 0x7fbee4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.613+0000 7fbf01600640 1 -- 172.21.15.98:0/723682088 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fbefc0862e0 con 0x7fbf04144220 2024-05-27T00:16:07.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.613+0000 7fbf09800640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbee40678f0 0x7fbee4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:07.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.613+0000 7fbf09800640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbee40678f0 0x7fbee4069db0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009a30 tx=0x7fbef8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:07.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.617+0000 7fbf01600640 1 -- 172.21.15.98:0/723682088 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fbefc056d90 con 0x7fbf04144220 2024-05-27T00:16:07.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.801+0000 7fbf0c200640 1 -- 172.21.15.98:0/723682088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf041551e0 con 0x7fbf04144220 2024-05-27T00:16:07.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf01600640 1 -- 172.21.15.98:0/723682088 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7fbefc056690 con 0x7fbf04144220 2024-05-27T00:16:07.806 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:07.806 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:16:07.807 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:16:07.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 -- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbee40678f0 msgr2=0x7fbee4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbee40678f0 0x7fbee4069db0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009a30 tx=0x7fbef8008040 comp rx=0 tx=0).stop 2024-05-27T00:16:07.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 -- 172.21.15.98:0/723682088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf04144220 msgr2=0x7fbf041446d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:07.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf04144220 0x7fbf041446d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbf04070bf0 tx=0x7fbefc0115a0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 -- 172.21.15.98:0/723682088 shutdown_connections 2024-05-27T00:16:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf04144220 0x7fbf041446d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf041f2210 0x7fbf0414cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbee40678f0 0x7fbee4069db0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 --2- 172.21.15.98:0/723682088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbeec003f40 0x7fbf04156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 -- 172.21.15.98:0/723682088 >> 172.21.15.98:0/723682088 conn(0x7fbf04067530 msgr2=0x7fbf0414dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 -- 172.21.15.98:0/723682088 shutdown_connections 2024-05-27T00:16:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:07.805+0000 7fbf0c200640 1 -- 172.21.15.98:0/723682088 wait complete. 2024-05-27T00:16:07.828 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:16:09.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.969+0000 7fd677a00640 1 Processor -- start 2024-05-27T00:16:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.969+0000 7fd677a00640 1 -- start start 2024-05-27T00:16:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd677a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 0x7fd6700c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd677a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6700c17d0 con 0x7fd6700c79f0 2024-05-27T00:16:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd677a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd670070e10 con 0x7fd6700735d0 2024-05-27T00:16:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd677a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd670071990 con 0x7fd6700731f0 2024-05-27T00:16:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd675a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd6700c79f0 0x7fd670157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58866/0 (socket says 172.21.15.98:58866) 2024-05-27T00:16:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd675a00640 1 -- 172.21.15.98:0/2837738228 learned_addr learned my addr 172.21.15.98:0/2837738228 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd675000640 1 --2- 172.21.15.98:0/2837738228 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 0x7fd6700c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd66fe00640 1 -- 172.21.15.98:0/2837738228 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2265967517 0 0) 0x7fd6700c17d0 con 0x7fd6700c79f0 2024-05-27T00:16:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd66fe00640 1 -- 172.21.15.98:0/2837738228 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd654005740 con 0x7fd6700c79f0 2024-05-27T00:16:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd675000640 1 -- 172.21.15.98:0/2837738228 >> v1:172.21.15.204:6789/0 conn(0x7fd6700735d0 legacy=0x7fd670151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd675000640 1 -- 172.21.15.98:0/2837738228 >> v1:172.21.15.98:6789/0 conn(0x7fd6700c79f0 legacy=0x7fd670157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd66fe00640 1 -- 172.21.15.98:0/2837738228 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2380348626 0 0) 0x7fd654005740 con 0x7fd6700c79f0 2024-05-27T00:16:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.973+0000 7fd675000640 1 -- 172.21.15.98:0/2837738228 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6700710f0 con 0x7fd6700731f0 2024-05-27T00:16:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd675000640 1 --2- 172.21.15.98:0/2837738228 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 0x7fd6700c0eb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6600097f0 tx=0x7fd66001ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50a01b43bc3cc54f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd66fe00640 1 -- 172.21.15.98:0/2837738228 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6600b5070 con 0x7fd6700731f0 2024-05-27T00:16:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd66fe00640 1 -- 172.21.15.98:0/2837738228 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6600a8670 con 0x7fd6700731f0 2024-05-27T00:16:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd66fe00640 1 -- 172.21.15.98:0/2837738228 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668001a00 con 0x7fd6700731f0 2024-05-27T00:16:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd677a00640 1 -- 172.21.15.98:0/2837738228 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 msgr2=0x7fd6700c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd677a00640 1 --2- 172.21.15.98:0/2837738228 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 0x7fd6700c0eb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6600097f0 tx=0x7fd66001ebf0 comp rx=0 tx=0).stop 2024-05-27T00:16:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd677a00640 1 -- 172.21.15.98:0/2837738228 shutdown_connections 2024-05-27T00:16:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd677a00640 1 --2- 172.21.15.98:0/2837738228 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 0x7fd6700c0eb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd677a00640 1 -- 172.21.15.98:0/2837738228 >> 172.21.15.98:0/2837738228 conn(0x7fd670067530 msgr2=0x7fd67006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd677a00640 1 -- 172.21.15.98:0/2837738228 shutdown_connections 2024-05-27T00:16:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd677a00640 1 -- 172.21.15.98:0/2837738228 wait complete. 2024-05-27T00:16:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd677a00640 1 Processor -- start 2024-05-27T00:16:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.977+0000 7fd677a00640 1 -- start start 2024-05-27T00:16:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd677a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 0x7fd670148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd677a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6700735d0 0x7fd6701f0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd677a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700c79f0 0x7fd6701f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd677a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd6700c8400 con 0x7fd6700c79f0 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd677a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd6700716b0 con 0x7fd6700735d0 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd677a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd6700c13f0 con 0x7fd6700731f0 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd675a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700c79f0 0x7fd6701f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd675a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700c79f0 0x7fd6701f14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50326/0 (socket says 172.21.15.98:50326) 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd675a00640 1 -- 172.21.15.98:0/513153744 learned_addr learned my addr 172.21.15.98:0/513153744 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd667e00640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6700735d0 0x7fd6701f0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd675000640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 0x7fd670148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd675a00640 1 -- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 msgr2=0x7fd670148620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd675a00640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 0x7fd670148620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd675a00640 1 -- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6700735d0 msgr2=0x7fd6701f0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd675a00640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6700735d0 0x7fd6701f0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd675a00640 1 -- 172.21.15.98:0/513153744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67014a5a0 con 0x7fd6700c79f0 2024-05-27T00:16:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd675a00640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700c79f0 0x7fd6701f14b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd66800cc90 tx=0x7fd66800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd66d600640 1 -- 172.21.15.98:0/513153744 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66800ce40 con 0x7fd6700c79f0 2024-05-27T00:16:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd66d600640 1 -- 172.21.15.98:0/513153744 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd668012400 con 0x7fd6700c79f0 2024-05-27T00:16:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd677a00640 1 -- 172.21.15.98:0/513153744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67014a860 con 0x7fd6700c79f0 2024-05-27T00:16:09.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd66d600640 1 -- 172.21.15.98:0/513153744 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6680127f0 con 0x7fd6700c79f0 2024-05-27T00:16:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd677a00640 1 -- 172.21.15.98:0/513153744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67014ae20 con 0x7fd6700c79f0 2024-05-27T00:16:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.981+0000 7fd677a00640 1 -- 172.21.15.98:0/513153744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd670075cb0 con 0x7fd6700c79f0 2024-05-27T00:16:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.985+0000 7fd66d600640 1 -- 172.21.15.98:0/513153744 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd668012990 con 0x7fd6700c79f0 2024-05-27T00:16:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.985+0000 7fd66d600640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd64c0678c0 0x7fd64c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.985+0000 7fd66d600640 1 -- 172.21.15.98:0/513153744 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fd668086510 con 0x7fd6700c79f0 2024-05-27T00:16:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.985+0000 7fd675000640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd64c0678c0 0x7fd64c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.985+0000 7fd675000640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd64c0678c0 0x7fd64c069d80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd6600b2000 tx=0x7fd660008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:09.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:09.989+0000 7fd66d600640 1 -- 172.21.15.98:0/513153744 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd668056850 con 0x7fd6700c79f0 2024-05-27T00:16:10.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.177+0000 7fd677a00640 1 -- 172.21.15.98:0/513153744 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd67014df00 con 0x7fd6700c79f0 2024-05-27T00:16:10.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.177+0000 7fd66d600640 1 -- 172.21.15.98:0/513153744 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7fd66805ab50 con 0x7fd6700c79f0 2024-05-27T00:16:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:10.181 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:16:10.181 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:16:10.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 -- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd64c0678c0 msgr2=0x7fd64c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:10.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd64c0678c0 0x7fd64c069d80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd6600b2000 tx=0x7fd660008040 comp rx=0 tx=0).stop 2024-05-27T00:16:10.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 -- 172.21.15.98:0/513153744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700c79f0 msgr2=0x7fd6701f14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:10.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700c79f0 0x7fd6701f14b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd66800cc90 tx=0x7fd66800a430 comp rx=0 tx=0).stop 2024-05-27T00:16:10.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 -- 172.21.15.98:0/513153744 shutdown_connections 2024-05-27T00:16:10.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6700c79f0 0x7fd6701f14b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:10.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd64c0678c0 0x7fd64c069d80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:10.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6700735d0 0x7fd6701f0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:10.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 --2- 172.21.15.98:0/513153744 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6700731f0 0x7fd670148620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:10.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 -- 172.21.15.98:0/513153744 >> 172.21.15.98:0/513153744 conn(0x7fd670067530 msgr2=0x7fd670151200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:10.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 -- 172.21.15.98:0/513153744 shutdown_connections 2024-05-27T00:16:10.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:10.181+0000 7fd677a00640 1 -- 172.21.15.98:0/513153744 wait complete. 2024-05-27T00:16:10.201 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:16:12.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6c3000640 1 Processor -- start 2024-05-27T00:16:12.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6c3000640 1 -- start start 2024-05-27T00:16:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6c3000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb6bc073570 0x7fb6bc1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6c3000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc0c1f60 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6c3000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc0c6dd0 con 0x7fb6bc0cbe10 2024-05-27T00:16:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6c3000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb6bc159810 con 0x7fb6bc073570 2024-05-27T00:16:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6c1000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb6bc073570 0x7fb6bc1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6b3400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb6bc0cc1f0 0x7fb6bc1559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58880/0 (socket says 172.21.15.98:58880) 2024-05-27T00:16:12.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6b3400640 1 -- 172.21.15.98:0/4260846996 learned_addr learned my addr 172.21.15.98:0/4260846996 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6c1000640 1 -- 172.21.15.98:0/4260846996 >> v1:172.21.15.204:6789/0 conn(0x7fb6bc0cbe10 legacy=0x7fb6bc0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6c1000640 1 -- 172.21.15.98:0/4260846996 >> v1:172.21.15.98:6789/0 conn(0x7fb6bc0cc1f0 legacy=0x7fb6bc1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6c1000640 1 -- 172.21.15.98:0/4260846996 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc0c70b0 con 0x7fb6bc073570 2024-05-27T00:16:12.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.345+0000 7fb6bb400640 1 -- 172.21.15.98:0/4260846996 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 515934467 0 0) 0x7fb6bc0c1f60 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6c1000640 1 --2- 172.21.15.98:0/4260846996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc073570 0x7fb6bc1592d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40097c0 tx=0x7fb6b401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63af0224deb0fa8b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6bb400640 1 -- 172.21.15.98:0/4260846996 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b40b5070 con 0x7fb6bc073570 2024-05-27T00:16:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6bb400640 1 -- 172.21.15.98:0/4260846996 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6b40a8990 con 0x7fb6bc073570 2024-05-27T00:16:12.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6bb400640 1 -- 172.21.15.98:0/4260846996 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b40a8d60 con 0x7fb6bc073570 2024-05-27T00:16:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6c3000640 1 -- 172.21.15.98:0/4260846996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc073570 msgr2=0x7fb6bc1592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6c3000640 1 --2- 172.21.15.98:0/4260846996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc073570 0x7fb6bc1592d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40097c0 tx=0x7fb6b401ec90 comp rx=0 tx=0).stop 2024-05-27T00:16:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6c3000640 1 -- 172.21.15.98:0/4260846996 shutdown_connections 2024-05-27T00:16:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6c3000640 1 --2- 172.21.15.98:0/4260846996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc073570 0x7fb6bc1592d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6c3000640 1 -- 172.21.15.98:0/4260846996 >> 172.21.15.98:0/4260846996 conn(0x7fb6bc067530 msgr2=0x7fb6bc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6c3000640 1 -- 172.21.15.98:0/4260846996 shutdown_connections 2024-05-27T00:16:12.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.349+0000 7fb6c3000640 1 -- 172.21.15.98:0/4260846996 wait complete. 2024-05-27T00:16:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c3000640 1 Processor -- start 2024-05-27T00:16:12.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c3000640 1 -- start start 2024-05-27T00:16:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c3000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6bc073570 0x7fb6bc1deb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c3000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6bc0cbe10 0x7fb6bc14f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c3000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc0cc1f0 0x7fb6bc14e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c3000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc0c1330 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c3000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc0c1050 con 0x7fb6bc073570 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c3000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb6bc0c79d0 con 0x7fb6bc0cbe10 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c1000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc0cc1f0 0x7fb6bc14e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c1000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc0cc1f0 0x7fb6bc14e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50340/0 (socket says 172.21.15.98:50340) 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c1000640 1 -- 172.21.15.98:0/459342822 learned_addr learned my addr 172.21.15.98:0/459342822 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6b3400640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6bc0cbe10 0x7fb6bc14f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6bbe00640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6bc073570 0x7fb6bc1deb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c1000640 1 -- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6bc0cbe10 msgr2=0x7fb6bc14f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c1000640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6bc0cbe10 0x7fb6bc14f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c1000640 1 -- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6bc073570 msgr2=0x7fb6bc1deb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c1000640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6bc073570 0x7fb6bc1deb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c1000640 1 -- 172.21.15.98:0/459342822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc1e1450 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6bbe00640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6bc073570 0x7fb6bc1deb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:16:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c1000640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc0cc1f0 0x7fb6bc14e300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40b3040 tx=0x7fb6b401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6b8c00640 1 -- 172.21.15.98:0/459342822 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b40c0070 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6b8c00640 1 -- 172.21.15.98:0/459342822 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6b4004230 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6b8c00640 1 -- 172.21.15.98:0/459342822 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b40b5040 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c3000640 1 -- 172.21.15.98:0/459342822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6bc1f1820 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.353+0000 7fb6c3000640 1 -- 172.21.15.98:0/459342822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6bc1df9e0 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.357+0000 7fb6c3000640 1 -- 172.21.15.98:0/459342822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6bc075cb0 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.357+0000 7fb6b8c00640 1 -- 172.21.15.98:0/459342822 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb6b4003620 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.357+0000 7fb6b8c00640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb6980678c0 0x7fb698069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.357+0000 7fb6b8c00640 1 -- 172.21.15.98:0/459342822 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7fb6b4122860 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.357+0000 7fb6bbe00640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb6980678c0 0x7fb698069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:12.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.361+0000 7fb6bbe00640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb6980678c0 0x7fb698069d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac007fd0 tx=0x7fb6ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:12.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.361+0000 7fb6b8c00640 1 -- 172.21.15.98:0/459342822 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb6b40187f0 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.549+0000 7fb6c3000640 1 -- 172.21.15.98:0/459342822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6bc0b20d0 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.549+0000 7fb6b8c00640 1 -- 172.21.15.98:0/459342822 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7fb6b40f3290 con 0x7fb6bc0cc1f0 2024-05-27T00:16:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:12.553 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:16:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:16:12.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 -- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb6980678c0 msgr2=0x7fb698069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:12.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb6980678c0 0x7fb698069d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac007fd0 tx=0x7fb6ac009040 comp rx=0 tx=0).stop 2024-05-27T00:16:12.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 -- 172.21.15.98:0/459342822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc0cc1f0 msgr2=0x7fb6bc14e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:12.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc0cc1f0 0x7fb6bc14e300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40b3040 tx=0x7fb6b401ba30 comp rx=0 tx=0).stop 2024-05-27T00:16:12.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 -- 172.21.15.98:0/459342822 shutdown_connections 2024-05-27T00:16:12.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6bc0cc1f0 0x7fb6bc14e300 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6bc0cbe10 0x7fb6bc14f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb6980678c0 0x7fb698069d80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 --2- 172.21.15.98:0/459342822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6bc073570 0x7fb6bc1deb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 -- 172.21.15.98:0/459342822 >> 172.21.15.98:0/459342822 conn(0x7fb6bc067530 msgr2=0x7fb6bc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 -- 172.21.15.98:0/459342822 shutdown_connections 2024-05-27T00:16:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:12.553+0000 7fb6c3000640 1 -- 172.21.15.98:0/459342822 wait complete. 2024-05-27T00:16:12.574 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:16:13.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:16:13.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:16:14.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:14.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f191d000640 1 Processor -- start 2024-05-27T00:16:14.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f191d000640 1 -- start start 2024-05-27T00:16:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f191d000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19180731f0 0x7f19180c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f191d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19180735d0 0x7f1918151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f191d000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19180c17d0 con 0x7f19180c79f0 2024-05-27T00:16:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f191d000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1918071990 con 0x7f19180731f0 2024-05-27T00:16:14.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f191d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1918070e10 con 0x7f19180735d0 2024-05-27T00:16:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f1915e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19180731f0 0x7f19180c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f1915e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19180731f0 0x7f19180c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36988/0 (socket says 172.21.15.98:36988) 2024-05-27T00:16:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f1915e00640 1 -- 172.21.15.98:0/575732180 learned_addr learned my addr 172.21.15.98:0/575732180 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f190d400640 1 --2- 172.21.15.98:0/575732180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19180735d0 0x7f1918151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f1915e00640 1 -- 172.21.15.98:0/575732180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19180735d0 msgr2=0x7f1918151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f1915e00640 1 --2- 172.21.15.98:0/575732180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19180735d0 0x7f1918151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f1915e00640 1 -- 172.21.15.98:0/575732180 >> v1:172.21.15.204:6789/0 conn(0x7f19180c79f0 legacy=0x7f1918157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:14.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.721+0000 7f1915e00640 1 -- 172.21.15.98:0/575732180 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19180710f0 con 0x7f19180731f0 2024-05-27T00:16:14.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f1915e00640 1 --2- 172.21.15.98:0/575732180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180731f0 0x7f19180c0eb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1908009870 tx=0x7f190801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2749532a60c133bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:14.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f1915400640 1 -- 172.21.15.98:0/575732180 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19080b5070 con 0x7f19180731f0 2024-05-27T00:16:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f1915400640 1 -- 172.21.15.98:0/575732180 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19080a8a50 con 0x7f19180731f0 2024-05-27T00:16:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f1915400640 1 -- 172.21.15.98:0/575732180 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19080a8e20 con 0x7f19180731f0 2024-05-27T00:16:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f191d000640 1 -- 172.21.15.98:0/575732180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180731f0 msgr2=0x7f19180c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f191d000640 1 --2- 172.21.15.98:0/575732180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180731f0 0x7f19180c0eb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1908009870 tx=0x7f190801ed50 comp rx=0 tx=0).stop 2024-05-27T00:16:14.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f191d000640 1 -- 172.21.15.98:0/575732180 shutdown_connections 2024-05-27T00:16:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f191d000640 1 --2- 172.21.15.98:0/575732180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19180735d0 0x7f1918151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f191d000640 1 --2- 172.21.15.98:0/575732180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180731f0 0x7f19180c0eb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f191d000640 1 -- 172.21.15.98:0/575732180 >> 172.21.15.98:0/575732180 conn(0x7f1918067530 msgr2=0x7f191806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f191d000640 1 -- 172.21.15.98:0/575732180 shutdown_connections 2024-05-27T00:16:14.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.725+0000 7f191d000640 1 -- 172.21.15.98:0/575732180 wait complete. 2024-05-27T00:16:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f191d000640 1 Processor -- start 2024-05-27T00:16:14.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f191d000640 1 -- start start 2024-05-27T00:16:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f191d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19180731f0 0x7f19181f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f191d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19180735d0 0x7f1918144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f191d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180c79f0 0x7f191815b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f191d000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19180c8400 con 0x7f19180c79f0 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f191d000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f19180716b0 con 0x7f19180735d0 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f191d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f19180c13f0 con 0x7f19180731f0 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f1916800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180c79f0 0x7f191815b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f1916800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180c79f0 0x7f191815b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37000/0 (socket says 172.21.15.98:37000) 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f1916800640 1 -- 172.21.15.98:0/3985167498 learned_addr learned my addr 172.21.15.98:0/3985167498 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f190d400640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19180735d0 0x7f1918144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f1915e00640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19180731f0 0x7f19181f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f1916800640 1 -- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19180731f0 msgr2=0x7f19181f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f1916800640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19180731f0 0x7f19181f18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f1916800640 1 -- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19180735d0 msgr2=0x7f1918144d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f1916800640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19180735d0 0x7f1918144d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f1916800640 1 -- 172.21.15.98:0/3985167498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f191815b840 con 0x7f19180c79f0 2024-05-27T00:16:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f1916800640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180c79f0 0x7f191815b0e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f191000c730 tx=0x7f191000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f190ea00640 1 -- 172.21.15.98:0/3985167498 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910003b80 con 0x7f19180c79f0 2024-05-27T00:16:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f190ea00640 1 -- 172.21.15.98:0/3985167498 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1910003d20 con 0x7f19180c79f0 2024-05-27T00:16:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f191d000640 1 -- 172.21.15.98:0/3985167498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19181588f0 con 0x7f19180c79f0 2024-05-27T00:16:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f190ea00640 1 -- 172.21.15.98:0/3985167498 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191000aa30 con 0x7f19180c79f0 2024-05-27T00:16:14.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.729+0000 7f191d000640 1 -- 172.21.15.98:0/3985167498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1918158eb0 con 0x7f19180c79f0 2024-05-27T00:16:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.733+0000 7f191d000640 1 -- 172.21.15.98:0/3985167498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1918075cb0 con 0x7f19180c79f0 2024-05-27T00:16:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.733+0000 7f190ea00640 1 -- 172.21.15.98:0/3985167498 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f191000abd0 con 0x7f19180c79f0 2024-05-27T00:16:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.733+0000 7f190ea00640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f40678c0 0x7f18f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.733+0000 7f190ea00640 1 -- 172.21.15.98:0/3985167498 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f1910085b80 con 0x7f19180c79f0 2024-05-27T00:16:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.733+0000 7f1915e00640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f40678c0 0x7f18f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.733+0000 7f1915e00640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f40678c0 0x7f18f4069d80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f19080b2000 tx=0x7f1908008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.737+0000 7f190ea00640 1 -- 172.21.15.98:0/3985167498 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1910055ec0 con 0x7f19180c79f0 2024-05-27T00:16:14.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 -- 172.21.15.98:0/3985167498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1918158510 con 0x7f19180c79f0 2024-05-27T00:16:14.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f190ea00640 1 -- 172.21.15.98:0/3985167498 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f191005a1c0 con 0x7f19180c79f0 2024-05-27T00:16:14.932 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:14.932 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:16:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:16:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 -- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f40678c0 msgr2=0x7f18f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f40678c0 0x7f18f4069d80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f19080b2000 tx=0x7f1908008040 comp rx=0 tx=0).stop 2024-05-27T00:16:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 -- 172.21.15.98:0/3985167498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180c79f0 msgr2=0x7f191815b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180c79f0 0x7f191815b0e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f191000c730 tx=0x7f191000cbf0 comp rx=0 tx=0).stop 2024-05-27T00:16:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 -- 172.21.15.98:0/3985167498 shutdown_connections 2024-05-27T00:16:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19180c79f0 0x7f191815b0e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f40678c0 0x7f18f4069d80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19180735d0 0x7f1918144d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 --2- 172.21.15.98:0/3985167498 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19180731f0 0x7f19181f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 -- 172.21.15.98:0/3985167498 >> 172.21.15.98:0/3985167498 conn(0x7f1918067530 msgr2=0x7f1918156e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 -- 172.21.15.98:0/3985167498 shutdown_connections 2024-05-27T00:16:14.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:14.929+0000 7f191d000640 1 -- 172.21.15.98:0/3985167498 wait complete. 2024-05-27T00:16:14.953 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:16:16.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.097+0000 7faa54800640 1 Processor -- start 2024-05-27T00:16:17.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.097+0000 7faa54800640 1 -- start start 2024-05-27T00:16:17.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.097+0000 7faa54800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa4c0731f0 0x7faa4c0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa54800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7faa4c0735d0 0x7faa4c151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa54800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4c0c17d0 con 0x7faa4c0c79f0 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa54800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7faa4c071990 con 0x7faa4c0731f0 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa54800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7faa4c070e10 con 0x7faa4c0735d0 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa51e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa4c0731f0 0x7faa4c0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa51e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa4c0731f0 0x7faa4c0c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37016/0 (socket says 172.21.15.98:37016) 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa51e00640 1 -- 172.21.15.98:0/3969605115 learned_addr learned my addr 172.21.15.98:0/3969605115 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa51400640 1 --2- 172.21.15.98:0/3969605115 >> v2:172.21.15.204:3300/0 conn(0x7faa4c0735d0 0x7faa4c151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa51e00640 1 -- 172.21.15.98:0/3969605115 >> v2:172.21.15.204:3300/0 conn(0x7faa4c0735d0 msgr2=0x7faa4c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa51e00640 1 --2- 172.21.15.98:0/3969605115 >> v2:172.21.15.204:3300/0 conn(0x7faa4c0735d0 0x7faa4c151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa51e00640 1 -- 172.21.15.98:0/3969605115 >> v1:172.21.15.204:6789/0 conn(0x7faa4c0c79f0 legacy=0x7faa4c157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa51e00640 1 -- 172.21.15.98:0/3969605115 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c0710f0 con 0x7faa4c0731f0 2024-05-27T00:16:17.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa51e00640 1 --2- 172.21.15.98:0/3969605115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 0x7faa4c0c0eb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0c7980 tx=0x7faa4001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d497de302a9c346e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:17.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa50a00640 1 -- 172.21.15.98:0/3969605115 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa400aa610 con 0x7faa4c0731f0 2024-05-27T00:16:17.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa50a00640 1 -- 172.21.15.98:0/3969605115 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa400af070 con 0x7faa4c0731f0 2024-05-27T00:16:17.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.101+0000 7faa50a00640 1 -- 172.21.15.98:0/3969605115 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa400aa9d0 con 0x7faa4c0731f0 2024-05-27T00:16:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 -- 172.21.15.98:0/3969605115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 msgr2=0x7faa4c0c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 --2- 172.21.15.98:0/3969605115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 0x7faa4c0c0eb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0c7980 tx=0x7faa4001c4b0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 -- 172.21.15.98:0/3969605115 shutdown_connections 2024-05-27T00:16:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 --2- 172.21.15.98:0/3969605115 >> v2:172.21.15.204:3300/0 conn(0x7faa4c0735d0 0x7faa4c151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 --2- 172.21.15.98:0/3969605115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 0x7faa4c0c0eb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 -- 172.21.15.98:0/3969605115 >> 172.21.15.98:0/3969605115 conn(0x7faa4c067530 msgr2=0x7faa4c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 -- 172.21.15.98:0/3969605115 shutdown_connections 2024-05-27T00:16:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 -- 172.21.15.98:0/3969605115 wait complete. 2024-05-27T00:16:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 Processor -- start 2024-05-27T00:16:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 -- start start 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 0x7faa4c1f3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa4c0735d0 0x7faa4c1408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa4c0c79f0 0x7faa4c13d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0c13f0 con 0x7faa4c0731f0 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c151d80 con 0x7faa4c0735d0 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa54800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faa4c0c1570 con 0x7faa4c0c79f0 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa51e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 0x7faa4c1f3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa51e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 0x7faa4c1f3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37022/0 (socket says 172.21.15.98:37022) 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa51e00640 1 -- 172.21.15.98:0/3521467231 learned_addr learned my addr 172.21.15.98:0/3521467231 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa51400640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa4c0735d0 0x7faa4c1408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa52800640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa4c0c79f0 0x7faa4c13d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:17.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa51e00640 1 -- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa4c0c79f0 msgr2=0x7faa4c13d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:17.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa51e00640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa4c0c79f0 0x7faa4c13d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa51e00640 1 -- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa4c0735d0 msgr2=0x7faa4c1408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:17.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa51e00640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa4c0735d0 0x7faa4c1408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.105+0000 7faa51e00640 1 -- 172.21.15.98:0/3521467231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c140340 con 0x7faa4c0731f0 2024-05-27T00:16:17.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa51e00640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 0x7faa4c1f3260 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faa400ac040 tx=0x7faa4001acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:17.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa46000640 1 -- 172.21.15.98:0/3521467231 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa400bd5f0 con 0x7faa4c0731f0 2024-05-27T00:16:17.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa46000640 1 -- 172.21.15.98:0/3521467231 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa400bd790 con 0x7faa4c0731f0 2024-05-27T00:16:17.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa54800640 1 -- 172.21.15.98:0/3521467231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c141990 con 0x7faa4c0731f0 2024-05-27T00:16:17.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa46000640 1 -- 172.21.15.98:0/3521467231 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa400bdb60 con 0x7faa4c0731f0 2024-05-27T00:16:17.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa54800640 1 -- 172.21.15.98:0/3521467231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c141f50 con 0x7faa4c0731f0 2024-05-27T00:16:17.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa54800640 1 -- 172.21.15.98:0/3521467231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c075cb0 con 0x7faa4c0731f0 2024-05-27T00:16:17.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa46000640 1 -- 172.21.15.98:0/3521467231 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faa400b33f0 con 0x7faa4c0731f0 2024-05-27T00:16:17.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa46000640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa300678c0 0x7faa30069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:17.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa46000640 1 -- 172.21.15.98:0/3521467231 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7faa40120ba0 con 0x7faa4c0731f0 2024-05-27T00:16:17.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.109+0000 7faa51400640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa300678c0 0x7faa30069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:17.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.113+0000 7faa51400640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa300678c0 0x7faa30069d80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faa4c14b5d0 tx=0x7faa3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:17.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.113+0000 7faa46000640 1 -- 172.21.15.98:0/3521467231 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7faa400c3060 con 0x7faa4c0731f0 2024-05-27T00:16:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.301+0000 7faa54800640 1 -- 172.21.15.98:0/3521467231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa4c06f2e0 con 0x7faa4c0731f0 2024-05-27T00:16:17.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa46000640 1 -- 172.21.15.98:0/3521467231 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7faa4c140340 con 0x7faa4c0731f0 2024-05-27T00:16:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:17.307 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:16:17.307 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:16:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 -- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa300678c0 msgr2=0x7faa30069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa300678c0 0x7faa30069d80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faa4c14b5d0 tx=0x7faa3c008040 comp rx=0 tx=0).stop 2024-05-27T00:16:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 -- 172.21.15.98:0/3521467231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 msgr2=0x7faa4c1f3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 0x7faa4c1f3260 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faa400ac040 tx=0x7faa4001acf0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 -- 172.21.15.98:0/3521467231 shutdown_connections 2024-05-27T00:16:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa4c0c79f0 0x7faa4c13d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa300678c0 0x7faa30069d80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa4c0735d0 0x7faa4c1408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 --2- 172.21.15.98:0/3521467231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c0731f0 0x7faa4c1f3260 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:17.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 -- 172.21.15.98:0/3521467231 >> 172.21.15.98:0/3521467231 conn(0x7faa4c067530 msgr2=0x7faa4c0cbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 -- 172.21.15.98:0/3521467231 shutdown_connections 2024-05-27T00:16:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:17.305+0000 7faa54800640 1 -- 172.21.15.98:0/3521467231 wait complete. 2024-05-27T00:16:17.328 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:16:19.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68e7c00640 1 Processor -- start 2024-05-27T00:16:19.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68e7c00640 1 -- start start 2024-05-27T00:16:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68e7c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00731f0 0x7f68e00c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68e7c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f68e00735d0 0x7f68e0151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68e7c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68e00c17d0 con 0x7f68e00c79f0 2024-05-27T00:16:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68e7c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f68e0071990 con 0x7f68e00735d0 2024-05-27T00:16:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68e7c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f68e0070e10 con 0x7f68e00731f0 2024-05-27T00:16:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68e5c00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f68e00c79f0 0x7f68e0157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48792/0 (socket says 172.21.15.98:48792) 2024-05-27T00:16:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68e5c00640 1 -- 172.21.15.98:0/3590872163 learned_addr learned my addr 172.21.15.98:0/3590872163 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68dfe00640 1 --2- 172.21.15.98:0/3590872163 >> v2:172.21.15.204:3300/0 conn(0x7f68e00735d0 0x7f68e0151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:19.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.473+0000 7f68e5200640 1 --2- 172.21.15.98:0/3590872163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00731f0 0x7f68e00c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.477+0000 7f68df400640 1 -- 172.21.15.98:0/3590872163 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3729855830 0 0) 0x7f68e00c17d0 con 0x7f68e00c79f0 2024-05-27T00:16:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.477+0000 7f68df400640 1 -- 172.21.15.98:0/3590872163 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68c4003530 con 0x7f68e00c79f0 2024-05-27T00:16:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.477+0000 7f68e5200640 1 -- 172.21.15.98:0/3590872163 >> v2:172.21.15.204:3300/0 conn(0x7f68e00735d0 msgr2=0x7f68e0151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.477+0000 7f68e5200640 1 --2- 172.21.15.98:0/3590872163 >> v2:172.21.15.204:3300/0 conn(0x7f68e00735d0 0x7f68e0151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.477+0000 7f68df400640 1 -- 172.21.15.98:0/3590872163 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1342183932 0 0) 0x7f68c4003530 con 0x7f68e00c79f0 2024-05-27T00:16:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.477+0000 7f68e5200640 1 -- 172.21.15.98:0/3590872163 >> v1:172.21.15.98:6789/0 conn(0x7f68e00c79f0 legacy=0x7f68e0157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:19.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.477+0000 7f68e5200640 1 -- 172.21.15.98:0/3590872163 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e00710f0 con 0x7f68e00731f0 2024-05-27T00:16:19.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.477+0000 7f68e5200640 1 --2- 172.21.15.98:0/3590872163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00731f0 0x7f68e00c0eb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68d00097c0 tx=0x7f68d001ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b757d2acf0343643 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:19.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.477+0000 7f68df400640 1 -- 172.21.15.98:0/3590872163 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d00b5070 con 0x7f68e00731f0 2024-05-27T00:16:19.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68df400640 1 -- 172.21.15.98:0/3590872163 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68d00a85f0 con 0x7f68e00731f0 2024-05-27T00:16:19.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68df400640 1 -- 172.21.15.98:0/3590872163 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d8001a00 con 0x7f68e00731f0 2024-05-27T00:16:19.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 -- 172.21.15.98:0/3590872163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00731f0 msgr2=0x7f68e00c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:19.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 --2- 172.21.15.98:0/3590872163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00731f0 0x7f68e00c0eb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68d00097c0 tx=0x7f68d001ebb0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 -- 172.21.15.98:0/3590872163 shutdown_connections 2024-05-27T00:16:19.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 --2- 172.21.15.98:0/3590872163 >> v2:172.21.15.204:3300/0 conn(0x7f68e00735d0 0x7f68e0151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 --2- 172.21.15.98:0/3590872163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00731f0 0x7f68e00c0eb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 -- 172.21.15.98:0/3590872163 >> 172.21.15.98:0/3590872163 conn(0x7f68e0067530 msgr2=0x7f68e006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:19.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 -- 172.21.15.98:0/3590872163 shutdown_connections 2024-05-27T00:16:19.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 -- 172.21.15.98:0/3590872163 wait complete. 2024-05-27T00:16:19.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 Processor -- start 2024-05-27T00:16:19.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 -- start start 2024-05-27T00:16:19.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68e00731f0 0x7f68e0146750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:19.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e00735d0 0x7f68e015a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00c79f0 0x7f68e015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68e00c8400 con 0x7f68e00735d0 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f68e00716b0 con 0x7f68e00731f0 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e7c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f68e00c13f0 con 0x7f68e00c79f0 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68dfe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e00735d0 0x7f68e015a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68dfe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e00735d0 0x7f68e015a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37028/0 (socket says 172.21.15.98:37028) 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68dfe00640 1 -- 172.21.15.98:0/2948779228 learned_addr learned my addr 172.21.15.98:0/2948779228 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e5200640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68e00731f0 0x7f68e0146750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.481+0000 7f68e5c00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00c79f0 0x7f68e015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dfe00640 1 -- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00c79f0 msgr2=0x7f68e015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dfe00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00c79f0 0x7f68e015ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dfe00640 1 -- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68e00731f0 msgr2=0x7f68e0146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:19.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dfe00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68e00731f0 0x7f68e0146750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dfe00640 1 -- 172.21.15.98:0/2948779228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e01f0d10 con 0x7f68e00735d0 2024-05-27T00:16:19.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68e5c00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00c79f0 0x7f68e015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:16:19.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dfe00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e00735d0 0x7f68e015a910 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00e900 tx=0x7f68cc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:19.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dcc00640 1 -- 172.21.15.98:0/2948779228 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc017070 con 0x7f68e00735d0 2024-05-27T00:16:19.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dcc00640 1 -- 172.21.15.98:0/2948779228 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68cc00ca00 con 0x7f68e00735d0 2024-05-27T00:16:19.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dcc00640 1 -- 172.21.15.98:0/2948779228 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc00cdd0 con 0x7f68e00735d0 2024-05-27T00:16:19.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68e7c00640 1 -- 172.21.15.98:0/2948779228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e01f0fb0 con 0x7f68e00735d0 2024-05-27T00:16:19.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68e7c00640 1 -- 172.21.15.98:0/2948779228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e0159e30 con 0x7f68e00735d0 2024-05-27T00:16:19.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68e7c00640 1 -- 172.21.15.98:0/2948779228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e0075cb0 con 0x7f68e00735d0 2024-05-27T00:16:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dcc00640 1 -- 172.21.15.98:0/2948779228 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f68cc01a070 con 0x7f68e00735d0 2024-05-27T00:16:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dcc00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68c00678f0 0x7f68c0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68dcc00640 1 -- 172.21.15.98:0/2948779228 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f68cc085cf0 con 0x7f68e00735d0 2024-05-27T00:16:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.485+0000 7f68e5200640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68c00678f0 0x7f68c0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.489+0000 7f68e5200640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68c00678f0 0x7f68c0069db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f68d00b2000 tx=0x7f68d00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:19.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.489+0000 7f68dcc00640 1 -- 172.21.15.98:0/2948779228 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f68cc056150 con 0x7f68e00735d0 2024-05-27T00:16:19.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.681+0000 7f68e7c00640 1 -- 172.21.15.98:0/2948779228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68e0150910 con 0x7f68e00735d0 2024-05-27T00:16:19.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.681+0000 7f68dcc00640 1 -- 172.21.15.98:0/2948779228 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f68cc05a450 con 0x7f68e00735d0 2024-05-27T00:16:19.685 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:19.685 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:16:19.685 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:16:19.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 -- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68c00678f0 msgr2=0x7f68c0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68c00678f0 0x7f68c0069db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f68d00b2000 tx=0x7f68d00aa070 comp rx=0 tx=0).stop 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 -- 172.21.15.98:0/2948779228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e00735d0 msgr2=0x7f68e015a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e00735d0 0x7f68e015a910 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00e900 tx=0x7f68cc00edc0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 -- 172.21.15.98:0/2948779228 shutdown_connections 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68e00c79f0 0x7f68e015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68e00735d0 0x7f68e015a910 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68c00678f0 0x7f68c0069db0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 --2- 172.21.15.98:0/2948779228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68e00731f0 0x7f68e0146750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 -- 172.21.15.98:0/2948779228 >> 172.21.15.98:0/2948779228 conn(0x7f68e0067530 msgr2=0x7f68e0151200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 -- 172.21.15.98:0/2948779228 shutdown_connections 2024-05-27T00:16:19.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:19.685+0000 7f68e7c00640 1 -- 172.21.15.98:0/2948779228 wait complete. 2024-05-27T00:16:19.706 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:16:21.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.857+0000 7f51aa200640 1 Processor -- start 2024-05-27T00:16:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.857+0000 7f51aa200640 1 -- start start 2024-05-27T00:16:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51aa200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f51a41526f0 0x7f51a40c8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51aa200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f51a4073040 0x7f51a4067570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51aa200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a40b3130 con 0x7f51a4067130 2024-05-27T00:16:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51aa200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f51a4158780 con 0x7f51a4073040 2024-05-27T00:16:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51aa200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f51a4158a60 con 0x7f51a41526f0 2024-05-27T00:16:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51a3e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f51a4067130 0x7f51a4072930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48806/0 (socket says 172.21.15.98:48806) 2024-05-27T00:16:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51a3e00640 1 -- 172.21.15.98:0/2243251126 learned_addr learned my addr 172.21.15.98:0/2243251126 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:21.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51a8e00640 1 --2- 172.21.15.98:0/2243251126 >> v2:172.21.15.204:3300/0 conn(0x7f51a41526f0 0x7f51a40c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51a9800640 1 --2- 172.21.15.98:0/2243251126 >> v2:172.21.15.98:3300/0 conn(0x7f51a4073040 0x7f51a4067570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51a3400640 1 -- 172.21.15.98:0/2243251126 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2392330783 0 0) 0x7f5190001120 con 0x7f51a4067130 2024-05-27T00:16:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51a3400640 1 -- 172.21.15.98:0/2243251126 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f518c003530 con 0x7f51a4067130 2024-05-27T00:16:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51a9800640 1 -- 172.21.15.98:0/2243251126 >> v2:172.21.15.204:3300/0 conn(0x7f51a41526f0 msgr2=0x7f51a40c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51a9800640 1 --2- 172.21.15.98:0/2243251126 >> v2:172.21.15.204:3300/0 conn(0x7f51a41526f0 0x7f51a40c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51a9800640 1 -- 172.21.15.98:0/2243251126 >> v1:172.21.15.98:6789/0 conn(0x7f51a4067130 legacy=0x7f51a4072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.861+0000 7f51a9800640 1 -- 172.21.15.98:0/2243251126 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a4158d40 con 0x7f51a4073040 2024-05-27T00:16:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51a9800640 1 --2- 172.21.15.98:0/2243251126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 0x7f51a4067570 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f51a40681f0 tx=0x7f519c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a635505f53b0fa4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51a3400640 1 -- 172.21.15.98:0/2243251126 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c0b2070 con 0x7f51a4073040 2024-05-27T00:16:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51a3400640 1 -- 172.21.15.98:0/2243251126 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f519c0a5760 con 0x7f51a4073040 2024-05-27T00:16:21.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51a3400640 1 -- 172.21.15.98:0/2243251126 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c0a5b30 con 0x7f51a4073040 2024-05-27T00:16:21.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51aa200640 1 -- 172.21.15.98:0/2243251126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 msgr2=0x7f51a4067570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:21.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51aa200640 1 --2- 172.21.15.98:0/2243251126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 0x7f51a4067570 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f51a40681f0 tx=0x7f519c01c8a0 comp rx=0 tx=0).stop 2024-05-27T00:16:21.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51aa200640 1 -- 172.21.15.98:0/2243251126 shutdown_connections 2024-05-27T00:16:21.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51aa200640 1 --2- 172.21.15.98:0/2243251126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 0x7f51a4067570 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:21.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51aa200640 1 --2- 172.21.15.98:0/2243251126 >> v2:172.21.15.204:3300/0 conn(0x7f51a41526f0 0x7f51a40c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:21.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51aa200640 1 -- 172.21.15.98:0/2243251126 >> 172.21.15.98:0/2243251126 conn(0x7f51a40c9af0 msgr2=0x7f51a40c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:21.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51aa200640 1 -- 172.21.15.98:0/2243251126 shutdown_connections 2024-05-27T00:16:21.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51aa200640 1 -- 172.21.15.98:0/2243251126 wait complete. 2024-05-27T00:16:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51aa200640 1 Processor -- start 2024-05-27T00:16:21.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.865+0000 7f51aa200640 1 -- start start 2024-05-27T00:16:21.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51aa200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51a4067130 0x7f51a406b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:21.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51aa200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 0x7f51a414ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:21.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51aa200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51a41526f0 0x7f51a414e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:21.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51aa200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51a4067ff0 con 0x7f51a4073040 2024-05-27T00:16:21.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51aa200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f51a4067e70 con 0x7f51a4067130 2024-05-27T00:16:21.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51aa200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f51a40c96d0 con 0x7f51a41526f0 2024-05-27T00:16:21.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 0x7f51a414ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 0x7f51a414ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37046/0 (socket says 172.21.15.98:37046) 2024-05-27T00:16:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a3e00640 1 -- 172.21.15.98:0/2678183913 learned_addr learned my addr 172.21.15.98:0/2678183913 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a8e00640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51a4067130 0x7f51a406b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a9800640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51a41526f0 0x7f51a414e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a3e00640 1 -- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51a41526f0 msgr2=0x7f51a414e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a3e00640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51a41526f0 0x7f51a414e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a3e00640 1 -- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51a4067130 msgr2=0x7f51a406b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a3e00640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51a4067130 0x7f51a406b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a3e00640 1 -- 172.21.15.98:0/2678183913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a415da90 con 0x7f51a4073040 2024-05-27T00:16:21.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a3e00640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 0x7f51a414ddf0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f519000c730 tx=0x7f519000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a0c00640 1 -- 172.21.15.98:0/2678183913 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190017070 con 0x7f51a4073040 2024-05-27T00:16:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a0c00640 1 -- 172.21.15.98:0/2678183913 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5190005ae0 con 0x7f51a4073040 2024-05-27T00:16:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a0c00640 1 -- 172.21.15.98:0/2678183913 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190007e30 con 0x7f51a4073040 2024-05-27T00:16:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51aa200640 1 -- 172.21.15.98:0/2678183913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a4152c60 con 0x7f51a4073040 2024-05-27T00:16:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51aa200640 1 -- 172.21.15.98:0/2678183913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a4153220 con 0x7f51a4073040 2024-05-27T00:16:21.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51aa200640 1 -- 172.21.15.98:0/2678183913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a4075c40 con 0x7f51a4073040 2024-05-27T00:16:21.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.869+0000 7f51a0c00640 1 -- 172.21.15.98:0/2678183913 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f519001a070 con 0x7f51a4073040 2024-05-27T00:16:21.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.873+0000 7f51a0c00640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f51840678c0 0x7f5184069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:21.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.873+0000 7f51a0c00640 1 -- 172.21.15.98:0/2678183913 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f5190084ff0 con 0x7f51a4073040 2024-05-27T00:16:21.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.873+0000 7f51a8e00640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f51840678c0 0x7f5184069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:21.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.873+0000 7f51a8e00640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f51840678c0 0x7f5184069d80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5198009ac0 tx=0x7f5198009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:21.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:21.877+0000 7f51a0c00640 1 -- 172.21.15.98:0/2678183913 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f51a415da90 con 0x7f51a4073040 2024-05-27T00:16:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.077+0000 7f51aa200640 1 -- 172.21.15.98:0/2678183913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51a406cc10 con 0x7f51a4073040 2024-05-27T00:16:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.077+0000 7f51a0c00640 1 -- 172.21.15.98:0/2678183913 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f51a40b3130 con 0x7f51a4073040 2024-05-27T00:16:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:22.081 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:16:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:16:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 -- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f51840678c0 msgr2=0x7f5184069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f51840678c0 0x7f5184069d80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5198009ac0 tx=0x7f5198009190 comp rx=0 tx=0).stop 2024-05-27T00:16:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 -- 172.21.15.98:0/2678183913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 msgr2=0x7f51a414ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 0x7f51a414ddf0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f519000c730 tx=0x7f519000cbf0 comp rx=0 tx=0).stop 2024-05-27T00:16:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 -- 172.21.15.98:0/2678183913 shutdown_connections 2024-05-27T00:16:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f51a41526f0 0x7f51a414e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51a4073040 0x7f51a414ddf0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:22.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f51840678c0 0x7f5184069d80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:22.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 --2- 172.21.15.98:0/2678183913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51a4067130 0x7f51a406b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:22.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 -- 172.21.15.98:0/2678183913 >> 172.21.15.98:0/2678183913 conn(0x7f51a40c9af0 msgr2=0x7f51a40cc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:22.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 -- 172.21.15.98:0/2678183913 shutdown_connections 2024-05-27T00:16:22.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:22.081+0000 7f51aa200640 1 -- 172.21.15.98:0/2678183913 wait complete. 2024-05-27T00:16:22.104 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:16:24.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:24.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9da6800640 1 Processor -- start 2024-05-27T00:16:24.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9da6800640 1 -- start start 2024-05-27T00:16:24.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9da6800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9da00cb7e0 0x7f9da0158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:24.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9da6800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00b3ed0 0x7f9da0158c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:24.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9da6800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9da015be40 con 0x7f9da0152a50 2024-05-27T00:16:24.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9da6800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9da00c8a00 con 0x7f9da00cb7e0 2024-05-27T00:16:24.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9da6800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9da00c50e0 con 0x7f9da00b3ed0 2024-05-27T00:16:24.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9d97e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9da00cb7e0 0x7f9da0158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:24.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9d9fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00b3ed0 0x7f9da0158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:24.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9d97e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9da00cb7e0 0x7f9da0158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:39128/0 (socket says 172.21.15.98:39128) 2024-05-27T00:16:24.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9d9fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00b3ed0 0x7f9da0158c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:51530/0 (socket says 172.21.15.98:51530) 2024-05-27T00:16:24.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9d97e00640 1 -- 172.21.15.98:0/2807920322 learned_addr learned my addr 172.21.15.98:0/2807920322 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:24.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.245+0000 7f9d9f400640 1 --1- 172.21.15.98:0/2807920322 >> v1:172.21.15.204:6789/0 conn(0x7f9da0152a50 0x7f9da00709a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:47430/0 (socket says 172.21.15.98:47430) 2024-05-27T00:16:24.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.249+0000 7f9d97e00640 1 -- 172.21.15.98:0/2807920322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00b3ed0 msgr2=0x7f9da0158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:24.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.249+0000 7f9d97e00640 1 --2- 172.21.15.98:0/2807920322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00b3ed0 0x7f9da0158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:24.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.249+0000 7f9d97e00640 1 -- 172.21.15.98:0/2807920322 >> v1:172.21.15.204:6789/0 conn(0x7f9da0152a50 legacy=0x7f9da00709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:24.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.249+0000 7f9d97e00640 1 -- 172.21.15.98:0/2807920322 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da00c8c50 con 0x7f9da00cb7e0 2024-05-27T00:16:24.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.249+0000 7f9d97e00640 1 --2- 172.21.15.98:0/2807920322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9da00cb7e0 0x7f9da0158750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009870 tx=0x7f9d8801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=623e3013b8958d27 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:24.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.249+0000 7f9d9ea00640 1 -- 172.21.15.98:0/2807920322 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d880aa600 con 0x7f9da00cb7e0 2024-05-27T00:16:24.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.249+0000 7f9d9ea00640 1 -- 172.21.15.98:0/2807920322 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d880af070 con 0x7f9da00cb7e0 2024-05-27T00:16:24.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.249+0000 7f9d9ea00640 1 -- 172.21.15.98:0/2807920322 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d880aa9c0 con 0x7f9da00cb7e0 2024-05-27T00:16:24.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.249+0000 7f9da6800640 1 -- 172.21.15.98:0/2807920322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9da00cb7e0 msgr2=0x7f9da0158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:24.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.249+0000 7f9da6800640 1 --2- 172.21.15.98:0/2807920322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9da00cb7e0 0x7f9da0158750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009870 tx=0x7f9d8801c4a0 comp rx=0 tx=0).stop 2024-05-27T00:16:24.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 -- 172.21.15.98:0/2807920322 shutdown_connections 2024-05-27T00:16:24.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 --2- 172.21.15.98:0/2807920322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00b3ed0 0x7f9da0158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:24.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 --2- 172.21.15.98:0/2807920322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9da00cb7e0 0x7f9da0158750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:24.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 -- 172.21.15.98:0/2807920322 >> 172.21.15.98:0/2807920322 conn(0x7f9da0072f90 msgr2=0x7f9da00733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:24.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 -- 172.21.15.98:0/2807920322 shutdown_connections 2024-05-27T00:16:24.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 -- 172.21.15.98:0/2807920322 wait complete. 2024-05-27T00:16:24.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 Processor -- start 2024-05-27T00:16:24.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 -- start start 2024-05-27T00:16:24.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9da00b3ed0 0x7f9da00cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:24.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00cb7e0 0x7f9da0151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:24.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da0152a50 0x7f9da01f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:24.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9da0152e30 con 0x7f9da0152a50 2024-05-27T00:16:24.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9da00cb570 con 0x7f9da00b3ed0 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9da6800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9da00c9c10 con 0x7f9da00cb7e0 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d9fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da0152a50 0x7f9da01f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d9fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da0152a50 0x7f9da01f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33280/0 (socket says 172.21.15.98:33280) 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d9f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9da00b3ed0 0x7f9da00cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d9fe00640 1 -- 172.21.15.98:0/577053805 learned_addr learned my addr 172.21.15.98:0/577053805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d97e00640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00cb7e0 0x7f9da0151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d9fe00640 1 -- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00cb7e0 msgr2=0x7f9da0151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d9fe00640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00cb7e0 0x7f9da0151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d9fe00640 1 -- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9da00b3ed0 msgr2=0x7f9da00cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d9fe00640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9da00b3ed0 0x7f9da00cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:24.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d9fe00640 1 -- 172.21.15.98:0/577053805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da00ce1e0 con 0x7f9da0152a50 2024-05-27T00:16:24.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.253+0000 7f9d9fe00640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da0152a50 0x7f9da01f6560 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9d9800efc0 tx=0x7f9d9800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:24.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.257+0000 7f9d97400640 1 -- 172.21.15.98:0/577053805 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d98018020 con 0x7f9da0152a50 2024-05-27T00:16:24.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.257+0000 7f9d97400640 1 -- 172.21.15.98:0/577053805 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d98014480 con 0x7f9da0152a50 2024-05-27T00:16:24.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.257+0000 7f9d97400640 1 -- 172.21.15.98:0/577053805 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d98014870 con 0x7f9da0152a50 2024-05-27T00:16:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.257+0000 7f9da6800640 1 -- 172.21.15.98:0/577053805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9da00ce4c0 con 0x7f9da0152a50 2024-05-27T00:16:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.257+0000 7f9da6800640 1 -- 172.21.15.98:0/577053805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9da00c7220 con 0x7f9da0152a50 2024-05-27T00:16:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.257+0000 7f9da6800640 1 -- 172.21.15.98:0/577053805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9da0075cb0 con 0x7f9da0152a50 2024-05-27T00:16:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.257+0000 7f9d97400640 1 -- 172.21.15.98:0/577053805 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9d9801b020 con 0x7f9da0152a50 2024-05-27T00:16:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.257+0000 7f9d97400640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9d7c0678c0 0x7f9d7c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.257+0000 7f9d97400640 1 -- 172.21.15.98:0/577053805 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f9d98086af0 con 0x7f9da0152a50 2024-05-27T00:16:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.257+0000 7f9d9f400640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9d7c0678c0 0x7f9d7c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:24.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.261+0000 7f9d9f400640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9d7c0678c0 0x7f9d7c069d80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9d90000f70 tx=0x7f9d90007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:24.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.265+0000 7f9d97400640 1 -- 172.21.15.98:0/577053805 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9d98022030 con 0x7f9da0152a50 2024-05-27T00:16:24.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.461+0000 7f9da6800640 1 -- 172.21.15.98:0/577053805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9da0156840 con 0x7f9da0152a50 2024-05-27T00:16:24.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.461+0000 7f9d97400640 1 -- 172.21.15.98:0/577053805 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6833 (secure 0 0 0) 0x7f9d98010040 con 0x7f9da0152a50 2024-05-27T00:16:24.465 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:24.466 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:15:31.625115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.98:6837/3824013488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3824013488},{"type":"v1","addr":"172.21.15.98:6837","nonce":3824013488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4290,"qdb_cluster":[4290]},"id":1}]} 2024-05-27T00:16:24.466 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-05-27T00:16:24.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 -- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9d7c0678c0 msgr2=0x7f9d7c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:24.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9d7c0678c0 0x7f9d7c069d80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9d90000f70 tx=0x7f9d90007e10 comp rx=0 tx=0).stop 2024-05-27T00:16:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 -- 172.21.15.98:0/577053805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da0152a50 msgr2=0x7f9da01f6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da0152a50 0x7f9da01f6560 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9d9800efc0 tx=0x7f9d9800c490 comp rx=0 tx=0).stop 2024-05-27T00:16:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 -- 172.21.15.98:0/577053805 shutdown_connections 2024-05-27T00:16:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9da0152a50 0x7f9da01f6560 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9da00cb7e0 0x7f9da0151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9d7c0678c0 0x7f9d7c069d80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 --2- 172.21.15.98:0/577053805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9da00b3ed0 0x7f9da00cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 -- 172.21.15.98:0/577053805 >> 172.21.15.98:0/577053805 conn(0x7f9da0072f90 msgr2=0x7f9da0162730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 -- 172.21.15.98:0/577053805 shutdown_connections 2024-05-27T00:16:24.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:24.465+0000 7f9da6800640 1 -- 172.21.15.98:0/577053805 wait complete. 2024-05-27T00:16:24.486 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:16:26.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:26.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506fe00640 1 Processor -- start 2024-05-27T00:16:26.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506fe00640 1 -- start start 2024-05-27T00:16:26.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f50701548e0 0x7f507015b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:26.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506fe00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5070072ad0 0x7f507006b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:26.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506fe00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f507015e720 con 0x7f507015dd50 2024-05-27T00:16:26.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506fe00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f507015ece0 con 0x7f50701548e0 2024-05-27T00:16:26.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506fe00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f507015ea00 con 0x7f5070072ad0 2024-05-27T00:16:26.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506ea00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f50701548e0 0x7f507015b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:26.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506f400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f507015dd50 0x7f5070067130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48750/0 (socket says 172.21.15.98:48750) 2024-05-27T00:16:26.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506f400640 1 -- 172.21.15.98:0/1080995314 learned_addr learned my addr 172.21.15.98:0/1080995314 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:26.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f5067e00640 1 --2- 172.21.15.98:0/1080995314 >> v2:172.21.15.204:3300/0 conn(0x7f5070072ad0 0x7f507006b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:26.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506ea00640 1 -- 172.21.15.98:0/1080995314 >> v2:172.21.15.204:3300/0 conn(0x7f5070072ad0 msgr2=0x7f507006b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:26.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506ea00640 1 --2- 172.21.15.98:0/1080995314 >> v2:172.21.15.204:3300/0 conn(0x7f5070072ad0 0x7f507006b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:26.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506ea00640 1 -- 172.21.15.98:0/1080995314 >> v1:172.21.15.98:6789/0 conn(0x7f507015dd50 legacy=0x7f5070067130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:26.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506ea00640 1 -- 172.21.15.98:0/1080995314 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507015a900 con 0x7f50701548e0 2024-05-27T00:16:26.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f506e000640 1 -- 172.21.15.98:0/1080995314 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1709652465 0 0) 0x7f507015e720 con 0x7f507015dd50 2024-05-27T00:16:26.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.637+0000 7f5067e00640 1 --2- 172.21.15.98:0/1080995314 >> v2:172.21.15.204:3300/0 conn(0x7f5070072ad0 0x7f507006b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.641+0000 7f506ea00640 1 --2- 172.21.15.98:0/1080995314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50701548e0 0x7f507015b3e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5070071870 tx=0x7f506001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d85cef681971bb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506e000640 1 -- 172.21.15.98:0/1080995314 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50600b5070 con 0x7f50701548e0 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506e000640 1 -- 172.21.15.98:0/1080995314 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50600a8970 con 0x7f50701548e0 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506e000640 1 -- 172.21.15.98:0/1080995314 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50600a8d40 con 0x7f50701548e0 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 -- 172.21.15.98:0/1080995314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50701548e0 msgr2=0x7f507015b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 --2- 172.21.15.98:0/1080995314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50701548e0 0x7f507015b3e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5070071870 tx=0x7f506001ec70 comp rx=0 tx=0).stop 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 -- 172.21.15.98:0/1080995314 shutdown_connections 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 --2- 172.21.15.98:0/1080995314 >> v2:172.21.15.204:3300/0 conn(0x7f5070072ad0 0x7f507006b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 --2- 172.21.15.98:0/1080995314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50701548e0 0x7f507015b3e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 -- 172.21.15.98:0/1080995314 >> 172.21.15.98:0/1080995314 conn(0x7f50700c9af0 msgr2=0x7f50700c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 -- 172.21.15.98:0/1080995314 shutdown_connections 2024-05-27T00:16:26.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 -- 172.21.15.98:0/1080995314 wait complete. 2024-05-27T00:16:26.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 Processor -- start 2024-05-27T00:16:26.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 -- start start 2024-05-27T00:16:26.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5070072ad0 0x7f507014eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:26.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50701548e0 0x7f50701f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f507015dd50 0x7f50701499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5070067840 con 0x7f507015dd50 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5070067dd0 con 0x7f50701548e0 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.645+0000 7f506fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f50700679c0 con 0x7f5070072ad0 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f507015dd50 0x7f50701499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f507015dd50 0x7f50701499b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33296/0 (socket says 172.21.15.98:33296) 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506f400640 1 -- 172.21.15.98:0/2608403593 learned_addr learned my addr 172.21.15.98:0/2608403593 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f5067e00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50701548e0 0x7f50701f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506ea00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5070072ad0 0x7f507014eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506f400640 1 -- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5070072ad0 msgr2=0x7f507014eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506f400640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5070072ad0 0x7f507014eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506f400640 1 -- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50701548e0 msgr2=0x7f50701f3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:26.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506f400640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50701548e0 0x7f50701f3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:26.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506f400640 1 -- 172.21.15.98:0/2608403593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50701f03c0 con 0x7f507015dd50 2024-05-27T00:16:26.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506f400640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f507015dd50 0x7f50701499b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f506800c730 tx=0x7f506800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:26.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f5066a00640 1 -- 172.21.15.98:0/2608403593 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068018070 con 0x7f507015dd50 2024-05-27T00:16:26.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f5066a00640 1 -- 172.21.15.98:0/2608403593 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f506800ce40 con 0x7f507015dd50 2024-05-27T00:16:26.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f5066a00640 1 -- 172.21.15.98:0/2608403593 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506800a860 con 0x7f507015dd50 2024-05-27T00:16:26.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506fe00640 1 -- 172.21.15.98:0/2608403593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50701f3ad0 con 0x7f507015dd50 2024-05-27T00:16:26.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506fe00640 1 -- 172.21.15.98:0/2608403593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5070155590 con 0x7f507015dd50 2024-05-27T00:16:26.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f506fe00640 1 -- 172.21.15.98:0/2608403593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5070075c40 con 0x7f507015dd50 2024-05-27T00:16:26.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.649+0000 7f5066a00640 1 -- 172.21.15.98:0/2608403593 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f506801b070 con 0x7f507015dd50 2024-05-27T00:16:26.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.653+0000 7f5066a00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f504c0678f0 0x7f504c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:26.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.653+0000 7f5066a00640 1 -- 172.21.15.98:0/2608403593 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7f50680851d0 con 0x7f507015dd50 2024-05-27T00:16:26.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.653+0000 7f506ea00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f504c0678f0 0x7f504c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:26.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.653+0000 7f506ea00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f504c0678f0 0x7f504c069db0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f50600b2000 tx=0x7f5060008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:26.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.657+0000 7f5066a00640 1 -- 172.21.15.98:0/2608403593 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f50680555e0 con 0x7f507015dd50 2024-05-27T00:16:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 -- 172.21.15.98:0/2608403593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50700723e0 con 0x7f507015dd50 2024-05-27T00:16:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f5066a00640 1 -- 172.21.15.98:0/2608403593 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6003 (secure 0 0 0) 0x7f50680598e0 con 0x7f507015dd50 2024-05-27T00:16:26.891 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:26.891 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:25.159565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:16:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 9 2024-05-27T00:16:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 -- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f504c0678f0 msgr2=0x7f504c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f504c0678f0 0x7f504c069db0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f50600b2000 tx=0x7f5060008040 comp rx=0 tx=0).stop 2024-05-27T00:16:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 -- 172.21.15.98:0/2608403593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f507015dd50 msgr2=0x7f50701499b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f507015dd50 0x7f50701499b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f506800c730 tx=0x7f506800cbf0 comp rx=0 tx=0).stop 2024-05-27T00:16:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 -- 172.21.15.98:0/2608403593 shutdown_connections 2024-05-27T00:16:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f507015dd50 0x7f50701499b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f504c0678f0 0x7f504c069db0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50701548e0 0x7f50701f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 --2- 172.21.15.98:0/2608403593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5070072ad0 0x7f507014eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 -- 172.21.15.98:0/2608403593 >> 172.21.15.98:0/2608403593 conn(0x7f50700c9af0 msgr2=0x7f5070067130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 -- 172.21.15.98:0/2608403593 shutdown_connections 2024-05-27T00:16:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:26.889+0000 7f506fe00640 1 -- 172.21.15.98:0/2608403593 wait complete. 2024-05-27T00:16:26.913 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-05-27T00:16:26.914 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:16:26.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:27.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436fe00640 1 Processor -- start 2024-05-27T00:16:27.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436fe00640 1 -- start start 2024-05-27T00:16:27.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368073760 0x7f436815ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:27.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436fe00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4368072bd0 0x7f436815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:27.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436fe00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f436815def0 con 0x7f4368073380 2024-05-27T00:16:27.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436fe00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f436815aa00 con 0x7f4368072bd0 2024-05-27T00:16:27.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f436815e1b0 con 0x7f4368073760 2024-05-27T00:16:27.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436d400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4368073380 0x7f4368072350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48762/0 (socket says 172.21.15.98:48762) 2024-05-27T00:16:27.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436d400640 1 -- 172.21.15.98:0/898531755 learned_addr learned my addr 172.21.15.98:0/898531755 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:27.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436de00640 1 --2- 172.21.15.98:0/898531755 >> v2:172.21.15.204:3300/0 conn(0x7f4368072bd0 0x7f436815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:27.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436ca00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368073760 0x7f436815ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:27.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 19176558 0 0) 0x7f436815def0 con 0x7f4368073380 2024-05-27T00:16:27.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4350003530 con 0x7f4368073380 2024-05-27T00:16:27.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4240247830 0 0) 0x7f4350003530 con 0x7f4368073380 2024-05-27T00:16:27.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368073760 msgr2=0x7f436815ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:27.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368073760 0x7f436815ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 >> v2:172.21.15.204:3300/0 conn(0x7f4368072bd0 msgr2=0x7f436815b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:27.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 --2- 172.21.15.98:0/898531755 >> v2:172.21.15.204:3300/0 conn(0x7f4368072bd0 0x7f436815b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436815e490 con 0x7f4368073380 2024-05-27T00:16:27.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f435c001680 con 0x7f4368073380 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 >> v1:172.21.15.98:6789/0 conn(0x7f4368073380 legacy=0x7f4368072350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368073760 0x7f43500039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 0x7f4350004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4350004d50 0x7f43500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f436815aa00 con 0x7f4350003f40 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f436815e1b0 con 0x7f4350004d50 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f435000ba00 con 0x7f4368073760 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436de00640 1 --2- 172.21.15.98:0/898531755 >> v2:172.21.15.204:3300/0 conn(0x7f4368072bd0 0x7f436815b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436ca00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 0x7f4350004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436d400640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368073760 0x7f43500039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436de00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4350004d50 0x7f43500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436ca00640 1 -- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368073760 msgr2=0x7f43500039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:27.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436ca00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368073760 0x7f43500039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436ca00640 1 -- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4350004d50 msgr2=0x7f43500051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:27.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436ca00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4350004d50 0x7f43500051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.053+0000 7f436ca00640 1 -- 172.21.15.98:0/898531755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4350003530 con 0x7f4350003f40 2024-05-27T00:16:27.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f436ca00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 0x7f4350004810 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4354009ab0 tx=0x7f435401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f474769e45000b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:27.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43540b2070 con 0x7f4350003f40 2024-05-27T00:16:27.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43540a5640 con 0x7f4350003f40 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f4367e00640 1 -- 172.21.15.98:0/898531755 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43540096b0 con 0x7f4350003f40 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f436fe00640 1 -- 172.21.15.98:0/898531755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 msgr2=0x7f4350004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f436fe00640 1 --2- 172.21.15.98:0/898531755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 0x7f4350004810 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4354009ab0 tx=0x7f435401ca50 comp rx=0 tx=0).stop 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f436de00640 1 -- 172.21.15.98:0/898531755 reap_dead start 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f436fe00640 1 -- 172.21.15.98:0/898531755 shutdown_connections 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f436fe00640 1 -- 172.21.15.98:0/898531755 >> 172.21.15.98:0/898531755 conn(0x7f43680c9af0 msgr2=0x7f43680c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f436fe00640 1 -- 172.21.15.98:0/898531755 shutdown_connections 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.057+0000 7f436fe00640 1 -- 172.21.15.98:0/898531755 wait complete. 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436fe00640 1 Processor -- start 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436fe00640 1 -- start start 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 0x7f436815d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:27.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f436815bda0 0x7f43681f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368157ae0 0x7f43681f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436fe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f436815ed30 con 0x7f4350003f40 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f436815ea50 con 0x7f436815bda0 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f436815ae90 con 0x7f4368157ae0 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 0x7f436815d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 0x7f436815d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33302/0 (socket says 172.21.15.98:33302) 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436d400640 1 -- 172.21.15.98:0/3614623939 learned_addr learned my addr 172.21.15.98:0/3614623939 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436de00640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368157ae0 0x7f43681f4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436ca00640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f436815bda0 0x7f43681f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436d400640 1 -- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368157ae0 msgr2=0x7f43681f4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436d400640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368157ae0 0x7f43681f4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436d400640 1 -- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f436815bda0 msgr2=0x7f43681f0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:27.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436d400640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f436815bda0 0x7f43681f0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436d400640 1 -- 172.21.15.98:0/3614623939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436815c4c0 con 0x7f4350003f40 2024-05-27T00:16:27.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436d400640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 0x7f436815d760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f435c007310 tx=0x7f435c007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:27.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f4365600640 1 -- 172.21.15.98:0/3614623939 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c01a070 con 0x7f4350003f40 2024-05-27T00:16:27.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f4365600640 1 -- 172.21.15.98:0/3614623939 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f435c00f960 con 0x7f4350003f40 2024-05-27T00:16:27.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436fe00640 1 -- 172.21.15.98:0/3614623939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436815c7a0 con 0x7f4350003f40 2024-05-27T00:16:27.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f4365600640 1 -- 172.21.15.98:0/3614623939 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c00fd30 con 0x7f4350003f40 2024-05-27T00:16:27.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.061+0000 7f436fe00640 1 -- 172.21.15.98:0/3614623939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43681f1550 con 0x7f4350003f40 2024-05-27T00:16:27.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.065+0000 7f436fe00640 1 -- 172.21.15.98:0/3614623939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4368075cb0 con 0x7f4350003f40 2024-05-27T00:16:27.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.065+0000 7f4365600640 1 -- 172.21.15.98:0/3614623939 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f435c01d070 con 0x7f4350003f40 2024-05-27T00:16:27.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.065+0000 7f4365600640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f43480678c0 0x7f4348069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:27.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.065+0000 7f4365600640 1 -- 172.21.15.98:0/3614623939 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7f435c0855a0 con 0x7f4350003f40 2024-05-27T00:16:27.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.065+0000 7f436ca00640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f43480678c0 0x7f4348069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:27.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.069+0000 7f436ca00640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f43480678c0 0x7f4348069d80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f43540af000 tx=0x7f43540046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:27.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.069+0000 7f4365600640 1 -- 172.21.15.98:0/3614623939 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f435c0559c0 con 0x7f4350003f40 2024-05-27T00:16:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.269+0000 7f436fe00640 1 -- 172.21.15.98:0/3614623939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f436815b8a0 con 0x7f4350003f40 2024-05-27T00:16:27.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.269+0000 7f4365600640 1 -- 172.21.15.98:0/3614623939 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6003 (secure 0 0 0) 0x7f435c083e20 con 0x7f4350003f40 2024-05-27T00:16:27.273 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:27.273 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:25.159565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:16:27.274 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 9 2024-05-27T00:16:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 -- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f43480678c0 msgr2=0x7f4348069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f43480678c0 0x7f4348069d80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f43540af000 tx=0x7f43540046d0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 -- 172.21.15.98:0/3614623939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 msgr2=0x7f436815d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 0x7f436815d760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f435c007310 tx=0x7f435c007710 comp rx=0 tx=0).stop 2024-05-27T00:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 -- 172.21.15.98:0/3614623939 shutdown_connections 2024-05-27T00:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4368157ae0 0x7f43681f4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f43480678c0 0x7f4348069d80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f436815bda0 0x7f43681f0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 --2- 172.21.15.98:0/3614623939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4350003f40 0x7f436815d760 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 -- 172.21.15.98:0/3614623939 >> 172.21.15.98:0/3614623939 conn(0x7f43680c9af0 msgr2=0x7f4368071f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 -- 172.21.15.98:0/3614623939 shutdown_connections 2024-05-27T00:16:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:27.273+0000 7f436fe00640 1 -- 172.21.15.98:0/3614623939 wait complete. 2024-05-27T00:16:27.296 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 9, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4281": { "addr": "172.21.15.98:6834/3903574515", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 3903574515, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 3903574515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 9, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:16:25.159565+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4281 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:16:29.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac99c00640 1 Processor -- start 2024-05-27T00:16:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac99c00640 1 -- start start 2024-05-27T00:16:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac99c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fac9406b8e0 0x7fac9415b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac99c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac94067130 0x7fac940675c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac99c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac9406f430 con 0x7fac94154950 2024-05-27T00:16:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac99c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fac9415b5f0 con 0x7fac9406b8e0 2024-05-27T00:16:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac99c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fac9406c5e0 con 0x7fac94067130 2024-05-27T00:16:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac93400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fac9406b8e0 0x7fac9415b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac93e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fac94154950 0x7fac9415dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48764/0 (socket says 172.21.15.98:48764) 2024-05-27T00:16:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac93400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fac9406b8e0 0x7fac9415b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33312/0 (socket says 172.21.15.98:33312) 2024-05-27T00:16:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.437+0000 7fac93e00640 1 -- 172.21.15.98:0/3583152088 learned_addr learned my addr 172.21.15.98:0/3583152088 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.441+0000 7fac99200640 1 --2- 172.21.15.98:0/3583152088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac94067130 0x7fac940675c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.441+0000 7fac93400640 1 -- 172.21.15.98:0/3583152088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac94067130 msgr2=0x7fac940675c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.441+0000 7fac93400640 1 --2- 172.21.15.98:0/3583152088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac94067130 0x7fac940675c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.441+0000 7fac93400640 1 -- 172.21.15.98:0/3583152088 >> v1:172.21.15.98:6789/0 conn(0x7fac94154950 legacy=0x7fac9415dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.441+0000 7fac93400640 1 -- 172.21.15.98:0/3583152088 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9406be20 con 0x7fac9406b8e0 2024-05-27T00:16:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.441+0000 7fac92a00640 1 -- 172.21.15.98:0/3583152088 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 415635586 0 0) 0x7fac9406f430 con 0x7fac94154950 2024-05-27T00:16:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.441+0000 7fac93400640 1 --2- 172.21.15.98:0/3583152088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9406b8e0 0x7fac9415b0b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fac800097c0 tx=0x7fac8001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13fb2086c83cd22c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac92a00640 1 -- 172.21.15.98:0/3583152088 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac800b2070 con 0x7fac9406b8e0 2024-05-27T00:16:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac92a00640 1 -- 172.21.15.98:0/3583152088 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac800ad640 con 0x7fac9406b8e0 2024-05-27T00:16:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac92a00640 1 -- 172.21.15.98:0/3583152088 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac800ada10 con 0x7fac9406b8e0 2024-05-27T00:16:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 -- 172.21.15.98:0/3583152088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9406b8e0 msgr2=0x7fac9415b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 --2- 172.21.15.98:0/3583152088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9406b8e0 0x7fac9415b0b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fac800097c0 tx=0x7fac8001c4a0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 -- 172.21.15.98:0/3583152088 shutdown_connections 2024-05-27T00:16:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 --2- 172.21.15.98:0/3583152088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac94067130 0x7fac940675c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 --2- 172.21.15.98:0/3583152088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac9406b8e0 0x7fac9415b0b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 -- 172.21.15.98:0/3583152088 >> 172.21.15.98:0/3583152088 conn(0x7fac940cd060 msgr2=0x7fac940cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 -- 172.21.15.98:0/3583152088 shutdown_connections 2024-05-27T00:16:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 -- 172.21.15.98:0/3583152088 wait complete. 2024-05-27T00:16:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 Processor -- start 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 -- start start 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac94067130 0x7fac9415d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac9406b8e0 0x7fac9414da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac94154950 0x7fac9415ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac9406ca70 con 0x7fac94154950 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fac9415e360 con 0x7fac94067130 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fac940c7a60 con 0x7fac9406b8e0 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac94154950 0x7fac9415ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac94154950 0x7fac9415ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33318/0 (socket says 172.21.15.98:33318) 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac99200640 1 -- 172.21.15.98:0/796701045 learned_addr learned my addr 172.21.15.98:0/796701045 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.445+0000 7fac93400640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac9406b8e0 0x7fac9414da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac93e00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac94067130 0x7fac9415d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac99200640 1 -- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac9406b8e0 msgr2=0x7fac9414da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac99200640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac9406b8e0 0x7fac9414da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac99200640 1 -- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac94067130 msgr2=0x7fac9415d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac99200640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac94067130 0x7fac9415d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac99200640 1 -- 172.21.15.98:0/796701045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9415de70 con 0x7fac94154950 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac93e00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac94067130 0x7fac9415d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac99200640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac94154950 0x7fac9415ba80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fac8c007e90 tx=0x7fac8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac8be00640 1 -- 172.21.15.98:0/796701045 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c014420 con 0x7fac94154950 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac8be00640 1 -- 172.21.15.98:0/796701045 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac8c019070 con 0x7fac94154950 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac99c00640 1 -- 172.21.15.98:0/796701045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac9414df90 con 0x7fac94154950 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac8be00640 1 -- 172.21.15.98:0/796701045 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c014800 con 0x7fac94154950 2024-05-27T00:16:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac99c00640 1 -- 172.21.15.98:0/796701045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac941f6f00 con 0x7fac94154950 2024-05-27T00:16:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac99c00640 1 -- 172.21.15.98:0/796701045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac94075c40 con 0x7fac94154950 2024-05-27T00:16:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.449+0000 7fac8be00640 1 -- 172.21.15.98:0/796701045 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fac8c0149a0 con 0x7fac94154950 2024-05-27T00:16:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.453+0000 7fac8be00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fac740678c0 0x7fac74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.453+0000 7fac8be00640 1 -- 172.21.15.98:0/796701045 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7fac8c086590 con 0x7fac94154950 2024-05-27T00:16:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.453+0000 7fac93e00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fac740678c0 0x7fac74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.453+0000 7fac93e00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fac740678c0 0x7fac74069d80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fac84007fd0 tx=0x7fac84007c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.457+0000 7fac8be00640 1 -- 172.21.15.98:0/796701045 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fac8c056f60 con 0x7fac94154950 2024-05-27T00:16:29.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.709+0000 7fac99c00640 1 -- 172.21.15.98:0/796701045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac940c7530 con 0x7fac94154950 2024-05-27T00:16:29.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.709+0000 7fac8be00640 1 -- 172.21.15.98:0/796701045 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6003 (secure 0 0 0) 0x7fac8c056860 con 0x7fac94154950 2024-05-27T00:16:29.714 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:29.714 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:25.159565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:16:29.714 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 9 2024-05-27T00:16:29.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 -- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fac740678c0 msgr2=0x7fac74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:29.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fac740678c0 0x7fac74069d80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fac84007fd0 tx=0x7fac84007c20 comp rx=0 tx=0).stop 2024-05-27T00:16:29.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 -- 172.21.15.98:0/796701045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac94154950 msgr2=0x7fac9415ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:29.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac94154950 0x7fac9415ba80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fac8c007e90 tx=0x7fac8c00e4b0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 -- 172.21.15.98:0/796701045 shutdown_connections 2024-05-27T00:16:29.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac94154950 0x7fac9415ba80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac9406b8e0 0x7fac9414da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fac740678c0 0x7fac74069d80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 --2- 172.21.15.98:0/796701045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac94067130 0x7fac9415d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:29.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 -- 172.21.15.98:0/796701045 >> 172.21.15.98:0/796701045 conn(0x7fac940cd060 msgr2=0x7fac940cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:29.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 -- 172.21.15.98:0/796701045 shutdown_connections 2024-05-27T00:16:29.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:29.713+0000 7fac99c00640 1 -- 172.21.15.98:0/796701045 wait complete. 2024-05-27T00:16:29.738 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:16:31.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.877+0000 7fd18b200640 1 Processor -- start 2024-05-27T00:16:31.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.877+0000 7fd18b200640 1 -- start start 2024-05-27T00:16:31.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.877+0000 7fd18b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 0x7fd18415a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:31.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.877+0000 7fd18b200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1840c8230 con 0x7fd184157710 2024-05-27T00:16:31.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.877+0000 7fd18b200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1840c84d0 con 0x7fd18406eed0 2024-05-27T00:16:31.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.877+0000 7fd18b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd1840c8000 con 0x7fd184150c80 2024-05-27T00:16:31.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.877+0000 7fd189400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd184157710 0x7fd1840c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48768/0 (socket says 172.21.15.98:48768) 2024-05-27T00:16:31.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.877+0000 7fd189400640 1 -- 172.21.15.98:0/1588021299 learned_addr learned my addr 172.21.15.98:0/1588021299 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.881+0000 7fd189e00640 1 --2- 172.21.15.98:0/1588021299 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 0x7fd18415a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.881+0000 7fd188a00640 1 -- 172.21.15.98:0/1588021299 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3485276521 0 0) 0x7fd1840c8230 con 0x7fd184157710 2024-05-27T00:16:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.881+0000 7fd188a00640 1 -- 172.21.15.98:0/1588021299 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd16c003530 con 0x7fd184157710 2024-05-27T00:16:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.881+0000 7fd188a00640 1 -- 172.21.15.98:0/1588021299 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 236112155 0 0) 0x7fd1840c84d0 con 0x7fd18406eed0 2024-05-27T00:16:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.881+0000 7fd188a00640 1 -- 172.21.15.98:0/1588021299 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1840c8230 con 0x7fd18406eed0 2024-05-27T00:16:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.881+0000 7fd188a00640 1 -- 172.21.15.98:0/1588021299 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4060695196 0 0) 0x7fd16c003530 con 0x7fd184157710 2024-05-27T00:16:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.881+0000 7fd189e00640 1 -- 172.21.15.98:0/1588021299 >> v1:172.21.15.204:6789/0 conn(0x7fd18406eed0 legacy=0x7fd18406b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.881+0000 7fd189e00640 1 -- 172.21.15.98:0/1588021299 >> v1:172.21.15.98:6789/0 conn(0x7fd184157710 legacy=0x7fd1840c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:31.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.881+0000 7fd189e00640 1 -- 172.21.15.98:0/1588021299 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd184156c30 con 0x7fd184150c80 2024-05-27T00:16:31.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.881+0000 7fd189e00640 1 --2- 172.21.15.98:0/1588021299 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 0x7fd18415a9e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd1780097f0 tx=0x7fd17801ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d5127dae0dcf2ce server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:31.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd188a00640 1 -- 172.21.15.98:0/1588021299 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1780b5070 con 0x7fd184150c80 2024-05-27T00:16:31.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd188a00640 1 -- 172.21.15.98:0/1588021299 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1780a8670 con 0x7fd184150c80 2024-05-27T00:16:31.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd188a00640 1 -- 172.21.15.98:0/1588021299 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd174001680 con 0x7fd184150c80 2024-05-27T00:16:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 -- 172.21.15.98:0/1588021299 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 msgr2=0x7fd18415a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 --2- 172.21.15.98:0/1588021299 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 0x7fd18415a9e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd1780097f0 tx=0x7fd17801ebf0 comp rx=0 tx=0).stop 2024-05-27T00:16:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 -- 172.21.15.98:0/1588021299 shutdown_connections 2024-05-27T00:16:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 --2- 172.21.15.98:0/1588021299 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 0x7fd18415a9e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 -- 172.21.15.98:0/1588021299 >> 172.21.15.98:0/1588021299 conn(0x7fd1840cbe00 msgr2=0x7fd1840cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 -- 172.21.15.98:0/1588021299 shutdown_connections 2024-05-27T00:16:31.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 -- 172.21.15.98:0/1588021299 wait complete. 2024-05-27T00:16:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 Processor -- start 2024-05-27T00:16:31.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 -- start start 2024-05-27T00:16:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd18406eed0 0x7fd1840af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 0x7fd1840725c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd184157710 0x7fd184072b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd184151060 con 0x7fd184157710 2024-05-27T00:16:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1841571f0 con 0x7fd18406eed0 2024-05-27T00:16:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.885+0000 7fd18b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd1841511e0 con 0x7fd184150c80 2024-05-27T00:16:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd184157710 0x7fd184072b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:31.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd184157710 0x7fd184072b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33320/0 (socket says 172.21.15.98:33320) 2024-05-27T00:16:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18a800640 1 -- 172.21.15.98:0/3184766133 learned_addr learned my addr 172.21.15.98:0/3184766133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd189e00640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd18406eed0 0x7fd1840af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:31.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd189400640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 0x7fd1840725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18a800640 1 -- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 msgr2=0x7fd1840725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18a800640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 0x7fd1840725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18a800640 1 -- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd18406eed0 msgr2=0x7fd1840af2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18a800640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd18406eed0 0x7fd1840af2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18a800640 1 -- 172.21.15.98:0/3184766133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1840b09a0 con 0x7fd184157710 2024-05-27T00:16:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18a800640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd184157710 0x7fd184072b00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd18000c7c0 tx=0x7fd18000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd17e000640 1 -- 172.21.15.98:0/3184766133 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd180003b80 con 0x7fd184157710 2024-05-27T00:16:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd17e000640 1 -- 172.21.15.98:0/3184766133 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd180003d20 con 0x7fd184157710 2024-05-27T00:16:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18b200640 1 -- 172.21.15.98:0/3184766133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1840afb10 con 0x7fd184157710 2024-05-27T00:16:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd17e000640 1 -- 172.21.15.98:0/3184766133 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18000aa50 con 0x7fd184157710 2024-05-27T00:16:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18b200640 1 -- 172.21.15.98:0/3184766133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd184071ae0 con 0x7fd184157710 2024-05-27T00:16:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd18b200640 1 -- 172.21.15.98:0/3184766133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd184075c40 con 0x7fd184157710 2024-05-27T00:16:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.889+0000 7fd17e000640 1 -- 172.21.15.98:0/3184766133 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd18000abf0 con 0x7fd184157710 2024-05-27T00:16:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.893+0000 7fd17e000640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd168067910 0x7fd168069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.893+0000 7fd17e000640 1 -- 172.21.15.98:0/3184766133 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7fd180085b40 con 0x7fd184157710 2024-05-27T00:16:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.893+0000 7fd189e00640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd168067910 0x7fd168069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.893+0000 7fd189e00640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd168067910 0x7fd168069dd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd1780b2000 tx=0x7fd1780aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:31.897+0000 7fd17e000640 1 -- 172.21.15.98:0/3184766133 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd180055fd0 con 0x7fd184157710 2024-05-27T00:16:32.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.109+0000 7fd18b200640 1 -- 172.21.15.98:0/3184766133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd18406f520 con 0x7fd184157710 2024-05-27T00:16:32.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.109+0000 7fd17e000640 1 -- 172.21.15.98:0/3184766133 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6009 (secure 0 0 0) 0x7fd18005a2d0 con 0x7fd184157710 2024-05-27T00:16:32.114 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:32.114 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:30.173501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:16:32.114 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 10 2024-05-27T00:16:32.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 -- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd168067910 msgr2=0x7fd168069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:32.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd168067910 0x7fd168069dd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd1780b2000 tx=0x7fd1780aa070 comp rx=0 tx=0).stop 2024-05-27T00:16:32.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 -- 172.21.15.98:0/3184766133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd184157710 msgr2=0x7fd184072b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:32.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd184157710 0x7fd184072b00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd18000c7c0 tx=0x7fd18000cc80 comp rx=0 tx=0).stop 2024-05-27T00:16:32.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 -- 172.21.15.98:0/3184766133 shutdown_connections 2024-05-27T00:16:32.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd184157710 0x7fd184072b00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:32.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd184150c80 0x7fd1840725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:32.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd168067910 0x7fd168069dd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:32.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 --2- 172.21.15.98:0/3184766133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd18406eed0 0x7fd1840af2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:32.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 -- 172.21.15.98:0/3184766133 >> 172.21.15.98:0/3184766133 conn(0x7fd1840cbe00 msgr2=0x7fd18414ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:32.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 -- 172.21.15.98:0/3184766133 shutdown_connections 2024-05-27T00:16:32.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:32.113+0000 7fd18b200640 1 -- 172.21.15.98:0/3184766133 wait complete. 2024-05-27T00:16:32.135 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:16:34.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:34.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb047e00640 1 Processor -- start 2024-05-27T00:16:34.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb047e00640 1 -- start start 2024-05-27T00:16:34.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb047e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0480b3260 0x7fb0480c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:34.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb047e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb048070660 0x7fb0480c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:34.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb047e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb048071010 con 0x7fb0480c36c0 2024-05-27T00:16:34.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb047e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb0480c88c0 con 0x7fb048070660 2024-05-27T00:16:34.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb047e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb0480c8b80 con 0x7fb0480b3260 2024-05-27T00:16:34.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb046000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb048070660 0x7fb0480c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:34.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb046000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb048070660 0x7fb0480c3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46942/0 (socket says 172.21.15.98:46942) 2024-05-27T00:16:34.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb046000640 1 -- 172.21.15.98:0/2159817968 learned_addr learned my addr 172.21.15.98:0/2159817968 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:34.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb046a00640 1 --2- 172.21.15.98:0/2159817968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0480b3260 0x7fb0480c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:34.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb046000640 1 -- 172.21.15.98:0/2159817968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0480b3260 msgr2=0x7fb0480c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:34.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb046000640 1 --2- 172.21.15.98:0/2159817968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0480b3260 0x7fb0480c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:34.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb046000640 1 -- 172.21.15.98:0/2159817968 >> v1:172.21.15.204:6789/0 conn(0x7fb0480c36c0 legacy=0x7fb0480c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:34.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.285+0000 7fb046000640 1 -- 172.21.15.98:0/2159817968 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0480c8e60 con 0x7fb048070660 2024-05-27T00:16:34.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb046000640 1 --2- 172.21.15.98:0/2159817968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb048070660 0x7fb0480c3180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb034009870 tx=0x7fb03401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851489347a27daba server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb045600640 1 -- 172.21.15.98:0/2159817968 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0340b5070 con 0x7fb048070660 2024-05-27T00:16:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb045600640 1 -- 172.21.15.98:0/2159817968 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0340a8a50 con 0x7fb048070660 2024-05-27T00:16:34.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb045600640 1 -- 172.21.15.98:0/2159817968 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0340a8e20 con 0x7fb048070660 2024-05-27T00:16:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb047e00640 1 -- 172.21.15.98:0/2159817968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb048070660 msgr2=0x7fb0480c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb047e00640 1 --2- 172.21.15.98:0/2159817968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb048070660 0x7fb0480c3180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb034009870 tx=0x7fb03401ed50 comp rx=0 tx=0).stop 2024-05-27T00:16:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb047e00640 1 -- 172.21.15.98:0/2159817968 shutdown_connections 2024-05-27T00:16:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb047e00640 1 --2- 172.21.15.98:0/2159817968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb048070660 0x7fb0480c3180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:34.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb047e00640 1 --2- 172.21.15.98:0/2159817968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0480b3260 0x7fb0480c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:34.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb047e00640 1 -- 172.21.15.98:0/2159817968 >> 172.21.15.98:0/2159817968 conn(0x7fb0480cdeb0 msgr2=0x7fb0480ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:34.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb047e00640 1 -- 172.21.15.98:0/2159817968 shutdown_connections 2024-05-27T00:16:34.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.289+0000 7fb047e00640 1 -- 172.21.15.98:0/2159817968 wait complete. 2024-05-27T00:16:34.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047e00640 1 Processor -- start 2024-05-27T00:16:34.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047e00640 1 -- start start 2024-05-27T00:16:34.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb048070660 0x7fb0481e7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:34.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0480c36c0 0x7fb04815b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:34.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0481f0500 0x7fb04815bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:34.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0480c8200 con 0x7fb0481f0500 2024-05-27T00:16:34.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb0480c9420 con 0x7fb0480c36c0 2024-05-27T00:16:34.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb0480c9940 con 0x7fb048070660 2024-05-27T00:16:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0481f0500 0x7fb04815bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb046a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb048070660 0x7fb0481e7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb046000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0480c36c0 0x7fb04815b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb046000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0480c36c0 0x7fb04815b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:33784/0 (socket says 172.21.15.98:33784) 2024-05-27T00:16:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb046000640 1 -- 172.21.15.98:0/31755821 learned_addr learned my addr 172.21.15.98:0/31755821 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047400640 1 -- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb048070660 msgr2=0x7fb0481e7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047400640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb048070660 0x7fb0481e7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047400640 1 -- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0480c36c0 msgr2=0x7fb04815b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047400640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0480c36c0 0x7fb04815b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047400640 1 -- 172.21.15.98:0/31755821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb048157c90 con 0x7fb0481f0500 2024-05-27T00:16:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047400640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0481f0500 0x7fb04815bda0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb0480c5450 tx=0x7fb04000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb03ea00640 1 -- 172.21.15.98:0/31755821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb040003b80 con 0x7fb0481f0500 2024-05-27T00:16:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb03ea00640 1 -- 172.21.15.98:0/31755821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb040003d20 con 0x7fb0481f0500 2024-05-27T00:16:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047e00640 1 -- 172.21.15.98:0/31755821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04815d0c0 con 0x7fb0481f0500 2024-05-27T00:16:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb03ea00640 1 -- 172.21.15.98:0/31755821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0400052f0 con 0x7fb0481f0500 2024-05-27T00:16:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.293+0000 7fb047e00640 1 -- 172.21.15.98:0/31755821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0481f0c70 con 0x7fb0481f0500 2024-05-27T00:16:34.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.297+0000 7fb047e00640 1 -- 172.21.15.98:0/31755821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb048075c40 con 0x7fb0481f0500 2024-05-27T00:16:34.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.297+0000 7fb03ea00640 1 -- 172.21.15.98:0/31755821 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb0400056a0 con 0x7fb0481f0500 2024-05-27T00:16:34.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.297+0000 7fb03ea00640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0280678f0 0x7fb028069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:34.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.297+0000 7fb03ea00640 1 -- 172.21.15.98:0/31755821 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7fb040085c20 con 0x7fb0481f0500 2024-05-27T00:16:34.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.297+0000 7fb046a00640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0280678f0 0x7fb028069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:34.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.297+0000 7fb046a00640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0280678f0 0x7fb028069db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb0380079c0 tx=0x7fb038008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:34.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.305+0000 7fb03ea00640 1 -- 172.21.15.98:0/31755821 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb048071010 con 0x7fb0481f0500 2024-05-27T00:16:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.533+0000 7fb047e00640 1 -- 172.21.15.98:0/31755821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0480c5480 con 0x7fb0481f0500 2024-05-27T00:16:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.533+0000 7fb03ea00640 1 -- 172.21.15.98:0/31755821 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+6009 (secure 0 0 0) 0x7fb040056030 con 0x7fb0481f0500 2024-05-27T00:16:34.537 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:34.537 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:30.173501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:16:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 10 2024-05-27T00:16:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 -- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0280678f0 msgr2=0x7fb028069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0280678f0 0x7fb028069db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb0380079c0 tx=0x7fb038008040 comp rx=0 tx=0).stop 2024-05-27T00:16:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 -- 172.21.15.98:0/31755821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0481f0500 msgr2=0x7fb04815bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0481f0500 0x7fb04815bda0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb0480c5450 tx=0x7fb04000cb90 comp rx=0 tx=0).stop 2024-05-27T00:16:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 -- 172.21.15.98:0/31755821 shutdown_connections 2024-05-27T00:16:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0481f0500 0x7fb04815bda0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0280678f0 0x7fb028069db0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0480c36c0 0x7fb04815b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 --2- 172.21.15.98:0/31755821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb048070660 0x7fb0481e7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 -- 172.21.15.98:0/31755821 >> 172.21.15.98:0/31755821 conn(0x7fb0480cdeb0 msgr2=0x7fb0480c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 -- 172.21.15.98:0/31755821 shutdown_connections 2024-05-27T00:16:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:34.537+0000 7fb047e00640 1 -- 172.21.15.98:0/31755821 wait complete. 2024-05-27T00:16:34.559 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:16:36.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:36.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.725+0000 7f34ba400640 1 Processor -- start 2024-05-27T00:16:36.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.725+0000 7f34ba400640 1 -- start start 2024-05-27T00:16:36.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34ba400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34b4152850 0x7f34b40c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:36.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34ba400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b40c9f30 0x7f34b4070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:36.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34ba400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b40ca440 con 0x7f34b4070ba0 2024-05-27T00:16:36.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34ba400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f34b40c58f0 con 0x7f34b4152850 2024-05-27T00:16:36.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34ba400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f34b415bab0 con 0x7f34b40c9f30 2024-05-27T00:16:36.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34b9000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34b4152850 0x7f34b40c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:36.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34b9000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f34b4152850 0x7f34b40c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46970/0 (socket says 172.21.15.98:46970) 2024-05-27T00:16:36.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34b9000640 1 -- 172.21.15.98:0/704449977 learned_addr learned my addr 172.21.15.98:0/704449977 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:36.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34abe00640 1 --2- 172.21.15.98:0/704449977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b40c9f30 0x7f34b4070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:36.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34b9000640 1 -- 172.21.15.98:0/704449977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b40c9f30 msgr2=0x7f34b4070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:36.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34b9000640 1 --2- 172.21.15.98:0/704449977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b40c9f30 0x7f34b4070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:36.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34b9000640 1 -- 172.21.15.98:0/704449977 >> v1:172.21.15.204:6789/0 conn(0x7f34b4070ba0 legacy=0x7f34b415bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:36.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.729+0000 7f34b9000640 1 -- 172.21.15.98:0/704449977 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b4152c30 con 0x7f34b4152850 2024-05-27T00:16:36.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34b9000640 1 --2- 172.21.15.98:0/704449977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b4152850 0x7f34b40c53b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f34a4009870 tx=0x7f34a401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2201d7a8c67f5c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:36.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34b3e00640 1 -- 172.21.15.98:0/704449977 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a40b5070 con 0x7f34b4152850 2024-05-27T00:16:36.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34b3e00640 1 -- 172.21.15.98:0/704449977 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a40a8a50 con 0x7f34b4152850 2024-05-27T00:16:36.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34b3e00640 1 -- 172.21.15.98:0/704449977 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a40a8e20 con 0x7f34b4152850 2024-05-27T00:16:36.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34ba400640 1 -- 172.21.15.98:0/704449977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b4152850 msgr2=0x7f34b40c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:36.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34ba400640 1 --2- 172.21.15.98:0/704449977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b4152850 0x7f34b40c53b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f34a4009870 tx=0x7f34a401ed50 comp rx=0 tx=0).stop 2024-05-27T00:16:36.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34ba400640 1 -- 172.21.15.98:0/704449977 shutdown_connections 2024-05-27T00:16:36.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34ba400640 1 --2- 172.21.15.98:0/704449977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b40c9f30 0x7f34b4070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:36.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34ba400640 1 --2- 172.21.15.98:0/704449977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b4152850 0x7f34b40c53b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:36.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34ba400640 1 -- 172.21.15.98:0/704449977 >> 172.21.15.98:0/704449977 conn(0x7f34b40cdeb0 msgr2=0x7f34b40ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:36.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34ba400640 1 -- 172.21.15.98:0/704449977 shutdown_connections 2024-05-27T00:16:36.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.733+0000 7f34ba400640 1 -- 172.21.15.98:0/704449977 wait complete. 2024-05-27T00:16:36.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34ba400640 1 Processor -- start 2024-05-27T00:16:36.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34ba400640 1 -- start start 2024-05-27T00:16:36.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34ba400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b4070ba0 0x7f34b4147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:36.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34ba400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34b40c9f30 0x7f34b41496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:36.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34ba400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b4152850 0x7f34b4149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:36.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34ba400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f34b406fb50 con 0x7f34b4152850 2024-05-27T00:16:36.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34ba400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f34b406f870 con 0x7f34b40c9f30 2024-05-27T00:16:36.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34ba400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f34b406fe30 con 0x7f34b4070ba0 2024-05-27T00:16:36.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34abe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34b40c9f30 0x7f34b41496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:36.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34b9000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b4070ba0 0x7f34b4147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:36.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34b9a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b4152850 0x7f34b4149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:36.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34abe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34b40c9f30 0x7f34b41496d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:33792/0 (socket says 172.21.15.98:33792) 2024-05-27T00:16:36.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.737+0000 7f34abe00640 1 -- 172.21.15.98:0/2808343436 learned_addr learned my addr 172.21.15.98:0/2808343436 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:36.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34b9a00640 1 -- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b4070ba0 msgr2=0x7f34b4147580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:36.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34b9a00640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b4070ba0 0x7f34b4147580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:36.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34b9a00640 1 -- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34b40c9f30 msgr2=0x7f34b41496d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:36.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34b9a00640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34b40c9f30 0x7f34b41496d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:36.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34b9a00640 1 -- 172.21.15.98:0/2808343436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b4147ac0 con 0x7f34b4152850 2024-05-27T00:16:36.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34b9a00640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b4152850 0x7f34b4149c10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00c760 tx=0x7f34ac00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:36.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34b1600640 1 -- 172.21.15.98:0/2808343436 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac016020 con 0x7f34b4152850 2024-05-27T00:16:36.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34b1600640 1 -- 172.21.15.98:0/2808343436 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34ac006d60 con 0x7f34b4152850 2024-05-27T00:16:36.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34b1600640 1 -- 172.21.15.98:0/2808343436 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac003bc0 con 0x7f34b4152850 2024-05-27T00:16:36.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34ba400640 1 -- 172.21.15.98:0/2808343436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b41ee490 con 0x7f34b4152850 2024-05-27T00:16:36.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34ba400640 1 -- 172.21.15.98:0/2808343436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b41eea50 con 0x7f34b4152850 2024-05-27T00:16:36.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34ba400640 1 -- 172.21.15.98:0/2808343436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b4075c40 con 0x7f34b4152850 2024-05-27T00:16:36.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.741+0000 7f34b1600640 1 -- 172.21.15.98:0/2808343436 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f34ac00a850 con 0x7f34b4152850 2024-05-27T00:16:36.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.745+0000 7f34b1600640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f34900678c0 0x7f3490069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:36.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.745+0000 7f34b1600640 1 -- 172.21.15.98:0/2808343436 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7f34ac0857a0 con 0x7f34b4152850 2024-05-27T00:16:36.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.745+0000 7f34b9000640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f34900678c0 0x7f3490069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:36.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.745+0000 7f34b9000640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f34900678c0 0x7f3490069d80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f34a40b2000 tx=0x7f34a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:36.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:36.753+0000 7f34b1600640 1 -- 172.21.15.98:0/2808343436 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f34ac020020 con 0x7f34b4152850 2024-05-27T00:16:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.093+0000 7f34ba400640 1 -- 172.21.15.98:0/2808343436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34b415b890 con 0x7f34b4152850 2024-05-27T00:16:37.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.093+0000 7f34b1600640 1 -- 172.21.15.98:0/2808343436 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6006 (secure 0 0 0) 0x7f34ac055c50 con 0x7f34b4152850 2024-05-27T00:16:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:37.097 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:35.399401+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:16:37.097 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 11 2024-05-27T00:16:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 -- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f34900678c0 msgr2=0x7f3490069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f34900678c0 0x7f3490069d80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f34a40b2000 tx=0x7f34a4009040 comp rx=0 tx=0).stop 2024-05-27T00:16:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 -- 172.21.15.98:0/2808343436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b4152850 msgr2=0x7f34b4149c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b4152850 0x7f34b4149c10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00c760 tx=0x7f34ac00cc20 comp rx=0 tx=0).stop 2024-05-27T00:16:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 -- 172.21.15.98:0/2808343436 shutdown_connections 2024-05-27T00:16:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34b4152850 0x7f34b4149c10 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f34900678c0 0x7f3490069d80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34b40c9f30 0x7f34b41496d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 --2- 172.21.15.98:0/2808343436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b4070ba0 0x7f34b4147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 -- 172.21.15.98:0/2808343436 >> 172.21.15.98:0/2808343436 conn(0x7f34b40cdeb0 msgr2=0x7f34b414d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 -- 172.21.15.98:0/2808343436 shutdown_connections 2024-05-27T00:16:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:37.097+0000 7f34ba400640 1 -- 172.21.15.98:0/2808343436 wait complete. 2024-05-27T00:16:37.126 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:16:39.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.305+0000 7fe87a800640 1 Processor -- start 2024-05-27T00:16:39.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.305+0000 7fe87a800640 1 -- start start 2024-05-27T00:16:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe87a800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8740c6e40 0x7fe874158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe87a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe874151f00 0x7fe8741523b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe87a800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe874159300 con 0x7fe8740c6a60 2024-05-27T00:16:39.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe87a800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe87415bb10 con 0x7fe8740c6e40 2024-05-27T00:16:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe87a800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe87415bdf0 con 0x7fe874151f00 2024-05-27T00:16:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe878a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8740c6e40 0x7fe874158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe878a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe8740c6e40 0x7fe874158dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46990/0 (socket says 172.21.15.98:46990) 2024-05-27T00:16:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe878a00640 1 -- 172.21.15.98:0/1388842577 learned_addr learned my addr 172.21.15.98:0/1388842577 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe879400640 1 --1- 172.21.15.98:0/1388842577 >> v1:172.21.15.204:6789/0 conn(0x7fe8740c6a60 0x7fe874073340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:58884/0 (socket says 172.21.15.98:58884) 2024-05-27T00:16:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe879e00640 1 --2- 172.21.15.98:0/1388842577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe874151f00 0x7fe8741523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:39.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe878a00640 1 -- 172.21.15.98:0/1388842577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe874151f00 msgr2=0x7fe8741523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:39.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe878a00640 1 --2- 172.21.15.98:0/1388842577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe874151f00 0x7fe8741523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:39.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe878a00640 1 -- 172.21.15.98:0/1388842577 >> v1:172.21.15.204:6789/0 conn(0x7fe8740c6a60 legacy=0x7fe874073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:39.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe873e00640 1 -- 172.21.15.98:0/1388842577 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1348240895 0 0) 0x7fe874159300 con 0x7fe8740c6a60 2024-05-27T00:16:39.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.309+0000 7fe878a00640 1 -- 172.21.15.98:0/1388842577 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87415c0d0 con 0x7fe8740c6e40 2024-05-27T00:16:39.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe878a00640 1 --2- 172.21.15.98:0/1388842577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 0x7fe874158dc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe860009870 tx=0x7fe86001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9d21a38da4bf8a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:39.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe873e00640 1 -- 172.21.15.98:0/1388842577 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600b2070 con 0x7fe8740c6e40 2024-05-27T00:16:39.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe873e00640 1 -- 172.21.15.98:0/1388842577 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8600a9070 con 0x7fe8740c6e40 2024-05-27T00:16:39.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe873e00640 1 -- 172.21.15.98:0/1388842577 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600ad880 con 0x7fe8740c6e40 2024-05-27T00:16:39.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe87a800640 1 -- 172.21.15.98:0/1388842577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 msgr2=0x7fe874158dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:39.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe87a800640 1 --2- 172.21.15.98:0/1388842577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 0x7fe874158dc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe860009870 tx=0x7fe86001c560 comp rx=0 tx=0).stop 2024-05-27T00:16:39.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe87a800640 1 -- 172.21.15.98:0/1388842577 shutdown_connections 2024-05-27T00:16:39.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe87a800640 1 --2- 172.21.15.98:0/1388842577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe874151f00 0x7fe8741523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:39.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe87a800640 1 --2- 172.21.15.98:0/1388842577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 0x7fe874158dc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:39.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe87a800640 1 -- 172.21.15.98:0/1388842577 >> 172.21.15.98:0/1388842577 conn(0x7fe8740cbe00 msgr2=0x7fe8740cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:39.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe87a800640 1 -- 172.21.15.98:0/1388842577 shutdown_connections 2024-05-27T00:16:39.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.313+0000 7fe87a800640 1 -- 172.21.15.98:0/1388842577 wait complete. 2024-05-27T00:16:39.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe87a800640 1 Processor -- start 2024-05-27T00:16:39.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe87a800640 1 -- start start 2024-05-27T00:16:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe87a800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8740c6a60 0x7fe8740c2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe87a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 0x7fe874156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe87a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe874151f00 0x7fe8741572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe87a800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8741650a0 con 0x7fe8740c6e40 2024-05-27T00:16:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe87a800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe874164f20 con 0x7fe8740c6a60 2024-05-27T00:16:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe87a800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe874165220 con 0x7fe874151f00 2024-05-27T00:16:39.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe878a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 0x7fe874156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe878a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 0x7fe874156db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47002/0 (socket says 172.21.15.98:47002) 2024-05-27T00:16:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe878a00640 1 -- 172.21.15.98:0/775358855 learned_addr learned my addr 172.21.15.98:0/775358855 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe879400640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8740c6a60 0x7fe8740c2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe879e00640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe874151f00 0x7fe8741572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe878a00640 1 -- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe874151f00 msgr2=0x7fe8741572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe878a00640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe874151f00 0x7fe8741572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe878a00640 1 -- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8740c6a60 msgr2=0x7fe8740c2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe878a00640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8740c6a60 0x7fe8740c2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:39.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe878a00640 1 -- 172.21.15.98:0/775358855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8741f3590 con 0x7fe8740c6e40 2024-05-27T00:16:39.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe878a00640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 0x7fe874156db0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe8600b0040 tx=0x7fe86001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:39.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe871600640 1 -- 172.21.15.98:0/775358855 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600b2040 con 0x7fe8740c6e40 2024-05-27T00:16:39.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe871600640 1 -- 172.21.15.98:0/775358855 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8600be070 con 0x7fe8740c6e40 2024-05-27T00:16:39.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe871600640 1 -- 172.21.15.98:0/775358855 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600adc50 con 0x7fe8740c6e40 2024-05-27T00:16:39.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe87a800640 1 -- 172.21.15.98:0/775358855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8741f3870 con 0x7fe8740c6e40 2024-05-27T00:16:39.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.317+0000 7fe87a800640 1 -- 172.21.15.98:0/775358855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87414c150 con 0x7fe8740c6e40 2024-05-27T00:16:39.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.321+0000 7fe87a800640 1 -- 172.21.15.98:0/775358855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe874075c40 con 0x7fe8740c6e40 2024-05-27T00:16:39.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.321+0000 7fe871600640 1 -- 172.21.15.98:0/775358855 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe8600ab070 con 0x7fe8740c6e40 2024-05-27T00:16:39.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.321+0000 7fe871600640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe8540678c0 0x7fe854069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:39.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.321+0000 7fe871600640 1 -- 172.21.15.98:0/775358855 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7fe86011f8e0 con 0x7fe8740c6e40 2024-05-27T00:16:39.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.325+0000 7fe879400640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe8540678c0 0x7fe854069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:39.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.325+0000 7fe879400640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe8540678c0 0x7fe854069d80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe868000f70 tx=0x7fe868007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:39.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.329+0000 7fe871600640 1 -- 172.21.15.98:0/775358855 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fe860016150 con 0x7fe8740c6e40 2024-05-27T00:16:39.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.629+0000 7fe87a800640 1 -- 172.21.15.98:0/775358855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe874071cb0 con 0x7fe8740c6e40 2024-05-27T00:16:39.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.629+0000 7fe871600640 1 -- 172.21.15.98:0/775358855 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+6006 (secure 0 0 0) 0x7fe8600efc60 con 0x7fe8740c6e40 2024-05-27T00:16:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:39.634 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:35.399401+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:16:39.634 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 11 2024-05-27T00:16:39.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 -- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe8540678c0 msgr2=0x7fe854069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:39.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe8540678c0 0x7fe854069d80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe868000f70 tx=0x7fe868007240 comp rx=0 tx=0).stop 2024-05-27T00:16:39.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 -- 172.21.15.98:0/775358855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 msgr2=0x7fe874156db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:39.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 0x7fe874156db0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe8600b0040 tx=0x7fe86001ada0 comp rx=0 tx=0).stop 2024-05-27T00:16:39.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 -- 172.21.15.98:0/775358855 shutdown_connections 2024-05-27T00:16:39.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe874151f00 0x7fe8741572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:39.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8740c6e40 0x7fe874156db0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:39.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe8540678c0 0x7fe854069d80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:39.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 --2- 172.21.15.98:0/775358855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8740c6a60 0x7fe8740c2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:39.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 -- 172.21.15.98:0/775358855 >> 172.21.15.98:0/775358855 conn(0x7fe8740cbe00 msgr2=0x7fe87406b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:39.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 -- 172.21.15.98:0/775358855 shutdown_connections 2024-05-27T00:16:39.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:39.633+0000 7fe87a800640 1 -- 172.21.15.98:0/775358855 wait complete. 2024-05-27T00:16:39.656 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 11, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 11, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4281": { "addr": "172.21.15.98:6834/3903574515", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 3903574515, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 3903574515, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 9, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:16:35.399401+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4281 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:16:41.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:41.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.881+0000 7f2773200640 1 Processor -- start 2024-05-27T00:16:41.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.881+0000 7f2773200640 1 -- start start 2024-05-27T00:16:41.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.881+0000 7f2773200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774152850 0x7f27740c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:41.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.881+0000 7f2773200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27740c9f30 0x7f2774070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:41.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2773200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27740ca440 con 0x7f2774070ba0 2024-05-27T00:16:41.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2773200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f27740c58f0 con 0x7f27740c9f30 2024-05-27T00:16:41.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2773200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f277415bab0 con 0x7f2774152850 2024-05-27T00:16:41.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2771400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27740c9f30 0x7f2774070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:41.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2771400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27740c9f30 0x7f2774070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47014/0 (socket says 172.21.15.98:47014) 2024-05-27T00:16:41.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2771400640 1 -- 172.21.15.98:0/2608700831 learned_addr learned my addr 172.21.15.98:0/2608700831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:41.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2771e00640 1 --2- 172.21.15.98:0/2608700831 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774152850 0x7f27740c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:41.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2771e00640 1 -- 172.21.15.98:0/2608700831 >> v2:172.21.15.98:3300/0 conn(0x7f27740c9f30 msgr2=0x7f2774070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:41.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2771e00640 1 --2- 172.21.15.98:0/2608700831 >> v2:172.21.15.98:3300/0 conn(0x7f27740c9f30 0x7f2774070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:41.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2770a00640 1 -- 172.21.15.98:0/2608700831 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 42216400 0 0) 0x7f27740ca440 con 0x7f2774070ba0 2024-05-27T00:16:41.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2771e00640 1 -- 172.21.15.98:0/2608700831 >> v1:172.21.15.204:6789/0 conn(0x7f2774070ba0 legacy=0x7f277415bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:41.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.885+0000 7f2771e00640 1 -- 172.21.15.98:0/2608700831 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2774152c30 con 0x7f2774152850 2024-05-27T00:16:41.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2771e00640 1 --2- 172.21.15.98:0/2608700831 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774152850 0x7f27740c53b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f27640097c0 tx=0x7f276401ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c759cb2412eb8161 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:41.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2770a00640 1 -- 172.21.15.98:0/2608700831 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27640b5070 con 0x7f2774152850 2024-05-27T00:16:41.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2770a00640 1 -- 172.21.15.98:0/2608700831 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27640a8990 con 0x7f2774152850 2024-05-27T00:16:41.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2770a00640 1 -- 172.21.15.98:0/2608700831 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27640a8d60 con 0x7f2774152850 2024-05-27T00:16:41.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2773200640 1 -- 172.21.15.98:0/2608700831 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774152850 msgr2=0x7f27740c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:41.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2773200640 1 --2- 172.21.15.98:0/2608700831 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774152850 0x7f27740c53b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f27640097c0 tx=0x7f276401ecb0 comp rx=0 tx=0).stop 2024-05-27T00:16:41.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2773200640 1 -- 172.21.15.98:0/2608700831 shutdown_connections 2024-05-27T00:16:41.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2773200640 1 --2- 172.21.15.98:0/2608700831 >> v2:172.21.15.98:3300/0 conn(0x7f27740c9f30 0x7f2774070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:41.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2773200640 1 --2- 172.21.15.98:0/2608700831 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774152850 0x7f27740c53b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:41.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2773200640 1 -- 172.21.15.98:0/2608700831 >> 172.21.15.98:0/2608700831 conn(0x7f27740cdeb0 msgr2=0x7f27740ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:41.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2773200640 1 -- 172.21.15.98:0/2608700831 shutdown_connections 2024-05-27T00:16:41.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.889+0000 7f2773200640 1 -- 172.21.15.98:0/2608700831 wait complete. 2024-05-27T00:16:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.893+0000 7f2773200640 1 Processor -- start 2024-05-27T00:16:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.893+0000 7f2773200640 1 -- start start 2024-05-27T00:16:41.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2773200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774070ba0 0x7f27741f7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:41.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2773200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27740c9f30 0x7f277414ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:41.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2773200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2774152850 0x7f277414c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:41.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2773200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f277406fb50 con 0x7f2774152850 2024-05-27T00:16:41.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2773200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f277406f870 con 0x7f27740c9f30 2024-05-27T00:16:41.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2773200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f277406fe30 con 0x7f2774070ba0 2024-05-27T00:16:41.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2772800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2774152850 0x7f277414c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:41.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2771400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27740c9f30 0x7f277414ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:41.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2771e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774070ba0 0x7f27741f7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:41.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2772800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2774152850 0x7f277414c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47016/0 (socket says 172.21.15.98:47016) 2024-05-27T00:16:41.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2772800640 1 -- 172.21.15.98:0/729567656 learned_addr learned my addr 172.21.15.98:0/729567656 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2772800640 1 -- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774070ba0 msgr2=0x7f27741f7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2772800640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774070ba0 0x7f27741f7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2772800640 1 -- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27740c9f30 msgr2=0x7f277414ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2772800640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27740c9f30 0x7f277414ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2772800640 1 -- 172.21.15.98:0/729567656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27741f8bf0 con 0x7f2774152850 2024-05-27T00:16:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2772800640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2774152850 0x7f277414c9e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f276c00c890 tx=0x7f276c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f276a000640 1 -- 172.21.15.98:0/729567656 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c017070 con 0x7f2774152850 2024-05-27T00:16:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f276a000640 1 -- 172.21.15.98:0/729567656 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f276c003db0 con 0x7f2774152850 2024-05-27T00:16:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f276a000640 1 -- 172.21.15.98:0/729567656 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c005270 con 0x7f2774152850 2024-05-27T00:16:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2773200640 1 -- 172.21.15.98:0/729567656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27741576c0 con 0x7f2774152850 2024-05-27T00:16:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2773200640 1 -- 172.21.15.98:0/729567656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2774152fa0 con 0x7f2774152850 2024-05-27T00:16:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.897+0000 7f2773200640 1 -- 172.21.15.98:0/729567656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2774075c40 con 0x7f2774152850 2024-05-27T00:16:41.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.901+0000 7f276a000640 1 -- 172.21.15.98:0/729567656 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f276c01a070 con 0x7f2774152850 2024-05-27T00:16:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.901+0000 7f276a000640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2754067910 0x7f2754069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:41.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.901+0000 7f276a000640 1 -- 172.21.15.98:0/729567656 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7f276c085880 con 0x7f2774152850 2024-05-27T00:16:41.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.905+0000 7f2771e00640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2754067910 0x7f2754069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:41.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.905+0000 7f2771e00640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2754067910 0x7f2754069dd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f27640b2000 tx=0x7f27640aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:41.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:41.909+0000 7f276a000640 1 -- 172.21.15.98:0/729567656 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f276c055d40 con 0x7f2774152850 2024-05-27T00:16:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.301+0000 7f2773200640 1 -- 172.21.15.98:0/729567656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f277414e8c0 con 0x7f2774152850 2024-05-27T00:16:42.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.301+0000 7f276a000640 1 -- 172.21.15.98:0/729567656 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6013 (secure 0 0 0) 0x7f276c05a040 con 0x7f2774152850 2024-05-27T00:16:42.305 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:42.305 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:16:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 12 2024-05-27T00:16:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 -- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2754067910 msgr2=0x7f2754069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2754067910 0x7f2754069dd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f27640b2000 tx=0x7f27640aa070 comp rx=0 tx=0).stop 2024-05-27T00:16:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 -- 172.21.15.98:0/729567656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2774152850 msgr2=0x7f277414c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:42.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2774152850 0x7f277414c9e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f276c00c890 tx=0x7f276c00cd50 comp rx=0 tx=0).stop 2024-05-27T00:16:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 -- 172.21.15.98:0/729567656 shutdown_connections 2024-05-27T00:16:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2774152850 0x7f277414c9e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2754067910 0x7f2754069dd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27740c9f30 0x7f277414ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 --2- 172.21.15.98:0/729567656 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2774070ba0 0x7f27741f7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 -- 172.21.15.98:0/729567656 >> 172.21.15.98:0/729567656 conn(0x7f27740cdeb0 msgr2=0x7f27740c85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 -- 172.21.15.98:0/729567656 shutdown_connections 2024-05-27T00:16:42.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:42.305+0000 7f2773200640 1 -- 172.21.15.98:0/729567656 wait complete. 2024-05-27T00:16:42.338 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=4290 2024-05-27T00:16:42.338 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.c 2024-05-27T00:16:43.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:16:43.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:16:48.547 INFO:tasks.ceph.mds.a.smithi098.stderr:2024-05-27T00:16:48.545+0000 7fbef6800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:16:51.339 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-05-27T00:16:51.340 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-27T00:16:51.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-27T00:16:51.344 INFO:tasks.ceph.mds.c:Started 2024-05-27T00:16:51.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:51.483 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2024-05-27T00:16:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.569+0000 7f072e200640 1 Processor -- start 2024-05-27T00:16:51.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.569+0000 7f072e200640 1 -- start start 2024-05-27T00:16:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f072e200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f072815c650 0x7f07280c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f072e200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07280ca440 con 0x7f0728152850 2024-05-27T00:16:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f072e200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0728152c30 con 0x7f07280c9f30 2024-05-27T00:16:51.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f072e200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f07280c58f0 con 0x7f072815c650 2024-05-27T00:16:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f072d800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f072815c650 0x7f07280c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f0727e00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f07280c9f30 0x7f072815bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:42226/0 (socket says 172.21.15.98:42226) 2024-05-27T00:16:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f072d800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f072815c650 0x7f07280c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35500/0 (socket says 172.21.15.98:35500) 2024-05-27T00:16:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f0727e00640 1 -- 172.21.15.98:0/3496876672 learned_addr learned my addr 172.21.15.98:0/3496876672 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f072d800640 1 -- 172.21.15.98:0/3496876672 >> v1:172.21.15.204:6789/0 conn(0x7f07280c9f30 legacy=0x7f072815bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f0727400640 1 -- 172.21.15.98:0/3496876672 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161174867 0 0) 0x7f0728152c30 con 0x7f07280c9f30 2024-05-27T00:16:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f072d800640 1 -- 172.21.15.98:0/3496876672 >> v1:172.21.15.98:6789/0 conn(0x7f0728152850 legacy=0x7f0728070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.573+0000 7f072d800640 1 -- 172.21.15.98:0/3496876672 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072806f590 con 0x7f072815c650 2024-05-27T00:16:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f072d800640 1 --2- 172.21.15.98:0/3496876672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f072815c650 0x7f07280c53b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0720009d90 tx=0x7f072001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da1d4b760a6615 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f0727400640 1 -- 172.21.15.98:0/3496876672 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07200b4070 con 0x7f072815c650 2024-05-27T00:16:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f0727400640 1 -- 172.21.15.98:0/3496876672 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07200af600 con 0x7f072815c650 2024-05-27T00:16:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f0727400640 1 -- 172.21.15.98:0/3496876672 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07200af9d0 con 0x7f072815c650 2024-05-27T00:16:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f072e200640 1 -- 172.21.15.98:0/3496876672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f072815c650 msgr2=0x7f07280c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f072e200640 1 --2- 172.21.15.98:0/3496876672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f072815c650 0x7f07280c53b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0720009d90 tx=0x7f072001f150 comp rx=0 tx=0).stop 2024-05-27T00:16:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f072e200640 1 -- 172.21.15.98:0/3496876672 shutdown_connections 2024-05-27T00:16:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f072e200640 1 --2- 172.21.15.98:0/3496876672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f072815c650 0x7f07280c53b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f072e200640 1 -- 172.21.15.98:0/3496876672 >> 172.21.15.98:0/3496876672 conn(0x7f07280cdeb0 msgr2=0x7f07280ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:51.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f072e200640 1 -- 172.21.15.98:0/3496876672 shutdown_connections 2024-05-27T00:16:51.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.577+0000 7f072e200640 1 -- 172.21.15.98:0/3496876672 wait complete. 2024-05-27T00:16:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.581+0000 7f072e200640 1 Processor -- start 2024-05-27T00:16:51.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.581+0000 7f072e200640 1 -- start start 2024-05-27T00:16:51.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.581+0000 7f072e200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07280c9f30 0x7f07280cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:51.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.581+0000 7f072e200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0728152850 0x7f07281f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:51.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.581+0000 7f072e200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072815c650 0x7f072814d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:51.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.581+0000 7f072e200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f072806fb50 con 0x7f07280c9f30 2024-05-27T00:16:51.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.581+0000 7f072e200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f072815bab0 con 0x7f072815c650 2024-05-27T00:16:51.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.581+0000 7f072e200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f072806fe30 con 0x7f0728152850 2024-05-27T00:16:51.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072ce00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07280c9f30 0x7f07280cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:51.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072ce00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07280c9f30 0x7f07280cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35514/0 (socket says 172.21.15.98:35514) 2024-05-27T00:16:51.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072ce00640 1 -- 172.21.15.98:0/2434426710 learned_addr learned my addr 172.21.15.98:0/2434426710 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:51.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072d800640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072815c650 0x7f072814d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:51.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f0727e00640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0728152850 0x7f07281f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:51.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072ce00640 1 -- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0728152850 msgr2=0x7f07281f2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:51.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072ce00640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0728152850 0x7f07281f2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:51.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072ce00640 1 -- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072815c650 msgr2=0x7f072814d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:51.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072ce00640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072815c650 0x7f072814d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:51.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072ce00640 1 -- 172.21.15.98:0/2434426710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07281f27a0 con 0x7f07280c9f30 2024-05-27T00:16:51.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072ce00640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07280c9f30 0x7f07280cd330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f071c00c530 tx=0x7f071c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f0724c00640 1 -- 172.21.15.98:0/2434426710 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c016020 con 0x7f07280c9f30 2024-05-27T00:16:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f0724c00640 1 -- 172.21.15.98:0/2434426710 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f071c005220 con 0x7f07280c9f30 2024-05-27T00:16:51.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072e200640 1 -- 172.21.15.98:0/2434426710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07280cd9e0 con 0x7f07280c9f30 2024-05-27T00:16:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f0724c00640 1 -- 172.21.15.98:0/2434426710 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c007030 con 0x7f07280c9f30 2024-05-27T00:16:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072e200640 1 -- 172.21.15.98:0/2434426710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07281ee800 con 0x7f07280c9f30 2024-05-27T00:16:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.585+0000 7f072e200640 1 -- 172.21.15.98:0/2434426710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0728075c40 con 0x7f07280c9f30 2024-05-27T00:16:51.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.589+0000 7f0724c00640 1 -- 172.21.15.98:0/2434426710 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f071c0071d0 con 0x7f07280c9f30 2024-05-27T00:16:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.589+0000 7f0724c00640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07080678c0 0x7f0708069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.589+0000 7f0724c00640 1 -- 172.21.15.98:0/2434426710 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7f071c0863f0 con 0x7f07280c9f30 2024-05-27T00:16:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.589+0000 7f0727e00640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07080678c0 0x7f0708069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:51.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.593+0000 7f0727e00640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07080678c0 0x7f0708069d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f07280c4b80 tx=0x7f07140079e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.597+0000 7f0724c00640 1 -- 172.21.15.98:0/2434426710 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f071c020020 con 0x7f07280c9f30 2024-05-27T00:16:51.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.993+0000 7f072e200640 1 -- 172.21.15.98:0/2434426710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f072815b890 con 0x7f07280c9f30 2024-05-27T00:16:51.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.993+0000 7f0724c00640 1 -- 172.21.15.98:0/2434426710 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6013 (secure 0 0 0) 0x7f071c0567f0 con 0x7f07280c9f30 2024-05-27T00:16:51.996 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:51.996 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:16:51.997 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 12 2024-05-27T00:16:51.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 -- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07080678c0 msgr2=0x7f0708069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07080678c0 0x7f0708069d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f07280c4b80 tx=0x7f07140079e0 comp rx=0 tx=0).stop 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 -- 172.21.15.98:0/2434426710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07280c9f30 msgr2=0x7f07280cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07280c9f30 0x7f07280cd330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f071c00c530 tx=0x7f071c00c9f0 comp rx=0 tx=0).stop 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 -- 172.21.15.98:0/2434426710 shutdown_connections 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07080678c0 0x7f0708069d80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072815c650 0x7f072814d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0728152850 0x7f07281f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 --2- 172.21.15.98:0/2434426710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07280c9f30 0x7f07280cd330 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 -- 172.21.15.98:0/2434426710 >> 172.21.15.98:0/2434426710 conn(0x7f07280cdeb0 msgr2=0x7f07280c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 -- 172.21.15.98:0/2434426710 shutdown_connections 2024-05-27T00:16:52.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:51.997+0000 7f072e200640 1 -- 172.21.15.98:0/2434426710 wait complete. 2024-05-27T00:16:52.029 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-05-27T00:16:54.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:16:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.253+0000 7f93ace00640 1 Processor -- start 2024-05-27T00:16:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93ace00640 1 -- start start 2024-05-27T00:16:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93ace00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93a806e7a0 0x7f93a8164620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93ace00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f93a8164b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93ace00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93a80c6a10 con 0x7f93a806eb80 2024-05-27T00:16:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93ace00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f93a80c6cf0 con 0x7f93a806e7a0 2024-05-27T00:16:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93ace00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f93a80c6fd0 con 0x7f93a808cbd0 2024-05-27T00:16:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93a7e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f93a806eb80 0x7f93a816ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36420/0 (socket says 172.21.15.98:36420) 2024-05-27T00:16:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93a7400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93a806e7a0 0x7f93a8164620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:54.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93a7e00640 1 -- 172.21.15.98:0/1255251815 learned_addr learned my addr 172.21.15.98:0/1255251815 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93a6a00640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f93a8164b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.257+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1623825379 0 0) 0x7f93a80c6a10 con 0x7f93a806eb80 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f938c007940 con 0x7f93a806eb80 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1933378933 0 0) 0x7f938c007940 con 0x7f93a806eb80 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 msgr2=0x7f93a8164b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f93a8164b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 >> v2:172.21.15.98:3300/0 conn(0x7f93a806e7a0 msgr2=0x7f93a8164620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 --2- 172.21.15.98:0/1255251815 >> v2:172.21.15.98:3300/0 conn(0x7f93a806e7a0 0x7f93a8164620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a80c72b0 con 0x7f93a806eb80 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f93a0001a00 con 0x7f93a806eb80 2024-05-27T00:16:54.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 >> v1:172.21.15.98:6789/0 conn(0x7f93a806eb80 legacy=0x7f93a816ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f938c007dd0 0x7f938c0081f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f938c008c20 0x7f938c0090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f938c0095e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93a80c6cf0 con 0x7f938c008c20 2024-05-27T00:16:54.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f93a80c6fd0 con 0x7f938c007dd0 2024-05-27T00:16:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f938c00fe90 con 0x7f93a808cbd0 2024-05-27T00:16:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a7400640 1 --2- 172.21.15.98:0/1255251815 >> v2:172.21.15.98:3300/0 conn(0x7f93a806e7a0 0x7f93a8164620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:16:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a7e00640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f938c0095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a7400640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f938c007dd0 0x7f938c0081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a6a00640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f938c008c20 0x7f938c0090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a7e00640 1 -- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f938c007dd0 msgr2=0x7f938c0081f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a7e00640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f938c007dd0 0x7f938c0081f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a7e00640 1 -- 172.21.15.98:0/1255251815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f938c008c20 msgr2=0x7f938c0090a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:16:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a7e00640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f938c008c20 0x7f938c0090a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.449+0000 7f93a7e00640 1 -- 172.21.15.98:0/1255251815 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a80c72b0 con 0x7f93a808cbd0 2024-05-27T00:16:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93a7e00640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f938c0095e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f93a806efa0 tx=0x7f93a001d3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f90fd19516fa517 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a000a500 con 0x7f93a808cbd0 2024-05-27T00:16:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93a00072d0 con 0x7f93a808cbd0 2024-05-27T00:16:54.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93a6000640 1 -- 172.21.15.98:0/1255251815 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a00076a0 con 0x7f93a808cbd0 2024-05-27T00:16:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93ace00640 1 -- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 msgr2=0x7f938c0095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93ace00640 1 --2- 172.21.15.98:0/1255251815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f938c0095e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f93a806efa0 tx=0x7f93a001d3c0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93a7e00640 1 -- 172.21.15.98:0/1255251815 reap_dead start 2024-05-27T00:16:54.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93ace00640 1 -- 172.21.15.98:0/1255251815 shutdown_connections 2024-05-27T00:16:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93ace00640 1 -- 172.21.15.98:0/1255251815 >> 172.21.15.98:0/1255251815 conn(0x7f93a80cbe00 msgr2=0x7f93a80cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93ace00640 1 -- 172.21.15.98:0/1255251815 shutdown_connections 2024-05-27T00:16:54.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.457+0000 7f93ace00640 1 -- 172.21.15.98:0/1255251815 wait complete. 2024-05-27T00:16:54.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93ace00640 1 Processor -- start 2024-05-27T00:16:54.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93ace00640 1 -- start start 2024-05-27T00:16:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93ace00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f93a816cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93ace00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f93a8153bf0 0x7f93a8168ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93ace00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a8072f90 0x7f93a81690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93ace00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f93a806f410 con 0x7f93a8072f90 2024-05-27T00:16:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93ace00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f93a8164b60 con 0x7f93a8153bf0 2024-05-27T00:16:54.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93ace00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f93a81596d0 con 0x7f93a808cbd0 2024-05-27T00:16:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93a7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f93a816cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93a7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f93a816cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:37750/0 (socket says 172.21.15.98:37750) 2024-05-27T00:16:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93a7400640 1 -- 172.21.15.98:0/1235540354 learned_addr learned my addr 172.21.15.98:0/1235540354 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:16:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.461+0000 7f93a7e00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a8072f90 0x7f93a81690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:54.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93a6a00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f93a8153bf0 0x7f93a8168ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93a7400640 1 -- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f93a8153bf0 msgr2=0x7f93a8168ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93a7400640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f93a8153bf0 0x7f93a8168ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93a7400640 1 -- 172.21.15.98:0/1235540354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a8072f90 msgr2=0x7f93a81690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93a7400640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a8072f90 0x7f93a81690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93a7400640 1 -- 172.21.15.98:0/1235540354 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a816a4d0 con 0x7f93a808cbd0 2024-05-27T00:16:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93a6a00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f93a8153bf0 0x7f93a8168ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:16:54.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93a7400640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f93a816cd20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93980038d0 tx=0x7f9398003d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f939f400640 1 -- 172.21.15.98:0/1235540354 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939801b020 con 0x7f93a808cbd0 2024-05-27T00:16:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f939f400640 1 -- 172.21.15.98:0/1235540354 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9398011bb0 con 0x7f93a808cbd0 2024-05-27T00:16:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f939f400640 1 -- 172.21.15.98:0/1235540354 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9398017450 con 0x7f93a808cbd0 2024-05-27T00:16:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93ace00640 1 -- 172.21.15.98:0/1235540354 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93a8159110 con 0x7f93a808cbd0 2024-05-27T00:16:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93ace00640 1 -- 172.21.15.98:0/1235540354 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93a816d870 con 0x7f93a808cbd0 2024-05-27T00:16:54.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.465+0000 7f93ace00640 1 -- 172.21.15.98:0/1235540354 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93a8075c40 con 0x7f93a808cbd0 2024-05-27T00:16:54.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.469+0000 7f939f400640 1 -- 172.21.15.98:0/1235540354 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f939801e020 con 0x7f93a808cbd0 2024-05-27T00:16:54.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.469+0000 7f939f400640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93880678c0 0x7f9388069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:16:54.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.469+0000 7f939f400640 1 -- 172.21.15.98:0/1235540354 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5911+0+0 (secure 0 0 0) 0x7f9398086f50 con 0x7f93a808cbd0 2024-05-27T00:16:54.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.473+0000 7f93a6a00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93880678c0 0x7f9388069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:16:54.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.473+0000 7f93a6a00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93880678c0 0x7f9388069d80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f93a815a690 tx=0x7f9394008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:16:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.481+0000 7f939f400640 1 -- 172.21.15.98:0/1235540354 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9398025020 con 0x7f93a808cbd0 2024-05-27T00:16:54.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.881+0000 7f93ace00640 1 -- 172.21.15.98:0/1235540354 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93a80c2ac0 con 0x7f93a808cbd0 2024-05-27T00:16:54.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.881+0000 7f939f400640 1 -- 172.21.15.98:0/1235540354 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6838 (secure 0 0 0) 0x7f9398057890 con 0x7f93a808cbd0 2024-05-27T00:16:54.884 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:16:54.884 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:16:54.884 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-05-27T00:16:54.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 -- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93880678c0 msgr2=0x7f9388069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:54.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93880678c0 0x7f9388069d80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f93a815a690 tx=0x7f9394008040 comp rx=0 tx=0).stop 2024-05-27T00:16:54.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 -- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 msgr2=0x7f93a816cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:16:54.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f93a816cd20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93980038d0 tx=0x7f9398003d90 comp rx=0 tx=0).stop 2024-05-27T00:16:54.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 -- 172.21.15.98:0/1235540354 shutdown_connections 2024-05-27T00:16:54.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93a8072f90 0x7f93a81690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f93880678c0 0x7f9388069d80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f93a8153bf0 0x7f93a8168ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 --2- 172.21.15.98:0/1235540354 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f93a808cbd0 0x7f93a816cd20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:16:54.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 -- 172.21.15.98:0/1235540354 >> 172.21.15.98:0/1235540354 conn(0x7f93a80cbe00 msgr2=0x7f93a8061e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:16:54.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 -- 172.21.15.98:0/1235540354 shutdown_connections 2024-05-27T00:16:54.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:16:54.885+0000 7f93ace00640 1 -- 172.21.15.98:0/1235540354 wait complete. 2024-05-27T00:16:54.917 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-05-27T00:16:54.917 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 74 secs before thrashing 2024-05-27T00:17:01.791 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 69% (8765/12563) Updating files: 70% (8795/12563) Updating files: 71% (8920/12563) Updating files: 72% (9046/12563) Updating files: 73% (9171/12563) Updating files: 74% (9297/12563) Updating files: 75% (9423/12563) Updating files: 76% (9548/12563) Updating files: 77% (9674/12563) Updating files: 78% (9800/12563) Updating files: 79% (9925/12563) Updating files: 80% (10051/12563) Updating files: 81% (10177/12563) Updating files: 82% (10302/12563) Updating files: 83% (10428/12563) Updating files: 84% (10553/12563) Updating files: 85% (10679/12563) Updating files: 86% (10805/12563) Updating files: 87% (10930/12563) Updating files: 88% (11056/12563) Updating files: 89% (11182/12563) Updating files: 90% (11307/12563) Updating files: 91% (11433/12563) Updating files: 92% (11558/12563) Updating files: 93% (11684/12563) Updating files: 94% (11810/12563) Updating files: 95% (11935/12563) Updating files: 96% (12061/12563) Updating files: 97% (12187/12563) Updating files: 98% (12312/12563) Updating files: 99% (12438/12563) Updating files: 100% (12563/12563) Updating files: 100% (12563/12563), done. 2024-05-27T00:17:02.594 INFO:tasks.workunit.client.0.smithi098.stderr:Note: switching to '7d7cb32d1d130135a98121a564d709025c2154b6'. 2024-05-27T00:17:02.594 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-27T00:17:02.594 INFO:tasks.workunit.client.0.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr:state without impacting any branches by switching back to a branch. 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr: git switch -c 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr:Or undo this operation with: 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr: git switch - 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-27T00:17:02.595 INFO:tasks.workunit.client.0.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-27T00:17:02.596 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-05-27T00:17:02.596 INFO:tasks.workunit.client.0.smithi098.stderr:HEAD is now at 7d7cb32d1d1 Merge pull request #57636 from rhcs-dashboard/wip-66174-squid 2024-05-27T00:17:02.603 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-27T00:17:02.617 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-27T00:17:02.617 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T00:17:02.618 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-27T00:17:03.377 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-27T00:17:03.458 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-27T00:17:03.511 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T00:17:03.513 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T00:17:03.513 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-27T00:17:03.569 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T00:17:03.574 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:17:03.574 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-27T00:17:03.626 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2024-05-27T00:17:03.627 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2024-05-27T00:17:03.628 DEBUG:teuthology.orchestra.run.smithi098:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-05-27T00:17:03.692 INFO:tasks.workunit.client.0.smithi098.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2024-05-27T00:17:06.691 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir foo 2024-05-27T00:17:06.698 INFO:tasks.workunit.client.0.smithi098.stderr:+ cp linux-2.6.33.tar.bz2 foo 2024-05-27T00:17:06.788 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir foo/.snap/barsnap 2024-05-27T00:17:06.797 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm foo/linux-2.6.33.tar.bz2 2024-05-27T00:17:06.806 INFO:tasks.workunit.client.0.smithi098.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2024-05-27T00:17:07.184 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo passed: files are identical 2024-05-27T00:17:07.185 INFO:tasks.workunit.client.0.smithi098.stdout:passed: files are identical 2024-05-27T00:17:07.185 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir foo/.snap/barsnap 2024-05-27T00:17:07.217 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-05-27T00:17:07.218 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo OK 2024-05-27T00:17:07.219 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T00:17:07.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T00:17:07.291 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2024-05-27T00:17:07.292 DEBUG:teuthology.orchestra.run.smithi098:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2024-05-27T00:17:07.350 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 1 2024-05-27T00:17:07.351 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 2 2024-05-27T00:17:07.352 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 3 2024-05-27T00:17:07.353 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' -e file4 ']' 2024-05-27T00:17:07.353 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/snap1 2024-05-27T00:17:07.365 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 4 2024-05-27T00:17:07.366 INFO:tasks.workunit.client.0.smithi098.stderr:++ ls 2024-05-27T00:17:07.372 INFO:tasks.workunit.client.0.smithi098.stderr:+ now='file1 2024-05-27T00:17:07.372 INFO:tasks.workunit.client.0.smithi098.stderr:file2 2024-05-27T00:17:07.372 INFO:tasks.workunit.client.0.smithi098.stderr:file3 2024-05-27T00:17:07.372 INFO:tasks.workunit.client.0.smithi098.stderr:file4' 2024-05-27T00:17:07.373 INFO:tasks.workunit.client.0.smithi098.stderr:++ ls .snap/snap1 2024-05-27T00:17:07.379 INFO:tasks.workunit.client.0.smithi098.stderr:+ then='file1 2024-05-27T00:17:07.380 INFO:tasks.workunit.client.0.smithi098.stderr:file2 2024-05-27T00:17:07.380 INFO:tasks.workunit.client.0.smithi098.stderr:file3' 2024-05-27T00:17:07.380 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir .snap/snap1 2024-05-27T00:17:07.383 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' 'file1 2024-05-27T00:17:07.383 INFO:tasks.workunit.client.0.smithi098.stderr:file2 2024-05-27T00:17:07.384 INFO:tasks.workunit.client.0.smithi098.stderr:file3 2024-05-27T00:17:07.384 INFO:tasks.workunit.client.0.smithi098.stderr:file4' = 'file1 2024-05-27T00:17:07.384 INFO:tasks.workunit.client.0.smithi098.stderr:file2 2024-05-27T00:17:07.384 INFO:tasks.workunit.client.0.smithi098.stderr:file3' ']' 2024-05-27T00:17:07.384 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 1 2024-05-27T00:17:07.393 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 2 2024-05-27T00:17:07.401 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 3 2024-05-27T00:17:07.413 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/snap1 2024-05-27T00:17:07.425 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo 4 2024-05-27T00:17:07.432 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir .snap/snap1 2024-05-27T00:17:07.443 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm file1 file2 file3 file4 2024-05-27T00:17:07.448 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo OK 2024-05-27T00:17:07.449 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-05-27T00:17:07.449 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T00:17:07.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T00:17:07.517 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2024-05-27T00:17:07.517 DEBUG:teuthology.orchestra.run.smithi098:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2024-05-27T00:17:07.578 INFO:tasks.workunit.client.0.smithi098.stdout:Create dir 100 to 199 ... 2024-05-27T00:17:07.582 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 100 2024-05-27T00:17:07.597 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 101 2024-05-27T00:17:07.614 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 102 2024-05-27T00:17:07.626 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 103 2024-05-27T00:17:07.638 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 104 2024-05-27T00:17:07.651 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 105 2024-05-27T00:17:07.663 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 106 2024-05-27T00:17:07.675 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 107 2024-05-27T00:17:07.688 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 108 2024-05-27T00:17:07.701 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 109 2024-05-27T00:17:07.713 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 110 2024-05-27T00:17:07.726 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 111 2024-05-27T00:17:07.739 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 112 2024-05-27T00:17:07.751 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 113 2024-05-27T00:17:07.764 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 114 2024-05-27T00:17:07.776 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 115 2024-05-27T00:17:07.788 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 116 2024-05-27T00:17:07.801 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 117 2024-05-27T00:17:07.813 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 118 2024-05-27T00:17:07.826 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 119 2024-05-27T00:17:07.838 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 120 2024-05-27T00:17:07.850 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 121 2024-05-27T00:17:07.864 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 122 2024-05-27T00:17:07.877 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 123 2024-05-27T00:17:07.890 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 124 2024-05-27T00:17:07.902 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 125 2024-05-27T00:17:07.915 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 126 2024-05-27T00:17:07.928 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 127 2024-05-27T00:17:07.941 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 128 2024-05-27T00:17:07.953 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 129 2024-05-27T00:17:07.966 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 130 2024-05-27T00:17:07.979 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 131 2024-05-27T00:17:07.995 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 132 2024-05-27T00:17:08.004 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 133 2024-05-27T00:17:08.017 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 134 2024-05-27T00:17:08.029 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 135 2024-05-27T00:17:08.042 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 136 2024-05-27T00:17:08.055 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 137 2024-05-27T00:17:08.068 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 138 2024-05-27T00:17:08.081 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 139 2024-05-27T00:17:08.093 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 140 2024-05-27T00:17:08.106 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 141 2024-05-27T00:17:08.118 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 142 2024-05-27T00:17:08.130 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 143 2024-05-27T00:17:08.143 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 144 2024-05-27T00:17:08.156 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 145 2024-05-27T00:17:08.169 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 146 2024-05-27T00:17:08.181 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 147 2024-05-27T00:17:08.198 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 148 2024-05-27T00:17:08.206 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 149 2024-05-27T00:17:08.219 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 150 2024-05-27T00:17:08.233 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 151 2024-05-27T00:17:08.246 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 152 2024-05-27T00:17:08.258 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 153 2024-05-27T00:17:08.270 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 154 2024-05-27T00:17:08.283 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 155 2024-05-27T00:17:08.295 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 156 2024-05-27T00:17:08.307 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 157 2024-05-27T00:17:08.320 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 158 2024-05-27T00:17:08.332 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 159 2024-05-27T00:17:08.345 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 160 2024-05-27T00:17:08.358 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 161 2024-05-27T00:17:08.372 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 162 2024-05-27T00:17:08.389 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 163 2024-05-27T00:17:08.406 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 164 2024-05-27T00:17:08.420 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 165 2024-05-27T00:17:08.432 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 166 2024-05-27T00:17:08.450 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 167 2024-05-27T00:17:08.463 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 168 2024-05-27T00:17:08.477 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 169 2024-05-27T00:17:08.494 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 170 2024-05-27T00:17:08.511 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 171 2024-05-27T00:17:08.528 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 172 2024-05-27T00:17:08.545 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 173 2024-05-27T00:17:08.561 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 174 2024-05-27T00:17:08.578 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 175 2024-05-27T00:17:08.595 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 176 2024-05-27T00:17:08.612 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 177 2024-05-27T00:17:08.625 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 178 2024-05-27T00:17:08.637 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 179 2024-05-27T00:17:08.651 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 180 2024-05-27T00:17:08.662 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 181 2024-05-27T00:17:08.676 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 182 2024-05-27T00:17:08.964 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 183 2024-05-27T00:17:08.978 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 184 2024-05-27T00:17:08.990 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 185 2024-05-27T00:17:09.002 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 186 2024-05-27T00:17:09.017 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 187 2024-05-27T00:17:09.027 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 188 2024-05-27T00:17:09.040 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 189 2024-05-27T00:17:09.052 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 190 2024-05-27T00:17:09.065 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 191 2024-05-27T00:17:09.077 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 192 2024-05-27T00:17:09.090 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 193 2024-05-27T00:17:09.102 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 194 2024-05-27T00:17:09.115 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 195 2024-05-27T00:17:09.127 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 196 2024-05-27T00:17:09.139 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 197 2024-05-27T00:17:09.151 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 198 2024-05-27T00:17:09.163 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 199 2024-05-27T00:17:09.176 INFO:tasks.workunit.client.0.smithi098.stdout:Take first snapshot .snap/test1 2024-05-27T00:17:09.260 INFO:tasks.workunit.client.0.smithi098.stdout:Create dir 200 to 299 ... 2024-05-27T00:17:09.261 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 200 2024-05-27T00:17:09.274 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 201 2024-05-27T00:17:09.286 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 202 2024-05-27T00:17:09.298 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 203 2024-05-27T00:17:09.309 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 204 2024-05-27T00:17:09.321 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 205 2024-05-27T00:17:09.332 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 206 2024-05-27T00:17:09.345 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 207 2024-05-27T00:17:09.356 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 208 2024-05-27T00:17:09.367 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 209 2024-05-27T00:17:09.379 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 210 2024-05-27T00:17:09.390 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 211 2024-05-27T00:17:09.402 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 212 2024-05-27T00:17:09.414 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 213 2024-05-27T00:17:09.425 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 214 2024-05-27T00:17:09.437 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 215 2024-05-27T00:17:09.450 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 216 2024-05-27T00:17:09.462 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 217 2024-05-27T00:17:09.475 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 218 2024-05-27T00:17:09.486 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 219 2024-05-27T00:17:09.497 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 220 2024-05-27T00:17:09.509 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 221 2024-05-27T00:17:09.522 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 222 2024-05-27T00:17:09.534 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 223 2024-05-27T00:17:09.547 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 224 2024-05-27T00:17:09.559 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 225 2024-05-27T00:17:09.571 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 226 2024-05-27T00:17:09.583 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 227 2024-05-27T00:17:09.595 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 228 2024-05-27T00:17:09.607 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 229 2024-05-27T00:17:09.620 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 230 2024-05-27T00:17:09.631 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 231 2024-05-27T00:17:09.643 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 232 2024-05-27T00:17:09.655 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 233 2024-05-27T00:17:09.667 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 234 2024-05-27T00:17:09.679 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 235 2024-05-27T00:17:09.690 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 236 2024-05-27T00:17:09.702 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 237 2024-05-27T00:17:09.713 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 238 2024-05-27T00:17:09.725 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 239 2024-05-27T00:17:09.736 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 240 2024-05-27T00:17:09.748 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 241 2024-05-27T00:17:09.760 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 242 2024-05-27T00:17:09.771 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 243 2024-05-27T00:17:09.783 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 244 2024-05-27T00:17:09.794 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 245 2024-05-27T00:17:09.806 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 246 2024-05-27T00:17:09.817 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 247 2024-05-27T00:17:09.829 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 248 2024-05-27T00:17:09.841 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 249 2024-05-27T00:17:09.852 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 250 2024-05-27T00:17:09.868 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 251 2024-05-27T00:17:10.471 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 252 2024-05-27T00:17:10.483 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 253 2024-05-27T00:17:10.495 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 254 2024-05-27T00:17:10.507 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 255 2024-05-27T00:17:10.519 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 256 2024-05-27T00:17:10.531 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 257 2024-05-27T00:17:10.543 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 258 2024-05-27T00:17:10.555 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 259 2024-05-27T00:17:10.567 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 260 2024-05-27T00:17:10.579 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 261 2024-05-27T00:17:10.591 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 262 2024-05-27T00:17:10.602 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 263 2024-05-27T00:17:10.614 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 264 2024-05-27T00:17:10.625 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 265 2024-05-27T00:17:10.638 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 266 2024-05-27T00:17:10.650 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 267 2024-05-27T00:17:10.662 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 268 2024-05-27T00:17:10.674 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 269 2024-05-27T00:17:10.686 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 270 2024-05-27T00:17:10.697 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 271 2024-05-27T00:17:10.709 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 272 2024-05-27T00:17:10.724 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 273 2024-05-27T00:17:10.737 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 274 2024-05-27T00:17:10.750 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 275 2024-05-27T00:17:10.765 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 276 2024-05-27T00:17:10.779 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 277 2024-05-27T00:17:10.792 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 278 2024-05-27T00:17:10.805 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 279 2024-05-27T00:17:10.819 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 280 2024-05-27T00:17:10.833 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 281 2024-05-27T00:17:10.844 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 282 2024-05-27T00:17:10.856 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 283 2024-05-27T00:17:10.868 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 284 2024-05-27T00:17:10.879 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 285 2024-05-27T00:17:10.891 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 286 2024-05-27T00:17:10.902 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 287 2024-05-27T00:17:10.914 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 288 2024-05-27T00:17:10.926 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 289 2024-05-27T00:17:10.937 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 290 2024-05-27T00:17:10.949 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 291 2024-05-27T00:17:10.960 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 292 2024-05-27T00:17:10.972 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 293 2024-05-27T00:17:10.984 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 294 2024-05-27T00:17:10.995 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 295 2024-05-27T00:17:11.007 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 296 2024-05-27T00:17:11.019 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 297 2024-05-27T00:17:11.030 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 298 2024-05-27T00:17:11.042 INFO:tasks.workunit.client.0.smithi098.stdout: create dir 299 2024-05-27T00:17:11.054 INFO:tasks.workunit.client.0.smithi098.stdout:Create a snapshot in every first level dir ... 2024-05-27T00:17:11.173 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/.snap/snap-subdir-test 2024-05-27T00:17:11.187 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/30 file after the snapshot 2024-05-27T00:17:11.190 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/31 file after the snapshot 2024-05-27T00:17:11.198 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/32 file after the snapshot 2024-05-27T00:17:11.199 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/33 file after the snapshot 2024-05-27T00:17:11.200 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/34 file after the snapshot 2024-05-27T00:17:11.201 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/35 file after the snapshot 2024-05-27T00:17:11.203 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/36 file after the snapshot 2024-05-27T00:17:11.204 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/37 file after the snapshot 2024-05-27T00:17:11.205 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/38 file after the snapshot 2024-05-27T00:17:11.206 INFO:tasks.workunit.client.0.smithi098.stdout: create 100/39 file after the snapshot 2024-05-27T00:17:11.208 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/.snap/snap-subdir-test 2024-05-27T00:17:11.227 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/30 file after the snapshot 2024-05-27T00:17:11.229 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/31 file after the snapshot 2024-05-27T00:17:11.230 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/32 file after the snapshot 2024-05-27T00:17:11.233 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/33 file after the snapshot 2024-05-27T00:17:11.234 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/34 file after the snapshot 2024-05-27T00:17:11.235 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/35 file after the snapshot 2024-05-27T00:17:11.236 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/36 file after the snapshot 2024-05-27T00:17:11.237 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/37 file after the snapshot 2024-05-27T00:17:11.238 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/38 file after the snapshot 2024-05-27T00:17:11.239 INFO:tasks.workunit.client.0.smithi098.stdout: create 101/39 file after the snapshot 2024-05-27T00:17:11.240 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/.snap/snap-subdir-test 2024-05-27T00:17:11.286 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/30 file after the snapshot 2024-05-27T00:17:11.287 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/31 file after the snapshot 2024-05-27T00:17:11.292 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/32 file after the snapshot 2024-05-27T00:17:11.298 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/33 file after the snapshot 2024-05-27T00:17:11.299 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/34 file after the snapshot 2024-05-27T00:17:11.300 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/35 file after the snapshot 2024-05-27T00:17:11.302 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/36 file after the snapshot 2024-05-27T00:17:11.303 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/37 file after the snapshot 2024-05-27T00:17:11.305 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/38 file after the snapshot 2024-05-27T00:17:11.306 INFO:tasks.workunit.client.0.smithi098.stdout: create 102/39 file after the snapshot 2024-05-27T00:17:11.306 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/.snap/snap-subdir-test 2024-05-27T00:17:11.324 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/30 file after the snapshot 2024-05-27T00:17:11.327 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/31 file after the snapshot 2024-05-27T00:17:11.331 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/32 file after the snapshot 2024-05-27T00:17:11.335 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/33 file after the snapshot 2024-05-27T00:17:11.336 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/34 file after the snapshot 2024-05-27T00:17:11.337 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/35 file after the snapshot 2024-05-27T00:17:11.338 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/36 file after the snapshot 2024-05-27T00:17:11.339 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/37 file after the snapshot 2024-05-27T00:17:11.341 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/38 file after the snapshot 2024-05-27T00:17:11.342 INFO:tasks.workunit.client.0.smithi098.stdout: create 103/39 file after the snapshot 2024-05-27T00:17:11.343 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/.snap/snap-subdir-test 2024-05-27T00:17:11.369 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/30 file after the snapshot 2024-05-27T00:17:11.370 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/31 file after the snapshot 2024-05-27T00:17:11.375 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/32 file after the snapshot 2024-05-27T00:17:11.376 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/33 file after the snapshot 2024-05-27T00:17:11.377 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/34 file after the snapshot 2024-05-27T00:17:11.378 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/35 file after the snapshot 2024-05-27T00:17:11.379 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/36 file after the snapshot 2024-05-27T00:17:11.379 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/37 file after the snapshot 2024-05-27T00:17:11.380 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/38 file after the snapshot 2024-05-27T00:17:11.381 INFO:tasks.workunit.client.0.smithi098.stdout: create 104/39 file after the snapshot 2024-05-27T00:17:11.382 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/.snap/snap-subdir-test 2024-05-27T00:17:11.399 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/30 file after the snapshot 2024-05-27T00:17:11.401 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/31 file after the snapshot 2024-05-27T00:17:11.405 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/32 file after the snapshot 2024-05-27T00:17:11.406 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/33 file after the snapshot 2024-05-27T00:17:11.407 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/34 file after the snapshot 2024-05-27T00:17:11.408 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/35 file after the snapshot 2024-05-27T00:17:11.409 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/36 file after the snapshot 2024-05-27T00:17:11.409 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/37 file after the snapshot 2024-05-27T00:17:11.410 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/38 file after the snapshot 2024-05-27T00:17:11.411 INFO:tasks.workunit.client.0.smithi098.stdout: create 105/39 file after the snapshot 2024-05-27T00:17:11.412 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/.snap/snap-subdir-test 2024-05-27T00:17:11.433 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/30 file after the snapshot 2024-05-27T00:17:11.435 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/31 file after the snapshot 2024-05-27T00:17:11.438 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/32 file after the snapshot 2024-05-27T00:17:11.440 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/33 file after the snapshot 2024-05-27T00:17:11.440 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/34 file after the snapshot 2024-05-27T00:17:11.441 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/35 file after the snapshot 2024-05-27T00:17:11.442 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/36 file after the snapshot 2024-05-27T00:17:11.443 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/37 file after the snapshot 2024-05-27T00:17:11.443 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/38 file after the snapshot 2024-05-27T00:17:11.444 INFO:tasks.workunit.client.0.smithi098.stdout: create 106/39 file after the snapshot 2024-05-27T00:17:11.445 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/.snap/snap-subdir-test 2024-05-27T00:17:11.465 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/30 file after the snapshot 2024-05-27T00:17:11.466 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/31 file after the snapshot 2024-05-27T00:17:11.468 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/32 file after the snapshot 2024-05-27T00:17:11.470 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/33 file after the snapshot 2024-05-27T00:17:11.473 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/34 file after the snapshot 2024-05-27T00:17:11.474 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/35 file after the snapshot 2024-05-27T00:17:11.474 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/36 file after the snapshot 2024-05-27T00:17:11.475 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/37 file after the snapshot 2024-05-27T00:17:11.478 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/38 file after the snapshot 2024-05-27T00:17:11.479 INFO:tasks.workunit.client.0.smithi098.stdout: create 107/39 file after the snapshot 2024-05-27T00:17:11.479 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/.snap/snap-subdir-test 2024-05-27T00:17:11.502 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/30 file after the snapshot 2024-05-27T00:17:11.504 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/31 file after the snapshot 2024-05-27T00:17:11.508 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/32 file after the snapshot 2024-05-27T00:17:11.514 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/33 file after the snapshot 2024-05-27T00:17:11.515 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/34 file after the snapshot 2024-05-27T00:17:11.517 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/35 file after the snapshot 2024-05-27T00:17:11.518 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/36 file after the snapshot 2024-05-27T00:17:11.519 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/37 file after the snapshot 2024-05-27T00:17:11.520 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/38 file after the snapshot 2024-05-27T00:17:11.522 INFO:tasks.workunit.client.0.smithi098.stdout: create 108/39 file after the snapshot 2024-05-27T00:17:11.523 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/.snap/snap-subdir-test 2024-05-27T00:17:11.543 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/30 file after the snapshot 2024-05-27T00:17:11.544 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/31 file after the snapshot 2024-05-27T00:17:11.547 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/32 file after the snapshot 2024-05-27T00:17:11.551 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/33 file after the snapshot 2024-05-27T00:17:11.551 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/34 file after the snapshot 2024-05-27T00:17:11.552 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/35 file after the snapshot 2024-05-27T00:17:11.553 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/36 file after the snapshot 2024-05-27T00:17:11.554 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/37 file after the snapshot 2024-05-27T00:17:11.555 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/38 file after the snapshot 2024-05-27T00:17:11.555 INFO:tasks.workunit.client.0.smithi098.stdout: create 109/39 file after the snapshot 2024-05-27T00:17:11.556 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/.snap/snap-subdir-test 2024-05-27T00:17:11.578 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/30 file after the snapshot 2024-05-27T00:17:11.579 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/31 file after the snapshot 2024-05-27T00:17:11.580 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/32 file after the snapshot 2024-05-27T00:17:11.581 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/33 file after the snapshot 2024-05-27T00:17:11.583 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/34 file after the snapshot 2024-05-27T00:17:11.585 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/35 file after the snapshot 2024-05-27T00:17:11.586 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/36 file after the snapshot 2024-05-27T00:17:11.587 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/37 file after the snapshot 2024-05-27T00:17:11.588 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/38 file after the snapshot 2024-05-27T00:17:11.588 INFO:tasks.workunit.client.0.smithi098.stdout: create 110/39 file after the snapshot 2024-05-27T00:17:11.589 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/.snap/snap-subdir-test 2024-05-27T00:17:11.609 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/30 file after the snapshot 2024-05-27T00:17:11.610 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/31 file after the snapshot 2024-05-27T00:17:11.612 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/32 file after the snapshot 2024-05-27T00:17:11.615 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/33 file after the snapshot 2024-05-27T00:17:11.616 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/34 file after the snapshot 2024-05-27T00:17:11.617 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/35 file after the snapshot 2024-05-27T00:17:11.617 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/36 file after the snapshot 2024-05-27T00:17:11.618 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/37 file after the snapshot 2024-05-27T00:17:11.619 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/38 file after the snapshot 2024-05-27T00:17:11.620 INFO:tasks.workunit.client.0.smithi098.stdout: create 111/39 file after the snapshot 2024-05-27T00:17:11.621 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/.snap/snap-subdir-test 2024-05-27T00:17:11.642 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/30 file after the snapshot 2024-05-27T00:17:11.643 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/31 file after the snapshot 2024-05-27T00:17:11.644 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/32 file after the snapshot 2024-05-27T00:17:11.647 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/33 file after the snapshot 2024-05-27T00:17:11.649 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/34 file after the snapshot 2024-05-27T00:17:11.650 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/35 file after the snapshot 2024-05-27T00:17:11.651 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/36 file after the snapshot 2024-05-27T00:17:11.651 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/37 file after the snapshot 2024-05-27T00:17:11.652 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/38 file after the snapshot 2024-05-27T00:17:11.653 INFO:tasks.workunit.client.0.smithi098.stdout: create 112/39 file after the snapshot 2024-05-27T00:17:11.654 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/.snap/snap-subdir-test 2024-05-27T00:17:11.673 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/30 file after the snapshot 2024-05-27T00:17:11.674 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/31 file after the snapshot 2024-05-27T00:17:11.678 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/32 file after the snapshot 2024-05-27T00:17:11.685 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/33 file after the snapshot 2024-05-27T00:17:11.687 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/34 file after the snapshot 2024-05-27T00:17:11.688 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/35 file after the snapshot 2024-05-27T00:17:11.689 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/36 file after the snapshot 2024-05-27T00:17:11.690 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/37 file after the snapshot 2024-05-27T00:17:11.692 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/38 file after the snapshot 2024-05-27T00:17:11.693 INFO:tasks.workunit.client.0.smithi098.stdout: create 113/39 file after the snapshot 2024-05-27T00:17:11.695 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/.snap/snap-subdir-test 2024-05-27T00:17:11.714 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/30 file after the snapshot 2024-05-27T00:17:11.715 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/31 file after the snapshot 2024-05-27T00:17:11.719 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/32 file after the snapshot 2024-05-27T00:17:11.721 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/33 file after the snapshot 2024-05-27T00:17:11.722 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/34 file after the snapshot 2024-05-27T00:17:11.723 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/35 file after the snapshot 2024-05-27T00:17:11.724 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/36 file after the snapshot 2024-05-27T00:17:11.725 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/37 file after the snapshot 2024-05-27T00:17:11.725 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/38 file after the snapshot 2024-05-27T00:17:11.726 INFO:tasks.workunit.client.0.smithi098.stdout: create 114/39 file after the snapshot 2024-05-27T00:17:11.727 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/.snap/snap-subdir-test 2024-05-27T00:17:11.746 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/30 file after the snapshot 2024-05-27T00:17:11.746 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/31 file after the snapshot 2024-05-27T00:17:11.747 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/32 file after the snapshot 2024-05-27T00:17:11.749 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/33 file after the snapshot 2024-05-27T00:17:11.750 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/34 file after the snapshot 2024-05-27T00:17:11.752 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/35 file after the snapshot 2024-05-27T00:17:11.754 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/36 file after the snapshot 2024-05-27T00:17:11.755 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/37 file after the snapshot 2024-05-27T00:17:11.756 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/38 file after the snapshot 2024-05-27T00:17:11.757 INFO:tasks.workunit.client.0.smithi098.stdout: create 115/39 file after the snapshot 2024-05-27T00:17:11.757 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/.snap/snap-subdir-test 2024-05-27T00:17:11.777 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/30 file after the snapshot 2024-05-27T00:17:11.779 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/31 file after the snapshot 2024-05-27T00:17:11.783 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/32 file after the snapshot 2024-05-27T00:17:11.784 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/33 file after the snapshot 2024-05-27T00:17:11.784 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/34 file after the snapshot 2024-05-27T00:17:11.785 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/35 file after the snapshot 2024-05-27T00:17:11.786 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/36 file after the snapshot 2024-05-27T00:17:11.787 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/37 file after the snapshot 2024-05-27T00:17:11.788 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/38 file after the snapshot 2024-05-27T00:17:11.789 INFO:tasks.workunit.client.0.smithi098.stdout: create 116/39 file after the snapshot 2024-05-27T00:17:11.789 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/.snap/snap-subdir-test 2024-05-27T00:17:11.813 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/30 file after the snapshot 2024-05-27T00:17:11.814 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/31 file after the snapshot 2024-05-27T00:17:11.818 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/32 file after the snapshot 2024-05-27T00:17:11.819 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/33 file after the snapshot 2024-05-27T00:17:11.820 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/34 file after the snapshot 2024-05-27T00:17:11.820 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/35 file after the snapshot 2024-05-27T00:17:11.821 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/36 file after the snapshot 2024-05-27T00:17:11.822 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/37 file after the snapshot 2024-05-27T00:17:11.823 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/38 file after the snapshot 2024-05-27T00:17:11.823 INFO:tasks.workunit.client.0.smithi098.stdout: create 117/39 file after the snapshot 2024-05-27T00:17:11.824 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/.snap/snap-subdir-test 2024-05-27T00:17:11.842 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/30 file after the snapshot 2024-05-27T00:17:11.843 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/31 file after the snapshot 2024-05-27T00:17:11.844 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/32 file after the snapshot 2024-05-27T00:17:11.845 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/33 file after the snapshot 2024-05-27T00:17:11.848 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/34 file after the snapshot 2024-05-27T00:17:11.850 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/35 file after the snapshot 2024-05-27T00:17:11.851 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/36 file after the snapshot 2024-05-27T00:17:11.852 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/37 file after the snapshot 2024-05-27T00:17:11.853 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/38 file after the snapshot 2024-05-27T00:17:11.854 INFO:tasks.workunit.client.0.smithi098.stdout: create 118/39 file after the snapshot 2024-05-27T00:17:11.854 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/.snap/snap-subdir-test 2024-05-27T00:17:11.873 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/30 file after the snapshot 2024-05-27T00:17:11.874 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/31 file after the snapshot 2024-05-27T00:17:11.875 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/32 file after the snapshot 2024-05-27T00:17:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/33 file after the snapshot 2024-05-27T00:17:11.882 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/34 file after the snapshot 2024-05-27T00:17:11.883 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/35 file after the snapshot 2024-05-27T00:17:11.884 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/36 file after the snapshot 2024-05-27T00:17:11.885 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/37 file after the snapshot 2024-05-27T00:17:11.886 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/38 file after the snapshot 2024-05-27T00:17:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: create 119/39 file after the snapshot 2024-05-27T00:17:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/.snap/snap-subdir-test 2024-05-27T00:17:11.906 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/30 file after the snapshot 2024-05-27T00:17:11.908 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/31 file after the snapshot 2024-05-27T00:17:11.909 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/32 file after the snapshot 2024-05-27T00:17:11.913 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/33 file after the snapshot 2024-05-27T00:17:11.915 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/34 file after the snapshot 2024-05-27T00:17:11.916 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/35 file after the snapshot 2024-05-27T00:17:11.917 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/36 file after the snapshot 2024-05-27T00:17:11.918 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/37 file after the snapshot 2024-05-27T00:17:11.918 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/38 file after the snapshot 2024-05-27T00:17:11.919 INFO:tasks.workunit.client.0.smithi098.stdout: create 120/39 file after the snapshot 2024-05-27T00:17:11.920 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/.snap/snap-subdir-test 2024-05-27T00:17:11.939 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/30 file after the snapshot 2024-05-27T00:17:11.940 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/31 file after the snapshot 2024-05-27T00:17:11.941 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/32 file after the snapshot 2024-05-27T00:17:11.941 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/33 file after the snapshot 2024-05-27T00:17:11.943 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/34 file after the snapshot 2024-05-27T00:17:11.947 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/35 file after the snapshot 2024-05-27T00:17:11.948 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/36 file after the snapshot 2024-05-27T00:17:11.949 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/37 file after the snapshot 2024-05-27T00:17:11.950 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/38 file after the snapshot 2024-05-27T00:17:11.950 INFO:tasks.workunit.client.0.smithi098.stdout: create 121/39 file after the snapshot 2024-05-27T00:17:11.951 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/.snap/snap-subdir-test 2024-05-27T00:17:11.970 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/30 file after the snapshot 2024-05-27T00:17:11.971 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/31 file after the snapshot 2024-05-27T00:17:11.972 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/32 file after the snapshot 2024-05-27T00:17:11.974 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/33 file after the snapshot 2024-05-27T00:17:11.978 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/34 file after the snapshot 2024-05-27T00:17:11.980 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/35 file after the snapshot 2024-05-27T00:17:11.981 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/36 file after the snapshot 2024-05-27T00:17:11.981 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/37 file after the snapshot 2024-05-27T00:17:11.982 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/38 file after the snapshot 2024-05-27T00:17:11.983 INFO:tasks.workunit.client.0.smithi098.stdout: create 122/39 file after the snapshot 2024-05-27T00:17:11.984 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/.snap/snap-subdir-test 2024-05-27T00:17:12.005 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/30 file after the snapshot 2024-05-27T00:17:12.005 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/31 file after the snapshot 2024-05-27T00:17:12.006 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/32 file after the snapshot 2024-05-27T00:17:12.008 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/33 file after the snapshot 2024-05-27T00:17:12.010 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/34 file after the snapshot 2024-05-27T00:17:12.011 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/35 file after the snapshot 2024-05-27T00:17:12.012 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/36 file after the snapshot 2024-05-27T00:17:12.013 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/37 file after the snapshot 2024-05-27T00:17:12.013 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/38 file after the snapshot 2024-05-27T00:17:12.014 INFO:tasks.workunit.client.0.smithi098.stdout: create 123/39 file after the snapshot 2024-05-27T00:17:12.015 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/.snap/snap-subdir-test 2024-05-27T00:17:12.034 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/30 file after the snapshot 2024-05-27T00:17:12.036 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/31 file after the snapshot 2024-05-27T00:17:12.041 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/32 file after the snapshot 2024-05-27T00:17:12.041 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/33 file after the snapshot 2024-05-27T00:17:12.042 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/34 file after the snapshot 2024-05-27T00:17:12.043 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/35 file after the snapshot 2024-05-27T00:17:12.044 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/36 file after the snapshot 2024-05-27T00:17:12.045 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/37 file after the snapshot 2024-05-27T00:17:12.045 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/38 file after the snapshot 2024-05-27T00:17:12.046 INFO:tasks.workunit.client.0.smithi098.stdout: create 124/39 file after the snapshot 2024-05-27T00:17:12.047 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/.snap/snap-subdir-test 2024-05-27T00:17:12.066 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/30 file after the snapshot 2024-05-27T00:17:12.067 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/31 file after the snapshot 2024-05-27T00:17:12.068 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/32 file after the snapshot 2024-05-27T00:17:12.070 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/33 file after the snapshot 2024-05-27T00:17:12.073 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/34 file after the snapshot 2024-05-27T00:17:12.074 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/35 file after the snapshot 2024-05-27T00:17:12.075 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/36 file after the snapshot 2024-05-27T00:17:12.076 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/37 file after the snapshot 2024-05-27T00:17:12.076 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/38 file after the snapshot 2024-05-27T00:17:12.077 INFO:tasks.workunit.client.0.smithi098.stdout: create 125/39 file after the snapshot 2024-05-27T00:17:12.078 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/.snap/snap-subdir-test 2024-05-27T00:17:12.097 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/30 file after the snapshot 2024-05-27T00:17:12.098 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/31 file after the snapshot 2024-05-27T00:17:12.100 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/32 file after the snapshot 2024-05-27T00:17:12.104 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/33 file after the snapshot 2024-05-27T00:17:12.106 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/34 file after the snapshot 2024-05-27T00:17:12.106 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/35 file after the snapshot 2024-05-27T00:17:12.107 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/36 file after the snapshot 2024-05-27T00:17:12.108 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/37 file after the snapshot 2024-05-27T00:17:12.109 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/38 file after the snapshot 2024-05-27T00:17:12.109 INFO:tasks.workunit.client.0.smithi098.stdout: create 126/39 file after the snapshot 2024-05-27T00:17:12.110 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/.snap/snap-subdir-test 2024-05-27T00:17:12.129 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/30 file after the snapshot 2024-05-27T00:17:12.130 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/31 file after the snapshot 2024-05-27T00:17:12.131 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/32 file after the snapshot 2024-05-27T00:17:12.133 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/33 file after the snapshot 2024-05-27T00:17:12.135 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/34 file after the snapshot 2024-05-27T00:17:12.137 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/35 file after the snapshot 2024-05-27T00:17:12.138 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/36 file after the snapshot 2024-05-27T00:17:12.139 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/37 file after the snapshot 2024-05-27T00:17:12.140 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/38 file after the snapshot 2024-05-27T00:17:12.141 INFO:tasks.workunit.client.0.smithi098.stdout: create 127/39 file after the snapshot 2024-05-27T00:17:12.141 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/.snap/snap-subdir-test 2024-05-27T00:17:12.162 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/30 file after the snapshot 2024-05-27T00:17:12.163 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/31 file after the snapshot 2024-05-27T00:17:12.164 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/32 file after the snapshot 2024-05-27T00:17:12.171 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/33 file after the snapshot 2024-05-27T00:17:12.175 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/34 file after the snapshot 2024-05-27T00:17:12.176 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/35 file after the snapshot 2024-05-27T00:17:12.177 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/36 file after the snapshot 2024-05-27T00:17:12.178 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/37 file after the snapshot 2024-05-27T00:17:12.178 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/38 file after the snapshot 2024-05-27T00:17:12.179 INFO:tasks.workunit.client.0.smithi098.stdout: create 128/39 file after the snapshot 2024-05-27T00:17:12.180 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/.snap/snap-subdir-test 2024-05-27T00:17:12.200 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/30 file after the snapshot 2024-05-27T00:17:12.203 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/31 file after the snapshot 2024-05-27T00:17:12.211 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/32 file after the snapshot 2024-05-27T00:17:12.212 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/33 file after the snapshot 2024-05-27T00:17:12.213 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/34 file after the snapshot 2024-05-27T00:17:12.215 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/35 file after the snapshot 2024-05-27T00:17:12.216 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/36 file after the snapshot 2024-05-27T00:17:12.217 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/37 file after the snapshot 2024-05-27T00:17:12.218 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/38 file after the snapshot 2024-05-27T00:17:12.220 INFO:tasks.workunit.client.0.smithi098.stdout: create 129/39 file after the snapshot 2024-05-27T00:17:12.221 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/.snap/snap-subdir-test 2024-05-27T00:17:12.244 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/30 file after the snapshot 2024-05-27T00:17:12.249 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/31 file after the snapshot 2024-05-27T00:17:12.252 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/32 file after the snapshot 2024-05-27T00:17:12.253 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/33 file after the snapshot 2024-05-27T00:17:12.254 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/34 file after the snapshot 2024-05-27T00:17:12.255 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/35 file after the snapshot 2024-05-27T00:17:12.256 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/36 file after the snapshot 2024-05-27T00:17:12.257 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/37 file after the snapshot 2024-05-27T00:17:12.257 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/38 file after the snapshot 2024-05-27T00:17:12.258 INFO:tasks.workunit.client.0.smithi098.stdout: create 130/39 file after the snapshot 2024-05-27T00:17:12.259 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/.snap/snap-subdir-test 2024-05-27T00:17:12.280 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/30 file after the snapshot 2024-05-27T00:17:12.283 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/31 file after the snapshot 2024-05-27T00:17:12.286 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/32 file after the snapshot 2024-05-27T00:17:12.287 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/33 file after the snapshot 2024-05-27T00:17:12.288 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/34 file after the snapshot 2024-05-27T00:17:12.289 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/35 file after the snapshot 2024-05-27T00:17:12.289 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/36 file after the snapshot 2024-05-27T00:17:12.291 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/37 file after the snapshot 2024-05-27T00:17:12.291 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/38 file after the snapshot 2024-05-27T00:17:12.292 INFO:tasks.workunit.client.0.smithi098.stdout: create 131/39 file after the snapshot 2024-05-27T00:17:12.293 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/.snap/snap-subdir-test 2024-05-27T00:17:12.314 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/30 file after the snapshot 2024-05-27T00:17:12.315 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/31 file after the snapshot 2024-05-27T00:17:12.316 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/32 file after the snapshot 2024-05-27T00:17:12.321 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/33 file after the snapshot 2024-05-27T00:17:12.322 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/34 file after the snapshot 2024-05-27T00:17:12.323 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/35 file after the snapshot 2024-05-27T00:17:12.324 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/36 file after the snapshot 2024-05-27T00:17:12.324 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/37 file after the snapshot 2024-05-27T00:17:12.325 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/38 file after the snapshot 2024-05-27T00:17:12.326 INFO:tasks.workunit.client.0.smithi098.stdout: create 132/39 file after the snapshot 2024-05-27T00:17:12.327 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/.snap/snap-subdir-test 2024-05-27T00:17:12.353 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/30 file after the snapshot 2024-05-27T00:17:12.354 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/31 file after the snapshot 2024-05-27T00:17:12.357 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/32 file after the snapshot 2024-05-27T00:17:12.365 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/33 file after the snapshot 2024-05-27T00:17:12.366 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/34 file after the snapshot 2024-05-27T00:17:12.368 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/35 file after the snapshot 2024-05-27T00:17:12.370 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/36 file after the snapshot 2024-05-27T00:17:12.371 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/37 file after the snapshot 2024-05-27T00:17:12.371 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/38 file after the snapshot 2024-05-27T00:17:12.373 INFO:tasks.workunit.client.0.smithi098.stdout: create 133/39 file after the snapshot 2024-05-27T00:17:12.374 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/.snap/snap-subdir-test 2024-05-27T00:17:12.395 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/30 file after the snapshot 2024-05-27T00:17:12.396 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/31 file after the snapshot 2024-05-27T00:17:12.398 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/32 file after the snapshot 2024-05-27T00:17:12.402 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/33 file after the snapshot 2024-05-27T00:17:12.403 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/34 file after the snapshot 2024-05-27T00:17:12.403 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/35 file after the snapshot 2024-05-27T00:17:12.404 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/36 file after the snapshot 2024-05-27T00:17:12.405 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/37 file after the snapshot 2024-05-27T00:17:12.406 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/38 file after the snapshot 2024-05-27T00:17:12.407 INFO:tasks.workunit.client.0.smithi098.stdout: create 134/39 file after the snapshot 2024-05-27T00:17:12.408 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/.snap/snap-subdir-test 2024-05-27T00:17:12.429 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/30 file after the snapshot 2024-05-27T00:17:12.431 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/31 file after the snapshot 2024-05-27T00:17:12.435 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/32 file after the snapshot 2024-05-27T00:17:12.436 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/33 file after the snapshot 2024-05-27T00:17:12.437 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/34 file after the snapshot 2024-05-27T00:17:12.438 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/35 file after the snapshot 2024-05-27T00:17:12.438 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/36 file after the snapshot 2024-05-27T00:17:12.439 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/37 file after the snapshot 2024-05-27T00:17:12.440 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/38 file after the snapshot 2024-05-27T00:17:12.441 INFO:tasks.workunit.client.0.smithi098.stdout: create 135/39 file after the snapshot 2024-05-27T00:17:12.442 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/.snap/snap-subdir-test 2024-05-27T00:17:12.463 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/30 file after the snapshot 2024-05-27T00:17:12.464 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/31 file after the snapshot 2024-05-27T00:17:12.467 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/32 file after the snapshot 2024-05-27T00:17:12.471 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/33 file after the snapshot 2024-05-27T00:17:12.472 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/34 file after the snapshot 2024-05-27T00:17:12.473 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/35 file after the snapshot 2024-05-27T00:17:12.474 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/36 file after the snapshot 2024-05-27T00:17:12.475 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/37 file after the snapshot 2024-05-27T00:17:12.475 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/38 file after the snapshot 2024-05-27T00:17:12.478 INFO:tasks.workunit.client.0.smithi098.stdout: create 136/39 file after the snapshot 2024-05-27T00:17:12.478 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/.snap/snap-subdir-test 2024-05-27T00:17:12.498 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/30 file after the snapshot 2024-05-27T00:17:12.502 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/31 file after the snapshot 2024-05-27T00:17:12.502 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/32 file after the snapshot 2024-05-27T00:17:12.505 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/33 file after the snapshot 2024-05-27T00:17:12.506 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/34 file after the snapshot 2024-05-27T00:17:12.507 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/35 file after the snapshot 2024-05-27T00:17:12.508 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/36 file after the snapshot 2024-05-27T00:17:12.509 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/37 file after the snapshot 2024-05-27T00:17:12.510 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/38 file after the snapshot 2024-05-27T00:17:12.510 INFO:tasks.workunit.client.0.smithi098.stdout: create 137/39 file after the snapshot 2024-05-27T00:17:12.511 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/.snap/snap-subdir-test 2024-05-27T00:17:12.533 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/30 file after the snapshot 2024-05-27T00:17:12.534 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/31 file after the snapshot 2024-05-27T00:17:12.536 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/32 file after the snapshot 2024-05-27T00:17:12.540 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/33 file after the snapshot 2024-05-27T00:17:12.541 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/34 file after the snapshot 2024-05-27T00:17:12.542 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/35 file after the snapshot 2024-05-27T00:17:12.542 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/36 file after the snapshot 2024-05-27T00:17:12.543 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/37 file after the snapshot 2024-05-27T00:17:12.544 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/38 file after the snapshot 2024-05-27T00:17:12.545 INFO:tasks.workunit.client.0.smithi098.stdout: create 138/39 file after the snapshot 2024-05-27T00:17:12.546 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/.snap/snap-subdir-test 2024-05-27T00:17:12.568 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/30 file after the snapshot 2024-05-27T00:17:12.571 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/31 file after the snapshot 2024-05-27T00:17:12.574 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/32 file after the snapshot 2024-05-27T00:17:12.575 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/33 file after the snapshot 2024-05-27T00:17:12.575 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/34 file after the snapshot 2024-05-27T00:17:12.576 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/35 file after the snapshot 2024-05-27T00:17:12.577 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/36 file after the snapshot 2024-05-27T00:17:12.578 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/37 file after the snapshot 2024-05-27T00:17:12.579 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/38 file after the snapshot 2024-05-27T00:17:12.580 INFO:tasks.workunit.client.0.smithi098.stdout: create 139/39 file after the snapshot 2024-05-27T00:17:12.580 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/.snap/snap-subdir-test 2024-05-27T00:17:12.602 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/30 file after the snapshot 2024-05-27T00:17:12.603 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/31 file after the snapshot 2024-05-27T00:17:12.605 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/32 file after the snapshot 2024-05-27T00:17:12.607 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/33 file after the snapshot 2024-05-27T00:17:12.610 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/34 file after the snapshot 2024-05-27T00:17:12.611 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/35 file after the snapshot 2024-05-27T00:17:12.611 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/36 file after the snapshot 2024-05-27T00:17:12.612 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/37 file after the snapshot 2024-05-27T00:17:12.612 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/38 file after the snapshot 2024-05-27T00:17:12.614 INFO:tasks.workunit.client.0.smithi098.stdout: create 140/39 file after the snapshot 2024-05-27T00:17:12.614 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/.snap/snap-subdir-test 2024-05-27T00:17:12.636 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/30 file after the snapshot 2024-05-27T00:17:12.637 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/31 file after the snapshot 2024-05-27T00:17:12.638 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/32 file after the snapshot 2024-05-27T00:17:12.641 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/33 file after the snapshot 2024-05-27T00:17:12.643 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/34 file after the snapshot 2024-05-27T00:17:12.644 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/35 file after the snapshot 2024-05-27T00:17:12.644 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/36 file after the snapshot 2024-05-27T00:17:12.645 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/37 file after the snapshot 2024-05-27T00:17:12.646 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/38 file after the snapshot 2024-05-27T00:17:12.647 INFO:tasks.workunit.client.0.smithi098.stdout: create 141/39 file after the snapshot 2024-05-27T00:17:12.648 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/.snap/snap-subdir-test 2024-05-27T00:17:12.668 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/30 file after the snapshot 2024-05-27T00:17:12.669 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/31 file after the snapshot 2024-05-27T00:17:12.670 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/32 file after the snapshot 2024-05-27T00:17:12.673 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/33 file after the snapshot 2024-05-27T00:17:12.676 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/34 file after the snapshot 2024-05-27T00:17:12.676 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/35 file after the snapshot 2024-05-27T00:17:12.677 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/36 file after the snapshot 2024-05-27T00:17:12.678 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/37 file after the snapshot 2024-05-27T00:17:12.679 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/38 file after the snapshot 2024-05-27T00:17:12.680 INFO:tasks.workunit.client.0.smithi098.stdout: create 142/39 file after the snapshot 2024-05-27T00:17:12.680 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/.snap/snap-subdir-test 2024-05-27T00:17:12.701 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/30 file after the snapshot 2024-05-27T00:17:12.702 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/31 file after the snapshot 2024-05-27T00:17:12.706 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/32 file after the snapshot 2024-05-27T00:17:12.707 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/33 file after the snapshot 2024-05-27T00:17:12.707 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/34 file after the snapshot 2024-05-27T00:17:12.709 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/35 file after the snapshot 2024-05-27T00:17:12.710 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/36 file after the snapshot 2024-05-27T00:17:12.711 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/37 file after the snapshot 2024-05-27T00:17:12.712 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/38 file after the snapshot 2024-05-27T00:17:12.713 INFO:tasks.workunit.client.0.smithi098.stdout: create 143/39 file after the snapshot 2024-05-27T00:17:12.714 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/.snap/snap-subdir-test 2024-05-27T00:17:12.734 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/30 file after the snapshot 2024-05-27T00:17:12.735 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/31 file after the snapshot 2024-05-27T00:17:12.737 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/32 file after the snapshot 2024-05-27T00:17:12.741 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/33 file after the snapshot 2024-05-27T00:17:12.742 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/34 file after the snapshot 2024-05-27T00:17:12.743 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/35 file after the snapshot 2024-05-27T00:17:12.744 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/36 file after the snapshot 2024-05-27T00:17:12.745 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/37 file after the snapshot 2024-05-27T00:17:12.745 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/38 file after the snapshot 2024-05-27T00:17:12.746 INFO:tasks.workunit.client.0.smithi098.stdout: create 144/39 file after the snapshot 2024-05-27T00:17:12.747 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/.snap/snap-subdir-test 2024-05-27T00:17:12.768 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/30 file after the snapshot 2024-05-27T00:17:12.769 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/31 file after the snapshot 2024-05-27T00:17:12.770 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/32 file after the snapshot 2024-05-27T00:17:12.772 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/33 file after the snapshot 2024-05-27T00:17:12.776 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/34 file after the snapshot 2024-05-27T00:17:12.776 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/35 file after the snapshot 2024-05-27T00:17:12.777 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/36 file after the snapshot 2024-05-27T00:17:12.778 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/37 file after the snapshot 2024-05-27T00:17:12.778 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/38 file after the snapshot 2024-05-27T00:17:12.779 INFO:tasks.workunit.client.0.smithi098.stdout: create 145/39 file after the snapshot 2024-05-27T00:17:12.780 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/.snap/snap-subdir-test 2024-05-27T00:17:12.802 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/30 file after the snapshot 2024-05-27T00:17:12.803 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/31 file after the snapshot 2024-05-27T00:17:12.808 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/32 file after the snapshot 2024-05-27T00:17:12.809 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/33 file after the snapshot 2024-05-27T00:17:12.809 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/34 file after the snapshot 2024-05-27T00:17:12.810 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/35 file after the snapshot 2024-05-27T00:17:12.811 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/36 file after the snapshot 2024-05-27T00:17:12.811 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/37 file after the snapshot 2024-05-27T00:17:12.812 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/38 file after the snapshot 2024-05-27T00:17:12.813 INFO:tasks.workunit.client.0.smithi098.stdout: create 146/39 file after the snapshot 2024-05-27T00:17:12.814 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/.snap/snap-subdir-test 2024-05-27T00:17:12.836 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/30 file after the snapshot 2024-05-27T00:17:12.839 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/31 file after the snapshot 2024-05-27T00:17:12.842 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/32 file after the snapshot 2024-05-27T00:17:12.843 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/33 file after the snapshot 2024-05-27T00:17:12.844 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/34 file after the snapshot 2024-05-27T00:17:12.844 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/35 file after the snapshot 2024-05-27T00:17:12.845 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/36 file after the snapshot 2024-05-27T00:17:12.846 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/37 file after the snapshot 2024-05-27T00:17:12.847 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/38 file after the snapshot 2024-05-27T00:17:12.848 INFO:tasks.workunit.client.0.smithi098.stdout: create 147/39 file after the snapshot 2024-05-27T00:17:12.849 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/.snap/snap-subdir-test 2024-05-27T00:17:12.870 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/30 file after the snapshot 2024-05-27T00:17:12.873 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/31 file after the snapshot 2024-05-27T00:17:12.877 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/32 file after the snapshot 2024-05-27T00:17:12.877 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/33 file after the snapshot 2024-05-27T00:17:12.878 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/34 file after the snapshot 2024-05-27T00:17:12.880 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/35 file after the snapshot 2024-05-27T00:17:12.880 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/36 file after the snapshot 2024-05-27T00:17:12.881 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/37 file after the snapshot 2024-05-27T00:17:12.882 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/38 file after the snapshot 2024-05-27T00:17:12.882 INFO:tasks.workunit.client.0.smithi098.stdout: create 148/39 file after the snapshot 2024-05-27T00:17:12.883 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/.snap/snap-subdir-test 2024-05-27T00:17:12.904 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/30 file after the snapshot 2024-05-27T00:17:12.909 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/31 file after the snapshot 2024-05-27T00:17:12.910 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/32 file after the snapshot 2024-05-27T00:17:12.910 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/33 file after the snapshot 2024-05-27T00:17:12.912 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/34 file after the snapshot 2024-05-27T00:17:12.917 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/35 file after the snapshot 2024-05-27T00:17:12.918 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/36 file after the snapshot 2024-05-27T00:17:12.922 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/37 file after the snapshot 2024-05-27T00:17:12.922 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/38 file after the snapshot 2024-05-27T00:17:12.922 INFO:tasks.workunit.client.0.smithi098.stdout: create 149/39 file after the snapshot 2024-05-27T00:17:12.923 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/.snap/snap-subdir-test 2024-05-27T00:17:12.944 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/30 file after the snapshot 2024-05-27T00:17:12.945 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/31 file after the snapshot 2024-05-27T00:17:12.947 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/32 file after the snapshot 2024-05-27T00:17:12.954 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/33 file after the snapshot 2024-05-27T00:17:12.955 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/34 file after the snapshot 2024-05-27T00:17:12.957 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/35 file after the snapshot 2024-05-27T00:17:12.958 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/36 file after the snapshot 2024-05-27T00:17:12.959 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/37 file after the snapshot 2024-05-27T00:17:12.960 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/38 file after the snapshot 2024-05-27T00:17:12.962 INFO:tasks.workunit.client.0.smithi098.stdout: create 150/39 file after the snapshot 2024-05-27T00:17:12.963 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/.snap/snap-subdir-test 2024-05-27T00:17:12.985 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/30 file after the snapshot 2024-05-27T00:17:12.987 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/31 file after the snapshot 2024-05-27T00:17:12.991 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/32 file after the snapshot 2024-05-27T00:17:12.991 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/33 file after the snapshot 2024-05-27T00:17:12.992 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/34 file after the snapshot 2024-05-27T00:17:12.993 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/35 file after the snapshot 2024-05-27T00:17:12.994 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/36 file after the snapshot 2024-05-27T00:17:12.995 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/37 file after the snapshot 2024-05-27T00:17:12.996 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/38 file after the snapshot 2024-05-27T00:17:12.996 INFO:tasks.workunit.client.0.smithi098.stdout: create 151/39 file after the snapshot 2024-05-27T00:17:12.997 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/.snap/snap-subdir-test 2024-05-27T00:17:13.019 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/30 file after the snapshot 2024-05-27T00:17:13.021 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/31 file after the snapshot 2024-05-27T00:17:13.030 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/32 file after the snapshot 2024-05-27T00:17:13.031 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/33 file after the snapshot 2024-05-27T00:17:13.032 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/34 file after the snapshot 2024-05-27T00:17:13.034 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/35 file after the snapshot 2024-05-27T00:17:13.035 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/36 file after the snapshot 2024-05-27T00:17:13.036 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/37 file after the snapshot 2024-05-27T00:17:13.037 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/38 file after the snapshot 2024-05-27T00:17:13.039 INFO:tasks.workunit.client.0.smithi098.stdout: create 152/39 file after the snapshot 2024-05-27T00:17:13.040 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/.snap/snap-subdir-test 2024-05-27T00:17:13.062 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/30 file after the snapshot 2024-05-27T00:17:13.064 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/31 file after the snapshot 2024-05-27T00:17:13.067 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/32 file after the snapshot 2024-05-27T00:17:13.069 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/33 file after the snapshot 2024-05-27T00:17:13.070 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/34 file after the snapshot 2024-05-27T00:17:13.071 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/35 file after the snapshot 2024-05-27T00:17:13.072 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/36 file after the snapshot 2024-05-27T00:17:13.072 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/37 file after the snapshot 2024-05-27T00:17:13.073 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/38 file after the snapshot 2024-05-27T00:17:13.074 INFO:tasks.workunit.client.0.smithi098.stdout: create 153/39 file after the snapshot 2024-05-27T00:17:13.075 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/.snap/snap-subdir-test 2024-05-27T00:17:13.096 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/30 file after the snapshot 2024-05-27T00:17:13.097 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/31 file after the snapshot 2024-05-27T00:17:13.098 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/32 file after the snapshot 2024-05-27T00:17:13.100 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/33 file after the snapshot 2024-05-27T00:17:13.102 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/34 file after the snapshot 2024-05-27T00:17:13.104 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/35 file after the snapshot 2024-05-27T00:17:13.105 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/36 file after the snapshot 2024-05-27T00:17:13.106 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/37 file after the snapshot 2024-05-27T00:17:13.107 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/38 file after the snapshot 2024-05-27T00:17:13.107 INFO:tasks.workunit.client.0.smithi098.stdout: create 154/39 file after the snapshot 2024-05-27T00:17:13.108 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/.snap/snap-subdir-test 2024-05-27T00:17:13.131 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/30 file after the snapshot 2024-05-27T00:17:13.133 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/31 file after the snapshot 2024-05-27T00:17:13.138 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/32 file after the snapshot 2024-05-27T00:17:13.139 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/33 file after the snapshot 2024-05-27T00:17:13.139 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/34 file after the snapshot 2024-05-27T00:17:13.140 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/35 file after the snapshot 2024-05-27T00:17:13.141 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/36 file after the snapshot 2024-05-27T00:17:13.142 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/37 file after the snapshot 2024-05-27T00:17:13.142 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/38 file after the snapshot 2024-05-27T00:17:13.143 INFO:tasks.workunit.client.0.smithi098.stdout: create 155/39 file after the snapshot 2024-05-27T00:17:13.144 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/.snap/snap-subdir-test 2024-05-27T00:17:13.164 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/30 file after the snapshot 2024-05-27T00:17:13.165 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/31 file after the snapshot 2024-05-27T00:17:13.166 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/32 file after the snapshot 2024-05-27T00:17:13.171 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/33 file after the snapshot 2024-05-27T00:17:13.173 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/34 file after the snapshot 2024-05-27T00:17:13.174 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/35 file after the snapshot 2024-05-27T00:17:13.175 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/36 file after the snapshot 2024-05-27T00:17:13.176 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/37 file after the snapshot 2024-05-27T00:17:13.177 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/38 file after the snapshot 2024-05-27T00:17:13.177 INFO:tasks.workunit.client.0.smithi098.stdout: create 156/39 file after the snapshot 2024-05-27T00:17:13.178 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/.snap/snap-subdir-test 2024-05-27T00:17:13.200 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/30 file after the snapshot 2024-05-27T00:17:13.201 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/31 file after the snapshot 2024-05-27T00:17:13.205 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/32 file after the snapshot 2024-05-27T00:17:13.207 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/33 file after the snapshot 2024-05-27T00:17:13.208 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/34 file after the snapshot 2024-05-27T00:17:13.209 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/35 file after the snapshot 2024-05-27T00:17:13.210 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/36 file after the snapshot 2024-05-27T00:17:13.211 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/37 file after the snapshot 2024-05-27T00:17:13.212 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/38 file after the snapshot 2024-05-27T00:17:13.212 INFO:tasks.workunit.client.0.smithi098.stdout: create 157/39 file after the snapshot 2024-05-27T00:17:13.213 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/.snap/snap-subdir-test 2024-05-27T00:17:13.235 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/30 file after the snapshot 2024-05-27T00:17:13.237 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/31 file after the snapshot 2024-05-27T00:17:13.239 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/32 file after the snapshot 2024-05-27T00:17:13.242 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/33 file after the snapshot 2024-05-27T00:17:13.243 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/34 file after the snapshot 2024-05-27T00:17:13.244 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/35 file after the snapshot 2024-05-27T00:17:13.245 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/36 file after the snapshot 2024-05-27T00:17:13.247 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/37 file after the snapshot 2024-05-27T00:17:13.248 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/38 file after the snapshot 2024-05-27T00:17:13.249 INFO:tasks.workunit.client.0.smithi098.stdout: create 158/39 file after the snapshot 2024-05-27T00:17:13.250 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/.snap/snap-subdir-test 2024-05-27T00:17:13.271 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/30 file after the snapshot 2024-05-27T00:17:13.272 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/31 file after the snapshot 2024-05-27T00:17:13.273 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/32 file after the snapshot 2024-05-27T00:17:13.276 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/33 file after the snapshot 2024-05-27T00:17:13.279 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/34 file after the snapshot 2024-05-27T00:17:13.280 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/35 file after the snapshot 2024-05-27T00:17:13.280 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/36 file after the snapshot 2024-05-27T00:17:13.281 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/37 file after the snapshot 2024-05-27T00:17:13.282 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/38 file after the snapshot 2024-05-27T00:17:13.283 INFO:tasks.workunit.client.0.smithi098.stdout: create 159/39 file after the snapshot 2024-05-27T00:17:13.284 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/.snap/snap-subdir-test 2024-05-27T00:17:13.307 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/30 file after the snapshot 2024-05-27T00:17:13.309 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/31 file after the snapshot 2024-05-27T00:17:13.311 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/32 file after the snapshot 2024-05-27T00:17:13.315 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/33 file after the snapshot 2024-05-27T00:17:13.316 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/34 file after the snapshot 2024-05-27T00:17:13.316 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/35 file after the snapshot 2024-05-27T00:17:13.317 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/36 file after the snapshot 2024-05-27T00:17:13.318 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/37 file after the snapshot 2024-05-27T00:17:13.321 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/38 file after the snapshot 2024-05-27T00:17:13.322 INFO:tasks.workunit.client.0.smithi098.stdout: create 160/39 file after the snapshot 2024-05-27T00:17:13.322 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/.snap/snap-subdir-test 2024-05-27T00:17:13.348 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/30 file after the snapshot 2024-05-27T00:17:13.349 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/31 file after the snapshot 2024-05-27T00:17:13.353 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/32 file after the snapshot 2024-05-27T00:17:13.356 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/33 file after the snapshot 2024-05-27T00:17:13.356 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/34 file after the snapshot 2024-05-27T00:17:13.357 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/35 file after the snapshot 2024-05-27T00:17:13.358 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/36 file after the snapshot 2024-05-27T00:17:13.359 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/37 file after the snapshot 2024-05-27T00:17:13.360 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/38 file after the snapshot 2024-05-27T00:17:13.360 INFO:tasks.workunit.client.0.smithi098.stdout: create 161/39 file after the snapshot 2024-05-27T00:17:13.361 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/.snap/snap-subdir-test 2024-05-27T00:17:13.385 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/30 file after the snapshot 2024-05-27T00:17:13.387 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/31 file after the snapshot 2024-05-27T00:17:13.392 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/32 file after the snapshot 2024-05-27T00:17:13.396 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/33 file after the snapshot 2024-05-27T00:17:13.397 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/34 file after the snapshot 2024-05-27T00:17:13.398 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/35 file after the snapshot 2024-05-27T00:17:13.399 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/36 file after the snapshot 2024-05-27T00:17:13.400 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/37 file after the snapshot 2024-05-27T00:17:13.401 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/38 file after the snapshot 2024-05-27T00:17:13.401 INFO:tasks.workunit.client.0.smithi098.stdout: create 162/39 file after the snapshot 2024-05-27T00:17:13.402 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/.snap/snap-subdir-test 2024-05-27T00:17:13.428 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/30 file after the snapshot 2024-05-27T00:17:13.429 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/31 file after the snapshot 2024-05-27T00:17:13.430 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/32 file after the snapshot 2024-05-27T00:17:13.432 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/33 file after the snapshot 2024-05-27T00:17:13.438 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/34 file after the snapshot 2024-05-27T00:17:13.441 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/35 file after the snapshot 2024-05-27T00:17:13.441 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/36 file after the snapshot 2024-05-27T00:17:13.442 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/37 file after the snapshot 2024-05-27T00:17:13.443 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/38 file after the snapshot 2024-05-27T00:17:13.444 INFO:tasks.workunit.client.0.smithi098.stdout: create 163/39 file after the snapshot 2024-05-27T00:17:13.445 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/.snap/snap-subdir-test 2024-05-27T00:17:13.467 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/30 file after the snapshot 2024-05-27T00:17:13.471 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/31 file after the snapshot 2024-05-27T00:17:13.478 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/32 file after the snapshot 2024-05-27T00:17:13.479 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/33 file after the snapshot 2024-05-27T00:17:13.479 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/34 file after the snapshot 2024-05-27T00:17:13.480 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/35 file after the snapshot 2024-05-27T00:17:13.481 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/36 file after the snapshot 2024-05-27T00:17:13.482 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/37 file after the snapshot 2024-05-27T00:17:13.483 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/38 file after the snapshot 2024-05-27T00:17:13.483 INFO:tasks.workunit.client.0.smithi098.stdout: create 164/39 file after the snapshot 2024-05-27T00:17:13.484 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/.snap/snap-subdir-test 2024-05-27T00:17:13.509 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/30 file after the snapshot 2024-05-27T00:17:13.512 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/31 file after the snapshot 2024-05-27T00:17:13.521 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/32 file after the snapshot 2024-05-27T00:17:13.521 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/33 file after the snapshot 2024-05-27T00:17:13.521 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/34 file after the snapshot 2024-05-27T00:17:13.522 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/35 file after the snapshot 2024-05-27T00:17:13.527 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/36 file after the snapshot 2024-05-27T00:17:13.527 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/37 file after the snapshot 2024-05-27T00:17:13.527 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/38 file after the snapshot 2024-05-27T00:17:13.527 INFO:tasks.workunit.client.0.smithi098.stdout: create 165/39 file after the snapshot 2024-05-27T00:17:13.527 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/.snap/snap-subdir-test 2024-05-27T00:17:13.548 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/30 file after the snapshot 2024-05-27T00:17:13.551 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/31 file after the snapshot 2024-05-27T00:17:13.557 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/32 file after the snapshot 2024-05-27T00:17:13.558 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/33 file after the snapshot 2024-05-27T00:17:13.559 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/34 file after the snapshot 2024-05-27T00:17:13.560 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/35 file after the snapshot 2024-05-27T00:17:13.560 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/36 file after the snapshot 2024-05-27T00:17:13.561 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/37 file after the snapshot 2024-05-27T00:17:13.562 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/38 file after the snapshot 2024-05-27T00:17:13.563 INFO:tasks.workunit.client.0.smithi098.stdout: create 166/39 file after the snapshot 2024-05-27T00:17:13.564 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/.snap/snap-subdir-test 2024-05-27T00:17:13.585 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/30 file after the snapshot 2024-05-27T00:17:13.587 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/31 file after the snapshot 2024-05-27T00:17:13.591 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/32 file after the snapshot 2024-05-27T00:17:13.596 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/33 file after the snapshot 2024-05-27T00:17:13.599 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/34 file after the snapshot 2024-05-27T00:17:13.600 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/35 file after the snapshot 2024-05-27T00:17:13.601 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/36 file after the snapshot 2024-05-27T00:17:13.602 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/37 file after the snapshot 2024-05-27T00:17:13.603 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/38 file after the snapshot 2024-05-27T00:17:13.603 INFO:tasks.workunit.client.0.smithi098.stdout: create 167/39 file after the snapshot 2024-05-27T00:17:13.604 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/.snap/snap-subdir-test 2024-05-27T00:17:13.629 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/30 file after the snapshot 2024-05-27T00:17:13.630 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/31 file after the snapshot 2024-05-27T00:17:13.632 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/32 file after the snapshot 2024-05-27T00:17:13.639 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/33 file after the snapshot 2024-05-27T00:17:13.640 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/34 file after the snapshot 2024-05-27T00:17:13.641 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/35 file after the snapshot 2024-05-27T00:17:13.642 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/36 file after the snapshot 2024-05-27T00:17:13.643 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/37 file after the snapshot 2024-05-27T00:17:13.644 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/38 file after the snapshot 2024-05-27T00:17:13.644 INFO:tasks.workunit.client.0.smithi098.stdout: create 168/39 file after the snapshot 2024-05-27T00:17:13.645 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/.snap/snap-subdir-test 2024-05-27T00:17:13.668 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/30 file after the snapshot 2024-05-27T00:17:13.669 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/31 file after the snapshot 2024-05-27T00:17:13.678 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/32 file after the snapshot 2024-05-27T00:17:13.678 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/33 file after the snapshot 2024-05-27T00:17:13.679 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/34 file after the snapshot 2024-05-27T00:17:13.680 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/35 file after the snapshot 2024-05-27T00:17:13.681 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/36 file after the snapshot 2024-05-27T00:17:13.682 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/37 file after the snapshot 2024-05-27T00:17:13.683 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/38 file after the snapshot 2024-05-27T00:17:13.683 INFO:tasks.workunit.client.0.smithi098.stdout: create 169/39 file after the snapshot 2024-05-27T00:17:13.684 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/.snap/snap-subdir-test 2024-05-27T00:17:13.706 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/30 file after the snapshot 2024-05-27T00:17:13.709 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/31 file after the snapshot 2024-05-27T00:17:13.715 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/32 file after the snapshot 2024-05-27T00:17:13.716 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/33 file after the snapshot 2024-05-27T00:17:13.717 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/34 file after the snapshot 2024-05-27T00:17:13.718 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/35 file after the snapshot 2024-05-27T00:17:13.718 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/36 file after the snapshot 2024-05-27T00:17:13.719 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/37 file after the snapshot 2024-05-27T00:17:13.720 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/38 file after the snapshot 2024-05-27T00:17:13.721 INFO:tasks.workunit.client.0.smithi098.stdout: create 170/39 file after the snapshot 2024-05-27T00:17:13.722 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/.snap/snap-subdir-test 2024-05-27T00:17:13.743 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/30 file after the snapshot 2024-05-27T00:17:13.745 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/31 file after the snapshot 2024-05-27T00:17:13.749 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/32 file after the snapshot 2024-05-27T00:17:13.753 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/33 file after the snapshot 2024-05-27T00:17:13.754 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/34 file after the snapshot 2024-05-27T00:17:13.755 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/35 file after the snapshot 2024-05-27T00:17:13.755 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/36 file after the snapshot 2024-05-27T00:17:13.756 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/37 file after the snapshot 2024-05-27T00:17:13.757 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/38 file after the snapshot 2024-05-27T00:17:13.758 INFO:tasks.workunit.client.0.smithi098.stdout: create 171/39 file after the snapshot 2024-05-27T00:17:13.759 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/.snap/snap-subdir-test 2024-05-27T00:17:13.784 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/30 file after the snapshot 2024-05-27T00:17:13.786 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/31 file after the snapshot 2024-05-27T00:17:13.792 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/32 file after the snapshot 2024-05-27T00:17:13.794 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/33 file after the snapshot 2024-05-27T00:17:13.794 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/34 file after the snapshot 2024-05-27T00:17:13.795 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/35 file after the snapshot 2024-05-27T00:17:13.796 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/36 file after the snapshot 2024-05-27T00:17:13.797 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/37 file after the snapshot 2024-05-27T00:17:13.798 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/38 file after the snapshot 2024-05-27T00:17:13.798 INFO:tasks.workunit.client.0.smithi098.stdout: create 172/39 file after the snapshot 2024-05-27T00:17:13.799 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/.snap/snap-subdir-test 2024-05-27T00:17:13.820 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/30 file after the snapshot 2024-05-27T00:17:13.822 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/31 file after the snapshot 2024-05-27T00:17:13.828 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/32 file after the snapshot 2024-05-27T00:17:13.831 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/33 file after the snapshot 2024-05-27T00:17:13.832 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/34 file after the snapshot 2024-05-27T00:17:13.832 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/35 file after the snapshot 2024-05-27T00:17:13.833 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/36 file after the snapshot 2024-05-27T00:17:13.834 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/37 file after the snapshot 2024-05-27T00:17:13.835 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/38 file after the snapshot 2024-05-27T00:17:13.836 INFO:tasks.workunit.client.0.smithi098.stdout: create 173/39 file after the snapshot 2024-05-27T00:17:13.837 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/.snap/snap-subdir-test 2024-05-27T00:17:13.865 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/30 file after the snapshot 2024-05-27T00:17:13.871 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/31 file after the snapshot 2024-05-27T00:17:13.882 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/32 file after the snapshot 2024-05-27T00:17:13.883 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/33 file after the snapshot 2024-05-27T00:17:13.884 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/34 file after the snapshot 2024-05-27T00:17:13.885 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/35 file after the snapshot 2024-05-27T00:17:13.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:17:13.887 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/36 file after the snapshot 2024-05-27T00:17:13.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:17:13.890 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/37 file after the snapshot 2024-05-27T00:17:13.890 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/38 file after the snapshot 2024-05-27T00:17:13.890 INFO:tasks.workunit.client.0.smithi098.stdout: create 174/39 file after the snapshot 2024-05-27T00:17:13.891 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/.snap/snap-subdir-test 2024-05-27T00:17:13.912 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/30 file after the snapshot 2024-05-27T00:17:13.913 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/31 file after the snapshot 2024-05-27T00:17:13.915 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/32 file after the snapshot 2024-05-27T00:17:13.921 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/33 file after the snapshot 2024-05-27T00:17:13.924 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/34 file after the snapshot 2024-05-27T00:17:13.924 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/35 file after the snapshot 2024-05-27T00:17:13.925 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/36 file after the snapshot 2024-05-27T00:17:13.926 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/37 file after the snapshot 2024-05-27T00:17:13.927 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/38 file after the snapshot 2024-05-27T00:17:13.927 INFO:tasks.workunit.client.0.smithi098.stdout: create 175/39 file after the snapshot 2024-05-27T00:17:13.928 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/.snap/snap-subdir-test 2024-05-27T00:17:13.949 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/30 file after the snapshot 2024-05-27T00:17:13.955 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/31 file after the snapshot 2024-05-27T00:17:13.959 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/32 file after the snapshot 2024-05-27T00:17:13.960 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/33 file after the snapshot 2024-05-27T00:17:13.961 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/34 file after the snapshot 2024-05-27T00:17:13.961 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/35 file after the snapshot 2024-05-27T00:17:13.962 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/36 file after the snapshot 2024-05-27T00:17:13.963 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/37 file after the snapshot 2024-05-27T00:17:13.964 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/38 file after the snapshot 2024-05-27T00:17:13.965 INFO:tasks.workunit.client.0.smithi098.stdout: create 176/39 file after the snapshot 2024-05-27T00:17:13.965 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/.snap/snap-subdir-test 2024-05-27T00:17:13.987 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/30 file after the snapshot 2024-05-27T00:17:13.988 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/31 file after the snapshot 2024-05-27T00:17:13.988 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/32 file after the snapshot 2024-05-27T00:17:13.991 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/33 file after the snapshot 2024-05-27T00:17:13.995 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/34 file after the snapshot 2024-05-27T00:17:13.998 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/35 file after the snapshot 2024-05-27T00:17:13.999 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/36 file after the snapshot 2024-05-27T00:17:14.000 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/37 file after the snapshot 2024-05-27T00:17:14.001 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/38 file after the snapshot 2024-05-27T00:17:14.002 INFO:tasks.workunit.client.0.smithi098.stdout: create 177/39 file after the snapshot 2024-05-27T00:17:14.003 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/.snap/snap-subdir-test 2024-05-27T00:17:14.024 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/30 file after the snapshot 2024-05-27T00:17:14.029 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/31 file after the snapshot 2024-05-27T00:17:14.034 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/32 file after the snapshot 2024-05-27T00:17:14.034 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/33 file after the snapshot 2024-05-27T00:17:14.035 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/34 file after the snapshot 2024-05-27T00:17:14.036 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/35 file after the snapshot 2024-05-27T00:17:14.037 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/36 file after the snapshot 2024-05-27T00:17:14.038 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/37 file after the snapshot 2024-05-27T00:17:14.039 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/38 file after the snapshot 2024-05-27T00:17:14.040 INFO:tasks.workunit.client.0.smithi098.stdout: create 178/39 file after the snapshot 2024-05-27T00:17:14.040 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/.snap/snap-subdir-test 2024-05-27T00:17:14.061 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/30 file after the snapshot 2024-05-27T00:17:14.065 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/31 file after the snapshot 2024-05-27T00:17:14.071 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/32 file after the snapshot 2024-05-27T00:17:14.071 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/33 file after the snapshot 2024-05-27T00:17:14.072 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/34 file after the snapshot 2024-05-27T00:17:14.073 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/35 file after the snapshot 2024-05-27T00:17:14.074 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/36 file after the snapshot 2024-05-27T00:17:14.075 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/37 file after the snapshot 2024-05-27T00:17:14.076 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/38 file after the snapshot 2024-05-27T00:17:14.076 INFO:tasks.workunit.client.0.smithi098.stdout: create 179/39 file after the snapshot 2024-05-27T00:17:14.077 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/.snap/snap-subdir-test 2024-05-27T00:17:14.098 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/30 file after the snapshot 2024-05-27T00:17:14.101 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/31 file after the snapshot 2024-05-27T00:17:14.106 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/32 file after the snapshot 2024-05-27T00:17:14.109 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/33 file after the snapshot 2024-05-27T00:17:14.110 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/34 file after the snapshot 2024-05-27T00:17:14.111 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/35 file after the snapshot 2024-05-27T00:17:14.111 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/36 file after the snapshot 2024-05-27T00:17:14.112 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/37 file after the snapshot 2024-05-27T00:17:14.113 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/38 file after the snapshot 2024-05-27T00:17:14.114 INFO:tasks.workunit.client.0.smithi098.stdout: create 180/39 file after the snapshot 2024-05-27T00:17:14.115 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/.snap/snap-subdir-test 2024-05-27T00:17:14.138 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/30 file after the snapshot 2024-05-27T00:17:14.142 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/31 file after the snapshot 2024-05-27T00:17:14.147 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/32 file after the snapshot 2024-05-27T00:17:14.148 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/33 file after the snapshot 2024-05-27T00:17:14.149 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/34 file after the snapshot 2024-05-27T00:17:14.150 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/35 file after the snapshot 2024-05-27T00:17:14.151 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/36 file after the snapshot 2024-05-27T00:17:14.152 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/37 file after the snapshot 2024-05-27T00:17:14.153 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/38 file after the snapshot 2024-05-27T00:17:14.153 INFO:tasks.workunit.client.0.smithi098.stdout: create 181/39 file after the snapshot 2024-05-27T00:17:14.154 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/.snap/snap-subdir-test 2024-05-27T00:17:14.175 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/30 file after the snapshot 2024-05-27T00:17:14.177 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/31 file after the snapshot 2024-05-27T00:17:14.180 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/32 file after the snapshot 2024-05-27T00:17:14.185 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/33 file after the snapshot 2024-05-27T00:17:14.186 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/34 file after the snapshot 2024-05-27T00:17:14.187 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/35 file after the snapshot 2024-05-27T00:17:14.188 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/36 file after the snapshot 2024-05-27T00:17:14.189 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/37 file after the snapshot 2024-05-27T00:17:14.189 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/38 file after the snapshot 2024-05-27T00:17:14.190 INFO:tasks.workunit.client.0.smithi098.stdout: create 182/39 file after the snapshot 2024-05-27T00:17:14.191 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/.snap/snap-subdir-test 2024-05-27T00:17:14.215 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/30 file after the snapshot 2024-05-27T00:17:14.225 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/31 file after the snapshot 2024-05-27T00:17:14.232 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/32 file after the snapshot 2024-05-27T00:17:14.233 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/33 file after the snapshot 2024-05-27T00:17:14.234 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/34 file after the snapshot 2024-05-27T00:17:14.236 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/35 file after the snapshot 2024-05-27T00:17:14.237 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/36 file after the snapshot 2024-05-27T00:17:14.238 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/37 file after the snapshot 2024-05-27T00:17:14.240 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/38 file after the snapshot 2024-05-27T00:17:14.241 INFO:tasks.workunit.client.0.smithi098.stdout: create 183/39 file after the snapshot 2024-05-27T00:17:14.242 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/.snap/snap-subdir-test 2024-05-27T00:17:14.265 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/30 file after the snapshot 2024-05-27T00:17:14.269 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/31 file after the snapshot 2024-05-27T00:17:14.275 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/32 file after the snapshot 2024-05-27T00:17:14.276 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/33 file after the snapshot 2024-05-27T00:17:14.277 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/34 file after the snapshot 2024-05-27T00:17:14.278 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/35 file after the snapshot 2024-05-27T00:17:14.279 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/36 file after the snapshot 2024-05-27T00:17:14.279 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/37 file after the snapshot 2024-05-27T00:17:14.280 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/38 file after the snapshot 2024-05-27T00:17:14.281 INFO:tasks.workunit.client.0.smithi098.stdout: create 184/39 file after the snapshot 2024-05-27T00:17:14.282 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/.snap/snap-subdir-test 2024-05-27T00:17:14.304 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/30 file after the snapshot 2024-05-27T00:17:14.307 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/31 file after the snapshot 2024-05-27T00:17:14.314 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/32 file after the snapshot 2024-05-27T00:17:14.315 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/33 file after the snapshot 2024-05-27T00:17:14.316 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/34 file after the snapshot 2024-05-27T00:17:14.317 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/35 file after the snapshot 2024-05-27T00:17:14.317 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/36 file after the snapshot 2024-05-27T00:17:14.318 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/37 file after the snapshot 2024-05-27T00:17:14.319 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/38 file after the snapshot 2024-05-27T00:17:14.320 INFO:tasks.workunit.client.0.smithi098.stdout: create 185/39 file after the snapshot 2024-05-27T00:17:14.321 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/.snap/snap-subdir-test 2024-05-27T00:17:14.346 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/30 file after the snapshot 2024-05-27T00:17:14.349 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/31 file after the snapshot 2024-05-27T00:17:14.358 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/32 file after the snapshot 2024-05-27T00:17:14.359 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/33 file after the snapshot 2024-05-27T00:17:14.360 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/34 file after the snapshot 2024-05-27T00:17:14.361 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/35 file after the snapshot 2024-05-27T00:17:14.362 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/36 file after the snapshot 2024-05-27T00:17:14.363 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/37 file after the snapshot 2024-05-27T00:17:14.364 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/38 file after the snapshot 2024-05-27T00:17:14.364 INFO:tasks.workunit.client.0.smithi098.stdout: create 186/39 file after the snapshot 2024-05-27T00:17:14.365 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/.snap/snap-subdir-test 2024-05-27T00:17:14.396 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/30 file after the snapshot 2024-05-27T00:17:14.398 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/31 file after the snapshot 2024-05-27T00:17:14.412 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/32 file after the snapshot 2024-05-27T00:17:14.413 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/33 file after the snapshot 2024-05-27T00:17:14.414 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/34 file after the snapshot 2024-05-27T00:17:14.415 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/35 file after the snapshot 2024-05-27T00:17:14.416 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/36 file after the snapshot 2024-05-27T00:17:14.416 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/37 file after the snapshot 2024-05-27T00:17:14.417 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/38 file after the snapshot 2024-05-27T00:17:14.418 INFO:tasks.workunit.client.0.smithi098.stdout: create 187/39 file after the snapshot 2024-05-27T00:17:14.419 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/.snap/snap-subdir-test 2024-05-27T00:17:14.444 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/30 file after the snapshot 2024-05-27T00:17:14.444 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/31 file after the snapshot 2024-05-27T00:17:14.447 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/32 file after the snapshot 2024-05-27T00:17:14.453 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/33 file after the snapshot 2024-05-27T00:17:14.457 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/34 file after the snapshot 2024-05-27T00:17:14.458 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/35 file after the snapshot 2024-05-27T00:17:14.459 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/36 file after the snapshot 2024-05-27T00:17:14.459 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/37 file after the snapshot 2024-05-27T00:17:14.460 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/38 file after the snapshot 2024-05-27T00:17:14.461 INFO:tasks.workunit.client.0.smithi098.stdout: create 188/39 file after the snapshot 2024-05-27T00:17:14.462 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/.snap/snap-subdir-test 2024-05-27T00:17:14.489 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/30 file after the snapshot 2024-05-27T00:17:14.490 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/31 file after the snapshot 2024-05-27T00:17:14.494 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/32 file after the snapshot 2024-05-27T00:17:14.497 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/33 file after the snapshot 2024-05-27T00:17:14.501 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/34 file after the snapshot 2024-05-27T00:17:14.503 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/35 file after the snapshot 2024-05-27T00:17:14.504 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/36 file after the snapshot 2024-05-27T00:17:14.505 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/37 file after the snapshot 2024-05-27T00:17:14.507 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/38 file after the snapshot 2024-05-27T00:17:14.508 INFO:tasks.workunit.client.0.smithi098.stdout: create 189/39 file after the snapshot 2024-05-27T00:17:14.509 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/.snap/snap-subdir-test 2024-05-27T00:17:14.534 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/30 file after the snapshot 2024-05-27T00:17:14.534 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/31 file after the snapshot 2024-05-27T00:17:14.535 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/32 file after the snapshot 2024-05-27T00:17:14.537 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/33 file after the snapshot 2024-05-27T00:17:14.543 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/34 file after the snapshot 2024-05-27T00:17:14.547 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/35 file after the snapshot 2024-05-27T00:17:14.549 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/36 file after the snapshot 2024-05-27T00:17:14.549 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/37 file after the snapshot 2024-05-27T00:17:14.550 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/38 file after the snapshot 2024-05-27T00:17:14.550 INFO:tasks.workunit.client.0.smithi098.stdout: create 190/39 file after the snapshot 2024-05-27T00:17:14.551 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/.snap/snap-subdir-test 2024-05-27T00:17:14.575 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/30 file after the snapshot 2024-05-27T00:17:14.578 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/31 file after the snapshot 2024-05-27T00:17:14.584 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/32 file after the snapshot 2024-05-27T00:17:14.585 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/33 file after the snapshot 2024-05-27T00:17:14.586 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/34 file after the snapshot 2024-05-27T00:17:14.587 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/35 file after the snapshot 2024-05-27T00:17:14.588 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/36 file after the snapshot 2024-05-27T00:17:14.588 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/37 file after the snapshot 2024-05-27T00:17:14.589 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/38 file after the snapshot 2024-05-27T00:17:14.590 INFO:tasks.workunit.client.0.smithi098.stdout: create 191/39 file after the snapshot 2024-05-27T00:17:14.591 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/.snap/snap-subdir-test 2024-05-27T00:17:14.614 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/30 file after the snapshot 2024-05-27T00:17:14.616 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/31 file after the snapshot 2024-05-27T00:17:14.617 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/32 file after the snapshot 2024-05-27T00:17:14.620 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/33 file after the snapshot 2024-05-27T00:17:14.626 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/34 file after the snapshot 2024-05-27T00:17:14.626 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/35 file after the snapshot 2024-05-27T00:17:14.627 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/36 file after the snapshot 2024-05-27T00:17:14.628 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/37 file after the snapshot 2024-05-27T00:17:14.629 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/38 file after the snapshot 2024-05-27T00:17:14.630 INFO:tasks.workunit.client.0.smithi098.stdout: create 192/39 file after the snapshot 2024-05-27T00:17:14.630 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/.snap/snap-subdir-test 2024-05-27T00:17:14.656 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/30 file after the snapshot 2024-05-27T00:17:14.662 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/31 file after the snapshot 2024-05-27T00:17:14.665 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/32 file after the snapshot 2024-05-27T00:17:14.665 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/33 file after the snapshot 2024-05-27T00:17:14.666 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/34 file after the snapshot 2024-05-27T00:17:14.667 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/35 file after the snapshot 2024-05-27T00:17:14.668 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/36 file after the snapshot 2024-05-27T00:17:14.669 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/37 file after the snapshot 2024-05-27T00:17:14.670 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/38 file after the snapshot 2024-05-27T00:17:14.670 INFO:tasks.workunit.client.0.smithi098.stdout: create 193/39 file after the snapshot 2024-05-27T00:17:14.671 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/.snap/snap-subdir-test 2024-05-27T00:17:14.696 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/30 file after the snapshot 2024-05-27T00:17:14.697 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/31 file after the snapshot 2024-05-27T00:17:14.699 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/32 file after the snapshot 2024-05-27T00:17:14.704 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/33 file after the snapshot 2024-05-27T00:17:14.707 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/34 file after the snapshot 2024-05-27T00:17:14.708 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/35 file after the snapshot 2024-05-27T00:17:14.709 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/36 file after the snapshot 2024-05-27T00:17:14.710 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/37 file after the snapshot 2024-05-27T00:17:14.711 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/38 file after the snapshot 2024-05-27T00:17:14.711 INFO:tasks.workunit.client.0.smithi098.stdout: create 194/39 file after the snapshot 2024-05-27T00:17:14.712 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/.snap/snap-subdir-test 2024-05-27T00:17:14.741 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/30 file after the snapshot 2024-05-27T00:17:14.744 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/31 file after the snapshot 2024-05-27T00:17:14.752 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/32 file after the snapshot 2024-05-27T00:17:14.753 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/33 file after the snapshot 2024-05-27T00:17:14.754 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/34 file after the snapshot 2024-05-27T00:17:14.754 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/35 file after the snapshot 2024-05-27T00:17:14.758 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/36 file after the snapshot 2024-05-27T00:17:14.758 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/37 file after the snapshot 2024-05-27T00:17:14.758 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/38 file after the snapshot 2024-05-27T00:17:14.758 INFO:tasks.workunit.client.0.smithi098.stdout: create 195/39 file after the snapshot 2024-05-27T00:17:14.759 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/.snap/snap-subdir-test 2024-05-27T00:17:14.782 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/30 file after the snapshot 2024-05-27T00:17:14.784 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/31 file after the snapshot 2024-05-27T00:17:14.790 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/32 file after the snapshot 2024-05-27T00:17:14.793 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/33 file after the snapshot 2024-05-27T00:17:14.794 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/34 file after the snapshot 2024-05-27T00:17:14.795 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/35 file after the snapshot 2024-05-27T00:17:14.796 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/36 file after the snapshot 2024-05-27T00:17:14.797 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/37 file after the snapshot 2024-05-27T00:17:14.798 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/38 file after the snapshot 2024-05-27T00:17:14.798 INFO:tasks.workunit.client.0.smithi098.stdout: create 196/39 file after the snapshot 2024-05-27T00:17:14.799 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/.snap/snap-subdir-test 2024-05-27T00:17:14.823 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/30 file after the snapshot 2024-05-27T00:17:14.824 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/31 file after the snapshot 2024-05-27T00:17:14.825 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/32 file after the snapshot 2024-05-27T00:17:14.827 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/33 file after the snapshot 2024-05-27T00:17:14.829 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/34 file after the snapshot 2024-05-27T00:17:14.832 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/35 file after the snapshot 2024-05-27T00:17:14.837 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/36 file after the snapshot 2024-05-27T00:17:14.837 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/37 file after the snapshot 2024-05-27T00:17:14.838 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/38 file after the snapshot 2024-05-27T00:17:14.839 INFO:tasks.workunit.client.0.smithi098.stdout: create 197/39 file after the snapshot 2024-05-27T00:17:14.840 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/.snap/snap-subdir-test 2024-05-27T00:17:14.863 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/30 file after the snapshot 2024-05-27T00:17:14.865 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/31 file after the snapshot 2024-05-27T00:17:14.870 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/32 file after the snapshot 2024-05-27T00:17:14.874 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/33 file after the snapshot 2024-05-27T00:17:14.875 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/34 file after the snapshot 2024-05-27T00:17:14.875 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/35 file after the snapshot 2024-05-27T00:17:14.876 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/36 file after the snapshot 2024-05-27T00:17:14.877 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/37 file after the snapshot 2024-05-27T00:17:14.878 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/38 file after the snapshot 2024-05-27T00:17:14.879 INFO:tasks.workunit.client.0.smithi098.stdout: create 198/39 file after the snapshot 2024-05-27T00:17:14.880 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/.snap/snap-subdir-test 2024-05-27T00:17:14.904 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/30 file after the snapshot 2024-05-27T00:17:14.906 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/31 file after the snapshot 2024-05-27T00:17:14.911 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/32 file after the snapshot 2024-05-27T00:17:14.915 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/33 file after the snapshot 2024-05-27T00:17:14.916 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/34 file after the snapshot 2024-05-27T00:17:14.916 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/35 file after the snapshot 2024-05-27T00:17:14.917 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/36 file after the snapshot 2024-05-27T00:17:14.918 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/37 file after the snapshot 2024-05-27T00:17:14.919 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/38 file after the snapshot 2024-05-27T00:17:14.920 INFO:tasks.workunit.client.0.smithi098.stdout: create 199/39 file after the snapshot 2024-05-27T00:17:14.920 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/.snap/snap-subdir-test 2024-05-27T00:17:14.943 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/30 file after the snapshot 2024-05-27T00:17:14.944 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/31 file after the snapshot 2024-05-27T00:17:14.944 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/32 file after the snapshot 2024-05-27T00:17:14.945 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/33 file after the snapshot 2024-05-27T00:17:14.946 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/34 file after the snapshot 2024-05-27T00:17:14.947 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/35 file after the snapshot 2024-05-27T00:17:14.948 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/36 file after the snapshot 2024-05-27T00:17:14.949 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/37 file after the snapshot 2024-05-27T00:17:14.949 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/38 file after the snapshot 2024-05-27T00:17:14.950 INFO:tasks.workunit.client.0.smithi098.stdout: create 200/39 file after the snapshot 2024-05-27T00:17:14.951 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/.snap/snap-subdir-test 2024-05-27T00:17:14.967 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/30 file after the snapshot 2024-05-27T00:17:14.968 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/31 file after the snapshot 2024-05-27T00:17:14.969 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/32 file after the snapshot 2024-05-27T00:17:14.970 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/33 file after the snapshot 2024-05-27T00:17:14.970 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/34 file after the snapshot 2024-05-27T00:17:14.971 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/35 file after the snapshot 2024-05-27T00:17:14.972 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/36 file after the snapshot 2024-05-27T00:17:14.973 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/37 file after the snapshot 2024-05-27T00:17:14.974 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/38 file after the snapshot 2024-05-27T00:17:14.974 INFO:tasks.workunit.client.0.smithi098.stdout: create 201/39 file after the snapshot 2024-05-27T00:17:14.975 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/.snap/snap-subdir-test 2024-05-27T00:17:14.996 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/30 file after the snapshot 2024-05-27T00:17:14.996 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/31 file after the snapshot 2024-05-27T00:17:14.997 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/32 file after the snapshot 2024-05-27T00:17:14.998 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/33 file after the snapshot 2024-05-27T00:17:14.999 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/34 file after the snapshot 2024-05-27T00:17:15.000 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/35 file after the snapshot 2024-05-27T00:17:15.001 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/36 file after the snapshot 2024-05-27T00:17:15.001 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/37 file after the snapshot 2024-05-27T00:17:15.002 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/38 file after the snapshot 2024-05-27T00:17:15.003 INFO:tasks.workunit.client.0.smithi098.stdout: create 202/39 file after the snapshot 2024-05-27T00:17:15.004 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/.snap/snap-subdir-test 2024-05-27T00:17:15.024 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/30 file after the snapshot 2024-05-27T00:17:15.024 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/31 file after the snapshot 2024-05-27T00:17:15.025 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/32 file after the snapshot 2024-05-27T00:17:15.026 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/33 file after the snapshot 2024-05-27T00:17:15.027 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/34 file after the snapshot 2024-05-27T00:17:15.028 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/35 file after the snapshot 2024-05-27T00:17:15.028 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/36 file after the snapshot 2024-05-27T00:17:15.029 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/37 file after the snapshot 2024-05-27T00:17:15.030 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/38 file after the snapshot 2024-05-27T00:17:15.031 INFO:tasks.workunit.client.0.smithi098.stdout: create 203/39 file after the snapshot 2024-05-27T00:17:15.032 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/.snap/snap-subdir-test 2024-05-27T00:17:15.052 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/30 file after the snapshot 2024-05-27T00:17:15.052 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/31 file after the snapshot 2024-05-27T00:17:15.053 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/32 file after the snapshot 2024-05-27T00:17:15.054 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/33 file after the snapshot 2024-05-27T00:17:15.055 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/34 file after the snapshot 2024-05-27T00:17:15.056 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/35 file after the snapshot 2024-05-27T00:17:15.057 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/36 file after the snapshot 2024-05-27T00:17:15.058 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/37 file after the snapshot 2024-05-27T00:17:15.058 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/38 file after the snapshot 2024-05-27T00:17:15.060 INFO:tasks.workunit.client.0.smithi098.stdout: create 204/39 file after the snapshot 2024-05-27T00:17:15.060 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/.snap/snap-subdir-test 2024-05-27T00:17:15.081 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/30 file after the snapshot 2024-05-27T00:17:15.082 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/31 file after the snapshot 2024-05-27T00:17:15.083 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/32 file after the snapshot 2024-05-27T00:17:15.083 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/33 file after the snapshot 2024-05-27T00:17:15.084 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/34 file after the snapshot 2024-05-27T00:17:15.085 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/35 file after the snapshot 2024-05-27T00:17:15.086 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/36 file after the snapshot 2024-05-27T00:17:15.087 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/37 file after the snapshot 2024-05-27T00:17:15.088 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/38 file after the snapshot 2024-05-27T00:17:15.089 INFO:tasks.workunit.client.0.smithi098.stdout: create 205/39 file after the snapshot 2024-05-27T00:17:15.090 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/.snap/snap-subdir-test 2024-05-27T00:17:15.111 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/30 file after the snapshot 2024-05-27T00:17:15.111 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/31 file after the snapshot 2024-05-27T00:17:15.112 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/32 file after the snapshot 2024-05-27T00:17:15.113 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/33 file after the snapshot 2024-05-27T00:17:15.114 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/34 file after the snapshot 2024-05-27T00:17:15.116 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/35 file after the snapshot 2024-05-27T00:17:15.116 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/36 file after the snapshot 2024-05-27T00:17:15.116 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/37 file after the snapshot 2024-05-27T00:17:15.117 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/38 file after the snapshot 2024-05-27T00:17:15.118 INFO:tasks.workunit.client.0.smithi098.stdout: create 206/39 file after the snapshot 2024-05-27T00:17:15.119 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/.snap/snap-subdir-test 2024-05-27T00:17:15.138 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/30 file after the snapshot 2024-05-27T00:17:15.138 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/31 file after the snapshot 2024-05-27T00:17:15.140 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/32 file after the snapshot 2024-05-27T00:17:15.141 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/33 file after the snapshot 2024-05-27T00:17:15.142 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/34 file after the snapshot 2024-05-27T00:17:15.144 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/35 file after the snapshot 2024-05-27T00:17:15.145 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/36 file after the snapshot 2024-05-27T00:17:15.147 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/37 file after the snapshot 2024-05-27T00:17:15.148 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/38 file after the snapshot 2024-05-27T00:17:15.149 INFO:tasks.workunit.client.0.smithi098.stdout: create 207/39 file after the snapshot 2024-05-27T00:17:15.150 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/.snap/snap-subdir-test 2024-05-27T00:17:15.168 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/30 file after the snapshot 2024-05-27T00:17:15.169 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/31 file after the snapshot 2024-05-27T00:17:15.171 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/32 file after the snapshot 2024-05-27T00:17:15.172 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/33 file after the snapshot 2024-05-27T00:17:15.173 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/34 file after the snapshot 2024-05-27T00:17:15.174 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/35 file after the snapshot 2024-05-27T00:17:15.175 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/36 file after the snapshot 2024-05-27T00:17:15.175 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/37 file after the snapshot 2024-05-27T00:17:15.176 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/38 file after the snapshot 2024-05-27T00:17:15.177 INFO:tasks.workunit.client.0.smithi098.stdout: create 208/39 file after the snapshot 2024-05-27T00:17:15.178 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/.snap/snap-subdir-test 2024-05-27T00:17:15.198 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/30 file after the snapshot 2024-05-27T00:17:15.199 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/31 file after the snapshot 2024-05-27T00:17:15.200 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/32 file after the snapshot 2024-05-27T00:17:15.200 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/33 file after the snapshot 2024-05-27T00:17:15.201 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/34 file after the snapshot 2024-05-27T00:17:15.202 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/35 file after the snapshot 2024-05-27T00:17:15.203 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/36 file after the snapshot 2024-05-27T00:17:15.204 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/37 file after the snapshot 2024-05-27T00:17:15.205 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/38 file after the snapshot 2024-05-27T00:17:15.205 INFO:tasks.workunit.client.0.smithi098.stdout: create 209/39 file after the snapshot 2024-05-27T00:17:15.206 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/.snap/snap-subdir-test 2024-05-27T00:17:15.224 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/30 file after the snapshot 2024-05-27T00:17:15.224 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/31 file after the snapshot 2024-05-27T00:17:15.225 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/32 file after the snapshot 2024-05-27T00:17:15.226 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/33 file after the snapshot 2024-05-27T00:17:15.227 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/34 file after the snapshot 2024-05-27T00:17:15.227 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/35 file after the snapshot 2024-05-27T00:17:15.228 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/36 file after the snapshot 2024-05-27T00:17:15.229 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/37 file after the snapshot 2024-05-27T00:17:15.230 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/38 file after the snapshot 2024-05-27T00:17:15.231 INFO:tasks.workunit.client.0.smithi098.stdout: create 210/39 file after the snapshot 2024-05-27T00:17:15.232 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/.snap/snap-subdir-test 2024-05-27T00:17:15.248 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/30 file after the snapshot 2024-05-27T00:17:15.249 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/31 file after the snapshot 2024-05-27T00:17:15.250 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/32 file after the snapshot 2024-05-27T00:17:15.251 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/33 file after the snapshot 2024-05-27T00:17:15.252 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/34 file after the snapshot 2024-05-27T00:17:15.252 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/35 file after the snapshot 2024-05-27T00:17:15.253 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/36 file after the snapshot 2024-05-27T00:17:15.254 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/37 file after the snapshot 2024-05-27T00:17:15.255 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/38 file after the snapshot 2024-05-27T00:17:15.256 INFO:tasks.workunit.client.0.smithi098.stdout: create 211/39 file after the snapshot 2024-05-27T00:17:15.257 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/.snap/snap-subdir-test 2024-05-27T00:17:15.274 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/30 file after the snapshot 2024-05-27T00:17:15.275 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/31 file after the snapshot 2024-05-27T00:17:15.276 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/32 file after the snapshot 2024-05-27T00:17:15.276 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/33 file after the snapshot 2024-05-27T00:17:15.277 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/34 file after the snapshot 2024-05-27T00:17:15.278 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/35 file after the snapshot 2024-05-27T00:17:15.279 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/36 file after the snapshot 2024-05-27T00:17:15.280 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/37 file after the snapshot 2024-05-27T00:17:15.280 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/38 file after the snapshot 2024-05-27T00:17:15.281 INFO:tasks.workunit.client.0.smithi098.stdout: create 212/39 file after the snapshot 2024-05-27T00:17:15.282 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/.snap/snap-subdir-test 2024-05-27T00:17:15.299 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/30 file after the snapshot 2024-05-27T00:17:15.300 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/31 file after the snapshot 2024-05-27T00:17:15.301 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/32 file after the snapshot 2024-05-27T00:17:15.302 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/33 file after the snapshot 2024-05-27T00:17:15.302 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/34 file after the snapshot 2024-05-27T00:17:15.303 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/35 file after the snapshot 2024-05-27T00:17:15.304 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/36 file after the snapshot 2024-05-27T00:17:15.305 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/37 file after the snapshot 2024-05-27T00:17:15.306 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/38 file after the snapshot 2024-05-27T00:17:15.306 INFO:tasks.workunit.client.0.smithi098.stdout: create 213/39 file after the snapshot 2024-05-27T00:17:15.307 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/.snap/snap-subdir-test 2024-05-27T00:17:15.324 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/30 file after the snapshot 2024-05-27T00:17:15.325 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/31 file after the snapshot 2024-05-27T00:17:15.326 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/32 file after the snapshot 2024-05-27T00:17:15.327 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/33 file after the snapshot 2024-05-27T00:17:15.328 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/34 file after the snapshot 2024-05-27T00:17:15.329 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/35 file after the snapshot 2024-05-27T00:17:15.329 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/36 file after the snapshot 2024-05-27T00:17:15.330 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/37 file after the snapshot 2024-05-27T00:17:15.331 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/38 file after the snapshot 2024-05-27T00:17:15.332 INFO:tasks.workunit.client.0.smithi098.stdout: create 214/39 file after the snapshot 2024-05-27T00:17:15.333 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/.snap/snap-subdir-test 2024-05-27T00:17:15.351 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/30 file after the snapshot 2024-05-27T00:17:15.352 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/31 file after the snapshot 2024-05-27T00:17:15.353 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/32 file after the snapshot 2024-05-27T00:17:15.354 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/33 file after the snapshot 2024-05-27T00:17:15.354 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/34 file after the snapshot 2024-05-27T00:17:15.355 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/35 file after the snapshot 2024-05-27T00:17:15.356 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/36 file after the snapshot 2024-05-27T00:17:15.357 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/37 file after the snapshot 2024-05-27T00:17:15.358 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/38 file after the snapshot 2024-05-27T00:17:15.358 INFO:tasks.workunit.client.0.smithi098.stdout: create 215/39 file after the snapshot 2024-05-27T00:17:15.359 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/.snap/snap-subdir-test 2024-05-27T00:17:15.378 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/30 file after the snapshot 2024-05-27T00:17:15.379 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/31 file after the snapshot 2024-05-27T00:17:15.380 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/32 file after the snapshot 2024-05-27T00:17:15.381 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/33 file after the snapshot 2024-05-27T00:17:15.381 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/34 file after the snapshot 2024-05-27T00:17:15.382 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/35 file after the snapshot 2024-05-27T00:17:15.383 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/36 file after the snapshot 2024-05-27T00:17:15.384 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/37 file after the snapshot 2024-05-27T00:17:15.385 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/38 file after the snapshot 2024-05-27T00:17:15.385 INFO:tasks.workunit.client.0.smithi098.stdout: create 216/39 file after the snapshot 2024-05-27T00:17:15.386 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/.snap/snap-subdir-test 2024-05-27T00:17:15.405 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/30 file after the snapshot 2024-05-27T00:17:15.405 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/31 file after the snapshot 2024-05-27T00:17:15.407 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/32 file after the snapshot 2024-05-27T00:17:15.408 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/33 file after the snapshot 2024-05-27T00:17:15.409 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/34 file after the snapshot 2024-05-27T00:17:15.411 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/35 file after the snapshot 2024-05-27T00:17:15.412 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/36 file after the snapshot 2024-05-27T00:17:15.413 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/37 file after the snapshot 2024-05-27T00:17:15.414 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/38 file after the snapshot 2024-05-27T00:17:15.414 INFO:tasks.workunit.client.0.smithi098.stdout: create 217/39 file after the snapshot 2024-05-27T00:17:15.415 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/.snap/snap-subdir-test 2024-05-27T00:17:15.433 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/30 file after the snapshot 2024-05-27T00:17:15.434 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/31 file after the snapshot 2024-05-27T00:17:15.434 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/32 file after the snapshot 2024-05-27T00:17:15.435 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/33 file after the snapshot 2024-05-27T00:17:15.436 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/34 file after the snapshot 2024-05-27T00:17:15.437 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/35 file after the snapshot 2024-05-27T00:17:15.438 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/36 file after the snapshot 2024-05-27T00:17:15.439 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/37 file after the snapshot 2024-05-27T00:17:15.439 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/38 file after the snapshot 2024-05-27T00:17:15.440 INFO:tasks.workunit.client.0.smithi098.stdout: create 218/39 file after the snapshot 2024-05-27T00:17:15.442 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/.snap/snap-subdir-test 2024-05-27T00:17:15.464 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/30 file after the snapshot 2024-05-27T00:17:15.465 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/31 file after the snapshot 2024-05-27T00:17:15.466 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/32 file after the snapshot 2024-05-27T00:17:15.466 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/33 file after the snapshot 2024-05-27T00:17:15.467 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/34 file after the snapshot 2024-05-27T00:17:15.468 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/35 file after the snapshot 2024-05-27T00:17:15.469 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/36 file after the snapshot 2024-05-27T00:17:15.470 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/37 file after the snapshot 2024-05-27T00:17:15.471 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/38 file after the snapshot 2024-05-27T00:17:15.472 INFO:tasks.workunit.client.0.smithi098.stdout: create 219/39 file after the snapshot 2024-05-27T00:17:15.472 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/.snap/snap-subdir-test 2024-05-27T00:17:15.486 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/30 file after the snapshot 2024-05-27T00:17:15.487 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/31 file after the snapshot 2024-05-27T00:17:15.489 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/32 file after the snapshot 2024-05-27T00:17:15.489 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/33 file after the snapshot 2024-05-27T00:17:15.490 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/34 file after the snapshot 2024-05-27T00:17:15.491 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/35 file after the snapshot 2024-05-27T00:17:15.491 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/36 file after the snapshot 2024-05-27T00:17:15.493 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/37 file after the snapshot 2024-05-27T00:17:15.493 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/38 file after the snapshot 2024-05-27T00:17:15.494 INFO:tasks.workunit.client.0.smithi098.stdout: create 220/39 file after the snapshot 2024-05-27T00:17:15.494 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/.snap/snap-subdir-test 2024-05-27T00:17:15.508 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/30 file after the snapshot 2024-05-27T00:17:15.508 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/31 file after the snapshot 2024-05-27T00:17:15.509 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/32 file after the snapshot 2024-05-27T00:17:15.510 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/33 file after the snapshot 2024-05-27T00:17:15.512 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/34 file after the snapshot 2024-05-27T00:17:15.512 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/35 file after the snapshot 2024-05-27T00:17:15.512 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/36 file after the snapshot 2024-05-27T00:17:15.513 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/37 file after the snapshot 2024-05-27T00:17:15.515 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/38 file after the snapshot 2024-05-27T00:17:15.515 INFO:tasks.workunit.client.0.smithi098.stdout: create 221/39 file after the snapshot 2024-05-27T00:17:15.516 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/.snap/snap-subdir-test 2024-05-27T00:17:15.530 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/30 file after the snapshot 2024-05-27T00:17:15.530 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/31 file after the snapshot 2024-05-27T00:17:15.531 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/32 file after the snapshot 2024-05-27T00:17:15.532 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/33 file after the snapshot 2024-05-27T00:17:15.533 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/34 file after the snapshot 2024-05-27T00:17:15.534 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/35 file after the snapshot 2024-05-27T00:17:15.534 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/36 file after the snapshot 2024-05-27T00:17:15.536 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/37 file after the snapshot 2024-05-27T00:17:15.536 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/38 file after the snapshot 2024-05-27T00:17:15.537 INFO:tasks.workunit.client.0.smithi098.stdout: create 222/39 file after the snapshot 2024-05-27T00:17:15.538 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/.snap/snap-subdir-test 2024-05-27T00:17:15.551 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/30 file after the snapshot 2024-05-27T00:17:15.552 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/31 file after the snapshot 2024-05-27T00:17:15.553 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/32 file after the snapshot 2024-05-27T00:17:15.554 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/33 file after the snapshot 2024-05-27T00:17:15.555 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/34 file after the snapshot 2024-05-27T00:17:15.556 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/35 file after the snapshot 2024-05-27T00:17:15.557 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/36 file after the snapshot 2024-05-27T00:17:15.558 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/37 file after the snapshot 2024-05-27T00:17:15.558 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/38 file after the snapshot 2024-05-27T00:17:15.559 INFO:tasks.workunit.client.0.smithi098.stdout: create 223/39 file after the snapshot 2024-05-27T00:17:15.560 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/.snap/snap-subdir-test 2024-05-27T00:17:15.574 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/30 file after the snapshot 2024-05-27T00:17:15.575 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/31 file after the snapshot 2024-05-27T00:17:15.575 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/32 file after the snapshot 2024-05-27T00:17:15.576 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/33 file after the snapshot 2024-05-27T00:17:15.577 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/34 file after the snapshot 2024-05-27T00:17:15.578 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/35 file after the snapshot 2024-05-27T00:17:15.579 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/36 file after the snapshot 2024-05-27T00:17:15.579 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/37 file after the snapshot 2024-05-27T00:17:15.580 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/38 file after the snapshot 2024-05-27T00:17:15.584 INFO:tasks.workunit.client.0.smithi098.stdout: create 224/39 file after the snapshot 2024-05-27T00:17:15.584 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/.snap/snap-subdir-test 2024-05-27T00:17:15.596 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/30 file after the snapshot 2024-05-27T00:17:15.596 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/31 file after the snapshot 2024-05-27T00:17:15.597 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/32 file after the snapshot 2024-05-27T00:17:15.598 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/33 file after the snapshot 2024-05-27T00:17:15.599 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/34 file after the snapshot 2024-05-27T00:17:15.600 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/35 file after the snapshot 2024-05-27T00:17:15.601 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/36 file after the snapshot 2024-05-27T00:17:15.601 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/37 file after the snapshot 2024-05-27T00:17:15.602 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/38 file after the snapshot 2024-05-27T00:17:15.603 INFO:tasks.workunit.client.0.smithi098.stdout: create 225/39 file after the snapshot 2024-05-27T00:17:15.604 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/.snap/snap-subdir-test 2024-05-27T00:17:15.618 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/30 file after the snapshot 2024-05-27T00:17:15.619 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/31 file after the snapshot 2024-05-27T00:17:15.619 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/32 file after the snapshot 2024-05-27T00:17:15.620 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/33 file after the snapshot 2024-05-27T00:17:15.621 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/34 file after the snapshot 2024-05-27T00:17:15.622 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/35 file after the snapshot 2024-05-27T00:17:15.623 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/36 file after the snapshot 2024-05-27T00:17:15.624 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/37 file after the snapshot 2024-05-27T00:17:15.625 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/38 file after the snapshot 2024-05-27T00:17:15.625 INFO:tasks.workunit.client.0.smithi098.stdout: create 226/39 file after the snapshot 2024-05-27T00:17:15.626 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/.snap/snap-subdir-test 2024-05-27T00:17:15.640 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/30 file after the snapshot 2024-05-27T00:17:15.641 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/31 file after the snapshot 2024-05-27T00:17:15.642 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/32 file after the snapshot 2024-05-27T00:17:15.642 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/33 file after the snapshot 2024-05-27T00:17:15.643 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/34 file after the snapshot 2024-05-27T00:17:15.644 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/35 file after the snapshot 2024-05-27T00:17:15.645 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/36 file after the snapshot 2024-05-27T00:17:15.646 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/37 file after the snapshot 2024-05-27T00:17:15.647 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/38 file after the snapshot 2024-05-27T00:17:15.647 INFO:tasks.workunit.client.0.smithi098.stdout: create 227/39 file after the snapshot 2024-05-27T00:17:15.648 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/.snap/snap-subdir-test 2024-05-27T00:17:15.662 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/30 file after the snapshot 2024-05-27T00:17:15.663 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/31 file after the snapshot 2024-05-27T00:17:15.664 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/32 file after the snapshot 2024-05-27T00:17:15.665 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/33 file after the snapshot 2024-05-27T00:17:15.666 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/34 file after the snapshot 2024-05-27T00:17:15.666 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/35 file after the snapshot 2024-05-27T00:17:15.668 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/36 file after the snapshot 2024-05-27T00:17:15.668 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/37 file after the snapshot 2024-05-27T00:17:15.669 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/38 file after the snapshot 2024-05-27T00:17:15.670 INFO:tasks.workunit.client.0.smithi098.stdout: create 228/39 file after the snapshot 2024-05-27T00:17:15.671 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/.snap/snap-subdir-test 2024-05-27T00:17:15.684 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/30 file after the snapshot 2024-05-27T00:17:15.685 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/31 file after the snapshot 2024-05-27T00:17:15.686 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/32 file after the snapshot 2024-05-27T00:17:15.687 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/33 file after the snapshot 2024-05-27T00:17:15.687 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/34 file after the snapshot 2024-05-27T00:17:15.689 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/35 file after the snapshot 2024-05-27T00:17:15.689 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/36 file after the snapshot 2024-05-27T00:17:15.690 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/37 file after the snapshot 2024-05-27T00:17:15.691 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/38 file after the snapshot 2024-05-27T00:17:15.692 INFO:tasks.workunit.client.0.smithi098.stdout: create 229/39 file after the snapshot 2024-05-27T00:17:15.693 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/.snap/snap-subdir-test 2024-05-27T00:17:15.707 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/30 file after the snapshot 2024-05-27T00:17:15.707 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/31 file after the snapshot 2024-05-27T00:17:15.708 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/32 file after the snapshot 2024-05-27T00:17:15.709 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/33 file after the snapshot 2024-05-27T00:17:15.710 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/34 file after the snapshot 2024-05-27T00:17:15.711 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/35 file after the snapshot 2024-05-27T00:17:15.711 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/36 file after the snapshot 2024-05-27T00:17:15.712 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/37 file after the snapshot 2024-05-27T00:17:15.713 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/38 file after the snapshot 2024-05-27T00:17:15.714 INFO:tasks.workunit.client.0.smithi098.stdout: create 230/39 file after the snapshot 2024-05-27T00:17:15.715 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/.snap/snap-subdir-test 2024-05-27T00:17:15.728 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/30 file after the snapshot 2024-05-27T00:17:15.729 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/31 file after the snapshot 2024-05-27T00:17:15.730 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/32 file after the snapshot 2024-05-27T00:17:15.731 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/33 file after the snapshot 2024-05-27T00:17:15.732 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/34 file after the snapshot 2024-05-27T00:17:15.732 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/35 file after the snapshot 2024-05-27T00:17:15.733 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/36 file after the snapshot 2024-05-27T00:17:15.734 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/37 file after the snapshot 2024-05-27T00:17:15.735 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/38 file after the snapshot 2024-05-27T00:17:15.736 INFO:tasks.workunit.client.0.smithi098.stdout: create 231/39 file after the snapshot 2024-05-27T00:17:15.736 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/.snap/snap-subdir-test 2024-05-27T00:17:15.751 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/30 file after the snapshot 2024-05-27T00:17:15.752 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/31 file after the snapshot 2024-05-27T00:17:15.752 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/32 file after the snapshot 2024-05-27T00:17:15.753 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/33 file after the snapshot 2024-05-27T00:17:15.753 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/34 file after the snapshot 2024-05-27T00:17:15.754 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/35 file after the snapshot 2024-05-27T00:17:15.755 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/36 file after the snapshot 2024-05-27T00:17:15.757 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/37 file after the snapshot 2024-05-27T00:17:15.757 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/38 file after the snapshot 2024-05-27T00:17:15.758 INFO:tasks.workunit.client.0.smithi098.stdout: create 232/39 file after the snapshot 2024-05-27T00:17:15.759 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/.snap/snap-subdir-test 2024-05-27T00:17:15.772 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/30 file after the snapshot 2024-05-27T00:17:15.773 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/31 file after the snapshot 2024-05-27T00:17:15.774 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/32 file after the snapshot 2024-05-27T00:17:15.774 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/33 file after the snapshot 2024-05-27T00:17:15.777 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/34 file after the snapshot 2024-05-27T00:17:15.777 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/35 file after the snapshot 2024-05-27T00:17:15.778 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/36 file after the snapshot 2024-05-27T00:17:15.778 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/37 file after the snapshot 2024-05-27T00:17:15.779 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/38 file after the snapshot 2024-05-27T00:17:15.780 INFO:tasks.workunit.client.0.smithi098.stdout: create 233/39 file after the snapshot 2024-05-27T00:17:15.780 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/.snap/snap-subdir-test 2024-05-27T00:17:15.794 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/30 file after the snapshot 2024-05-27T00:17:15.795 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/31 file after the snapshot 2024-05-27T00:17:15.796 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/32 file after the snapshot 2024-05-27T00:17:15.796 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/33 file after the snapshot 2024-05-27T00:17:15.797 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/34 file after the snapshot 2024-05-27T00:17:15.798 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/35 file after the snapshot 2024-05-27T00:17:15.799 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/36 file after the snapshot 2024-05-27T00:17:15.800 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/37 file after the snapshot 2024-05-27T00:17:15.800 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/38 file after the snapshot 2024-05-27T00:17:15.801 INFO:tasks.workunit.client.0.smithi098.stdout: create 234/39 file after the snapshot 2024-05-27T00:17:15.802 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/.snap/snap-subdir-test 2024-05-27T00:17:15.815 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/30 file after the snapshot 2024-05-27T00:17:15.816 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/31 file after the snapshot 2024-05-27T00:17:15.817 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/32 file after the snapshot 2024-05-27T00:17:15.818 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/33 file after the snapshot 2024-05-27T00:17:15.819 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/34 file after the snapshot 2024-05-27T00:17:15.819 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/35 file after the snapshot 2024-05-27T00:17:15.820 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/36 file after the snapshot 2024-05-27T00:17:15.821 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/37 file after the snapshot 2024-05-27T00:17:15.822 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/38 file after the snapshot 2024-05-27T00:17:15.823 INFO:tasks.workunit.client.0.smithi098.stdout: create 235/39 file after the snapshot 2024-05-27T00:17:15.823 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/.snap/snap-subdir-test 2024-05-27T00:17:15.837 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/30 file after the snapshot 2024-05-27T00:17:15.837 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/31 file after the snapshot 2024-05-27T00:17:15.838 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/32 file after the snapshot 2024-05-27T00:17:15.840 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/33 file after the snapshot 2024-05-27T00:17:15.840 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/34 file after the snapshot 2024-05-27T00:17:15.841 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/35 file after the snapshot 2024-05-27T00:17:15.841 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/36 file after the snapshot 2024-05-27T00:17:15.842 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/37 file after the snapshot 2024-05-27T00:17:15.843 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/38 file after the snapshot 2024-05-27T00:17:15.844 INFO:tasks.workunit.client.0.smithi098.stdout: create 236/39 file after the snapshot 2024-05-27T00:17:15.844 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/.snap/snap-subdir-test 2024-05-27T00:17:15.858 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/30 file after the snapshot 2024-05-27T00:17:15.859 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/31 file after the snapshot 2024-05-27T00:17:15.859 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/32 file after the snapshot 2024-05-27T00:17:15.860 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/33 file after the snapshot 2024-05-27T00:17:15.861 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/34 file after the snapshot 2024-05-27T00:17:15.862 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/35 file after the snapshot 2024-05-27T00:17:15.863 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/36 file after the snapshot 2024-05-27T00:17:15.864 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/37 file after the snapshot 2024-05-27T00:17:15.864 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/38 file after the snapshot 2024-05-27T00:17:15.865 INFO:tasks.workunit.client.0.smithi098.stdout: create 237/39 file after the snapshot 2024-05-27T00:17:15.866 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/.snap/snap-subdir-test 2024-05-27T00:17:15.880 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/30 file after the snapshot 2024-05-27T00:17:15.880 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/31 file after the snapshot 2024-05-27T00:17:15.881 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/32 file after the snapshot 2024-05-27T00:17:15.882 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/33 file after the snapshot 2024-05-27T00:17:15.883 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/34 file after the snapshot 2024-05-27T00:17:15.883 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/35 file after the snapshot 2024-05-27T00:17:15.884 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/36 file after the snapshot 2024-05-27T00:17:15.885 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/37 file after the snapshot 2024-05-27T00:17:15.886 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/38 file after the snapshot 2024-05-27T00:17:15.886 INFO:tasks.workunit.client.0.smithi098.stdout: create 238/39 file after the snapshot 2024-05-27T00:17:15.887 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/.snap/snap-subdir-test 2024-05-27T00:17:15.901 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/30 file after the snapshot 2024-05-27T00:17:15.901 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/31 file after the snapshot 2024-05-27T00:17:15.903 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/32 file after the snapshot 2024-05-27T00:17:15.903 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/33 file after the snapshot 2024-05-27T00:17:15.904 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/34 file after the snapshot 2024-05-27T00:17:15.904 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/35 file after the snapshot 2024-05-27T00:17:15.905 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/36 file after the snapshot 2024-05-27T00:17:15.906 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/37 file after the snapshot 2024-05-27T00:17:15.907 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/38 file after the snapshot 2024-05-27T00:17:15.907 INFO:tasks.workunit.client.0.smithi098.stdout: create 239/39 file after the snapshot 2024-05-27T00:17:15.908 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/.snap/snap-subdir-test 2024-05-27T00:17:15.922 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/30 file after the snapshot 2024-05-27T00:17:15.923 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/31 file after the snapshot 2024-05-27T00:17:15.924 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/32 file after the snapshot 2024-05-27T00:17:15.924 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/33 file after the snapshot 2024-05-27T00:17:15.925 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/34 file after the snapshot 2024-05-27T00:17:15.926 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/35 file after the snapshot 2024-05-27T00:17:15.927 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/36 file after the snapshot 2024-05-27T00:17:15.928 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/37 file after the snapshot 2024-05-27T00:17:15.928 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/38 file after the snapshot 2024-05-27T00:17:15.929 INFO:tasks.workunit.client.0.smithi098.stdout: create 240/39 file after the snapshot 2024-05-27T00:17:15.930 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/.snap/snap-subdir-test 2024-05-27T00:17:15.943 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/30 file after the snapshot 2024-05-27T00:17:15.945 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/31 file after the snapshot 2024-05-27T00:17:15.945 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/32 file after the snapshot 2024-05-27T00:17:15.946 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/33 file after the snapshot 2024-05-27T00:17:15.946 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/34 file after the snapshot 2024-05-27T00:17:15.947 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/35 file after the snapshot 2024-05-27T00:17:15.948 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/36 file after the snapshot 2024-05-27T00:17:15.949 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/37 file after the snapshot 2024-05-27T00:17:15.949 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/38 file after the snapshot 2024-05-27T00:17:15.950 INFO:tasks.workunit.client.0.smithi098.stdout: create 241/39 file after the snapshot 2024-05-27T00:17:15.951 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/.snap/snap-subdir-test 2024-05-27T00:17:15.965 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/30 file after the snapshot 2024-05-27T00:17:15.965 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/31 file after the snapshot 2024-05-27T00:17:15.966 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/32 file after the snapshot 2024-05-27T00:17:15.967 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/33 file after the snapshot 2024-05-27T00:17:15.968 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/34 file after the snapshot 2024-05-27T00:17:15.969 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/35 file after the snapshot 2024-05-27T00:17:15.969 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/36 file after the snapshot 2024-05-27T00:17:15.971 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/37 file after the snapshot 2024-05-27T00:17:15.971 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/38 file after the snapshot 2024-05-27T00:17:15.972 INFO:tasks.workunit.client.0.smithi098.stdout: create 242/39 file after the snapshot 2024-05-27T00:17:15.973 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/.snap/snap-subdir-test 2024-05-27T00:17:15.986 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/30 file after the snapshot 2024-05-27T00:17:15.987 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/31 file after the snapshot 2024-05-27T00:17:15.988 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/32 file after the snapshot 2024-05-27T00:17:15.989 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/33 file after the snapshot 2024-05-27T00:17:15.989 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/34 file after the snapshot 2024-05-27T00:17:15.990 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/35 file after the snapshot 2024-05-27T00:17:15.991 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/36 file after the snapshot 2024-05-27T00:17:15.992 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/37 file after the snapshot 2024-05-27T00:17:15.993 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/38 file after the snapshot 2024-05-27T00:17:15.994 INFO:tasks.workunit.client.0.smithi098.stdout: create 243/39 file after the snapshot 2024-05-27T00:17:15.994 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/.snap/snap-subdir-test 2024-05-27T00:17:16.009 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/30 file after the snapshot 2024-05-27T00:17:16.009 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/31 file after the snapshot 2024-05-27T00:17:16.010 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/32 file after the snapshot 2024-05-27T00:17:16.011 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/33 file after the snapshot 2024-05-27T00:17:16.012 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/34 file after the snapshot 2024-05-27T00:17:16.013 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/35 file after the snapshot 2024-05-27T00:17:16.013 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/36 file after the snapshot 2024-05-27T00:17:16.014 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/37 file after the snapshot 2024-05-27T00:17:16.015 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/38 file after the snapshot 2024-05-27T00:17:16.016 INFO:tasks.workunit.client.0.smithi098.stdout: create 244/39 file after the snapshot 2024-05-27T00:17:16.017 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/.snap/snap-subdir-test 2024-05-27T00:17:16.031 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/30 file after the snapshot 2024-05-27T00:17:16.031 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/31 file after the snapshot 2024-05-27T00:17:16.032 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/32 file after the snapshot 2024-05-27T00:17:16.033 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/33 file after the snapshot 2024-05-27T00:17:16.034 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/34 file after the snapshot 2024-05-27T00:17:16.035 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/35 file after the snapshot 2024-05-27T00:17:16.036 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/36 file after the snapshot 2024-05-27T00:17:16.036 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/37 file after the snapshot 2024-05-27T00:17:16.037 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/38 file after the snapshot 2024-05-27T00:17:16.038 INFO:tasks.workunit.client.0.smithi098.stdout: create 245/39 file after the snapshot 2024-05-27T00:17:16.039 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/.snap/snap-subdir-test 2024-05-27T00:17:16.053 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/30 file after the snapshot 2024-05-27T00:17:16.053 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/31 file after the snapshot 2024-05-27T00:17:16.054 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/32 file after the snapshot 2024-05-27T00:17:16.055 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/33 file after the snapshot 2024-05-27T00:17:16.056 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/34 file after the snapshot 2024-05-27T00:17:16.056 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/35 file after the snapshot 2024-05-27T00:17:16.057 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/36 file after the snapshot 2024-05-27T00:17:16.058 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/37 file after the snapshot 2024-05-27T00:17:16.059 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/38 file after the snapshot 2024-05-27T00:17:16.060 INFO:tasks.workunit.client.0.smithi098.stdout: create 246/39 file after the snapshot 2024-05-27T00:17:16.060 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/.snap/snap-subdir-test 2024-05-27T00:17:16.074 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/30 file after the snapshot 2024-05-27T00:17:16.075 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/31 file after the snapshot 2024-05-27T00:17:16.076 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/32 file after the snapshot 2024-05-27T00:17:16.077 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/33 file after the snapshot 2024-05-27T00:17:16.077 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/34 file after the snapshot 2024-05-27T00:17:16.078 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/35 file after the snapshot 2024-05-27T00:17:16.079 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/36 file after the snapshot 2024-05-27T00:17:16.080 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/37 file after the snapshot 2024-05-27T00:17:16.081 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/38 file after the snapshot 2024-05-27T00:17:16.082 INFO:tasks.workunit.client.0.smithi098.stdout: create 247/39 file after the snapshot 2024-05-27T00:17:16.082 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/.snap/snap-subdir-test 2024-05-27T00:17:16.096 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/30 file after the snapshot 2024-05-27T00:17:16.097 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/31 file after the snapshot 2024-05-27T00:17:16.097 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/32 file after the snapshot 2024-05-27T00:17:16.098 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/33 file after the snapshot 2024-05-27T00:17:16.099 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/34 file after the snapshot 2024-05-27T00:17:16.099 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/35 file after the snapshot 2024-05-27T00:17:16.100 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/36 file after the snapshot 2024-05-27T00:17:16.101 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/37 file after the snapshot 2024-05-27T00:17:16.102 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/38 file after the snapshot 2024-05-27T00:17:16.103 INFO:tasks.workunit.client.0.smithi098.stdout: create 248/39 file after the snapshot 2024-05-27T00:17:16.103 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/.snap/snap-subdir-test 2024-05-27T00:17:16.117 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/30 file after the snapshot 2024-05-27T00:17:16.118 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/31 file after the snapshot 2024-05-27T00:17:16.119 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/32 file after the snapshot 2024-05-27T00:17:16.119 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/33 file after the snapshot 2024-05-27T00:17:16.120 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/34 file after the snapshot 2024-05-27T00:17:16.121 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/35 file after the snapshot 2024-05-27T00:17:16.122 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/36 file after the snapshot 2024-05-27T00:17:16.123 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/37 file after the snapshot 2024-05-27T00:17:16.123 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/38 file after the snapshot 2024-05-27T00:17:16.124 INFO:tasks.workunit.client.0.smithi098.stdout: create 249/39 file after the snapshot 2024-05-27T00:17:16.125 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/.snap/snap-subdir-test 2024-05-27T00:17:16.139 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/30 file after the snapshot 2024-05-27T00:17:16.140 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/31 file after the snapshot 2024-05-27T00:17:16.140 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/32 file after the snapshot 2024-05-27T00:17:16.141 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/33 file after the snapshot 2024-05-27T00:17:16.142 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/34 file after the snapshot 2024-05-27T00:17:16.143 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/35 file after the snapshot 2024-05-27T00:17:16.144 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/36 file after the snapshot 2024-05-27T00:17:16.144 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/37 file after the snapshot 2024-05-27T00:17:16.145 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/38 file after the snapshot 2024-05-27T00:17:16.146 INFO:tasks.workunit.client.0.smithi098.stdout: create 250/39 file after the snapshot 2024-05-27T00:17:16.147 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/.snap/snap-subdir-test 2024-05-27T00:17:16.161 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/30 file after the snapshot 2024-05-27T00:17:16.162 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/31 file after the snapshot 2024-05-27T00:17:16.163 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/32 file after the snapshot 2024-05-27T00:17:16.164 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/33 file after the snapshot 2024-05-27T00:17:16.166 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/34 file after the snapshot 2024-05-27T00:17:16.167 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/35 file after the snapshot 2024-05-27T00:17:16.168 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/36 file after the snapshot 2024-05-27T00:17:16.169 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/37 file after the snapshot 2024-05-27T00:17:16.171 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/38 file after the snapshot 2024-05-27T00:17:16.172 INFO:tasks.workunit.client.0.smithi098.stdout: create 251/39 file after the snapshot 2024-05-27T00:17:16.173 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/.snap/snap-subdir-test 2024-05-27T00:17:16.188 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/30 file after the snapshot 2024-05-27T00:17:16.188 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/31 file after the snapshot 2024-05-27T00:17:16.189 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/32 file after the snapshot 2024-05-27T00:17:16.190 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/33 file after the snapshot 2024-05-27T00:17:16.191 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/34 file after the snapshot 2024-05-27T00:17:16.192 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/35 file after the snapshot 2024-05-27T00:17:16.193 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/36 file after the snapshot 2024-05-27T00:17:16.193 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/37 file after the snapshot 2024-05-27T00:17:16.194 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/38 file after the snapshot 2024-05-27T00:17:16.195 INFO:tasks.workunit.client.0.smithi098.stdout: create 252/39 file after the snapshot 2024-05-27T00:17:16.196 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/.snap/snap-subdir-test 2024-05-27T00:17:16.210 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/30 file after the snapshot 2024-05-27T00:17:16.210 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/31 file after the snapshot 2024-05-27T00:17:16.211 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/32 file after the snapshot 2024-05-27T00:17:16.212 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/33 file after the snapshot 2024-05-27T00:17:16.213 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/34 file after the snapshot 2024-05-27T00:17:16.214 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/35 file after the snapshot 2024-05-27T00:17:16.215 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/36 file after the snapshot 2024-05-27T00:17:16.216 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/37 file after the snapshot 2024-05-27T00:17:16.216 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/38 file after the snapshot 2024-05-27T00:17:16.217 INFO:tasks.workunit.client.0.smithi098.stdout: create 253/39 file after the snapshot 2024-05-27T00:17:16.218 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/.snap/snap-subdir-test 2024-05-27T00:17:16.232 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/30 file after the snapshot 2024-05-27T00:17:16.233 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/31 file after the snapshot 2024-05-27T00:17:16.234 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/32 file after the snapshot 2024-05-27T00:17:16.235 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/33 file after the snapshot 2024-05-27T00:17:16.236 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/34 file after the snapshot 2024-05-27T00:17:16.236 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/35 file after the snapshot 2024-05-27T00:17:16.237 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/36 file after the snapshot 2024-05-27T00:17:16.238 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/37 file after the snapshot 2024-05-27T00:17:16.239 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/38 file after the snapshot 2024-05-27T00:17:16.239 INFO:tasks.workunit.client.0.smithi098.stdout: create 254/39 file after the snapshot 2024-05-27T00:17:16.240 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/.snap/snap-subdir-test 2024-05-27T00:17:16.255 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/30 file after the snapshot 2024-05-27T00:17:16.256 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/31 file after the snapshot 2024-05-27T00:17:16.256 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/32 file after the snapshot 2024-05-27T00:17:16.257 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/33 file after the snapshot 2024-05-27T00:17:16.258 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/34 file after the snapshot 2024-05-27T00:17:16.259 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/35 file after the snapshot 2024-05-27T00:17:16.260 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/36 file after the snapshot 2024-05-27T00:17:16.260 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/37 file after the snapshot 2024-05-27T00:17:16.261 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/38 file after the snapshot 2024-05-27T00:17:16.262 INFO:tasks.workunit.client.0.smithi098.stdout: create 255/39 file after the snapshot 2024-05-27T00:17:16.262 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/.snap/snap-subdir-test 2024-05-27T00:17:16.276 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/30 file after the snapshot 2024-05-27T00:17:16.277 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/31 file after the snapshot 2024-05-27T00:17:16.278 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/32 file after the snapshot 2024-05-27T00:17:16.279 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/33 file after the snapshot 2024-05-27T00:17:16.280 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/34 file after the snapshot 2024-05-27T00:17:16.280 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/35 file after the snapshot 2024-05-27T00:17:16.281 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/36 file after the snapshot 2024-05-27T00:17:16.282 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/37 file after the snapshot 2024-05-27T00:17:16.283 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/38 file after the snapshot 2024-05-27T00:17:16.284 INFO:tasks.workunit.client.0.smithi098.stdout: create 256/39 file after the snapshot 2024-05-27T00:17:16.284 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/.snap/snap-subdir-test 2024-05-27T00:17:16.298 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/30 file after the snapshot 2024-05-27T00:17:16.299 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/31 file after the snapshot 2024-05-27T00:17:16.300 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/32 file after the snapshot 2024-05-27T00:17:16.301 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/33 file after the snapshot 2024-05-27T00:17:16.301 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/34 file after the snapshot 2024-05-27T00:17:16.302 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/35 file after the snapshot 2024-05-27T00:17:16.303 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/36 file after the snapshot 2024-05-27T00:17:16.304 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/37 file after the snapshot 2024-05-27T00:17:16.304 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/38 file after the snapshot 2024-05-27T00:17:16.305 INFO:tasks.workunit.client.0.smithi098.stdout: create 257/39 file after the snapshot 2024-05-27T00:17:16.306 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/.snap/snap-subdir-test 2024-05-27T00:17:16.321 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/30 file after the snapshot 2024-05-27T00:17:16.321 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/31 file after the snapshot 2024-05-27T00:17:16.322 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/32 file after the snapshot 2024-05-27T00:17:16.323 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/33 file after the snapshot 2024-05-27T00:17:16.323 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/34 file after the snapshot 2024-05-27T00:17:16.324 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/35 file after the snapshot 2024-05-27T00:17:16.325 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/36 file after the snapshot 2024-05-27T00:17:16.326 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/37 file after the snapshot 2024-05-27T00:17:16.327 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/38 file after the snapshot 2024-05-27T00:17:16.328 INFO:tasks.workunit.client.0.smithi098.stdout: create 258/39 file after the snapshot 2024-05-27T00:17:16.328 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/.snap/snap-subdir-test 2024-05-27T00:17:16.343 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/30 file after the snapshot 2024-05-27T00:17:16.343 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/31 file after the snapshot 2024-05-27T00:17:16.344 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/32 file after the snapshot 2024-05-27T00:17:16.345 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/33 file after the snapshot 2024-05-27T00:17:16.346 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/34 file after the snapshot 2024-05-27T00:17:16.346 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/35 file after the snapshot 2024-05-27T00:17:16.347 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/36 file after the snapshot 2024-05-27T00:17:16.348 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/37 file after the snapshot 2024-05-27T00:17:16.349 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/38 file after the snapshot 2024-05-27T00:17:16.350 INFO:tasks.workunit.client.0.smithi098.stdout: create 259/39 file after the snapshot 2024-05-27T00:17:16.351 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/.snap/snap-subdir-test 2024-05-27T00:17:16.364 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/30 file after the snapshot 2024-05-27T00:17:16.365 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/31 file after the snapshot 2024-05-27T00:17:16.366 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/32 file after the snapshot 2024-05-27T00:17:16.367 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/33 file after the snapshot 2024-05-27T00:17:16.367 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/34 file after the snapshot 2024-05-27T00:17:16.368 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/35 file after the snapshot 2024-05-27T00:17:16.369 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/36 file after the snapshot 2024-05-27T00:17:16.370 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/37 file after the snapshot 2024-05-27T00:17:16.370 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/38 file after the snapshot 2024-05-27T00:17:16.372 INFO:tasks.workunit.client.0.smithi098.stdout: create 260/39 file after the snapshot 2024-05-27T00:17:16.372 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/.snap/snap-subdir-test 2024-05-27T00:17:16.390 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/30 file after the snapshot 2024-05-27T00:17:16.390 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/31 file after the snapshot 2024-05-27T00:17:16.390 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/32 file after the snapshot 2024-05-27T00:17:16.390 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/33 file after the snapshot 2024-05-27T00:17:16.393 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/34 file after the snapshot 2024-05-27T00:17:16.393 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/35 file after the snapshot 2024-05-27T00:17:16.393 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/36 file after the snapshot 2024-05-27T00:17:16.393 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/37 file after the snapshot 2024-05-27T00:17:16.393 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/38 file after the snapshot 2024-05-27T00:17:16.394 INFO:tasks.workunit.client.0.smithi098.stdout: create 261/39 file after the snapshot 2024-05-27T00:17:16.395 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/.snap/snap-subdir-test 2024-05-27T00:17:16.409 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/30 file after the snapshot 2024-05-27T00:17:16.410 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/31 file after the snapshot 2024-05-27T00:17:16.411 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/32 file after the snapshot 2024-05-27T00:17:16.412 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/33 file after the snapshot 2024-05-27T00:17:16.414 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/34 file after the snapshot 2024-05-27T00:17:16.415 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/35 file after the snapshot 2024-05-27T00:17:16.416 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/36 file after the snapshot 2024-05-27T00:17:16.416 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/37 file after the snapshot 2024-05-27T00:17:16.417 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/38 file after the snapshot 2024-05-27T00:17:16.418 INFO:tasks.workunit.client.0.smithi098.stdout: create 262/39 file after the snapshot 2024-05-27T00:17:16.419 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/.snap/snap-subdir-test 2024-05-27T00:17:16.440 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/30 file after the snapshot 2024-05-27T00:17:16.440 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/31 file after the snapshot 2024-05-27T00:17:16.441 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/32 file after the snapshot 2024-05-27T00:17:16.442 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/33 file after the snapshot 2024-05-27T00:17:16.443 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/34 file after the snapshot 2024-05-27T00:17:16.444 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/35 file after the snapshot 2024-05-27T00:17:16.444 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/36 file after the snapshot 2024-05-27T00:17:16.445 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/37 file after the snapshot 2024-05-27T00:17:16.446 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/38 file after the snapshot 2024-05-27T00:17:16.447 INFO:tasks.workunit.client.0.smithi098.stdout: create 263/39 file after the snapshot 2024-05-27T00:17:16.448 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/.snap/snap-subdir-test 2024-05-27T00:17:16.468 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/30 file after the snapshot 2024-05-27T00:17:16.468 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/31 file after the snapshot 2024-05-27T00:17:16.470 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/32 file after the snapshot 2024-05-27T00:17:16.470 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/33 file after the snapshot 2024-05-27T00:17:16.471 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/34 file after the snapshot 2024-05-27T00:17:16.472 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/35 file after the snapshot 2024-05-27T00:17:16.473 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/36 file after the snapshot 2024-05-27T00:17:16.474 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/37 file after the snapshot 2024-05-27T00:17:16.475 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/38 file after the snapshot 2024-05-27T00:17:16.475 INFO:tasks.workunit.client.0.smithi098.stdout: create 264/39 file after the snapshot 2024-05-27T00:17:16.476 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/.snap/snap-subdir-test 2024-05-27T00:17:16.495 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/30 file after the snapshot 2024-05-27T00:17:16.495 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/31 file after the snapshot 2024-05-27T00:17:16.496 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/32 file after the snapshot 2024-05-27T00:17:16.497 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/33 file after the snapshot 2024-05-27T00:17:16.498 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/34 file after the snapshot 2024-05-27T00:17:16.498 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/35 file after the snapshot 2024-05-27T00:17:16.500 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/36 file after the snapshot 2024-05-27T00:17:16.500 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/37 file after the snapshot 2024-05-27T00:17:16.501 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/38 file after the snapshot 2024-05-27T00:17:16.502 INFO:tasks.workunit.client.0.smithi098.stdout: create 265/39 file after the snapshot 2024-05-27T00:17:16.503 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/.snap/snap-subdir-test 2024-05-27T00:17:16.521 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/30 file after the snapshot 2024-05-27T00:17:16.521 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/31 file after the snapshot 2024-05-27T00:17:16.522 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/32 file after the snapshot 2024-05-27T00:17:16.523 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/33 file after the snapshot 2024-05-27T00:17:16.524 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/34 file after the snapshot 2024-05-27T00:17:16.525 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/35 file after the snapshot 2024-05-27T00:17:16.526 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/36 file after the snapshot 2024-05-27T00:17:16.526 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/37 file after the snapshot 2024-05-27T00:17:16.527 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/38 file after the snapshot 2024-05-27T00:17:16.528 INFO:tasks.workunit.client.0.smithi098.stdout: create 266/39 file after the snapshot 2024-05-27T00:17:16.529 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/.snap/snap-subdir-test 2024-05-27T00:17:16.550 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/30 file after the snapshot 2024-05-27T00:17:16.550 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/31 file after the snapshot 2024-05-27T00:17:16.550 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/32 file after the snapshot 2024-05-27T00:17:16.550 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/33 file after the snapshot 2024-05-27T00:17:16.550 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/34 file after the snapshot 2024-05-27T00:17:16.551 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/35 file after the snapshot 2024-05-27T00:17:16.551 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/36 file after the snapshot 2024-05-27T00:17:16.552 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/37 file after the snapshot 2024-05-27T00:17:16.553 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/38 file after the snapshot 2024-05-27T00:17:16.554 INFO:tasks.workunit.client.0.smithi098.stdout: create 267/39 file after the snapshot 2024-05-27T00:17:16.554 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/.snap/snap-subdir-test 2024-05-27T00:17:16.572 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/30 file after the snapshot 2024-05-27T00:17:16.573 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/31 file after the snapshot 2024-05-27T00:17:16.573 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/32 file after the snapshot 2024-05-27T00:17:16.574 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/33 file after the snapshot 2024-05-27T00:17:16.575 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/34 file after the snapshot 2024-05-27T00:17:16.576 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/35 file after the snapshot 2024-05-27T00:17:16.577 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/36 file after the snapshot 2024-05-27T00:17:16.577 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/37 file after the snapshot 2024-05-27T00:17:16.578 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/38 file after the snapshot 2024-05-27T00:17:16.579 INFO:tasks.workunit.client.0.smithi098.stdout: create 268/39 file after the snapshot 2024-05-27T00:17:16.580 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/.snap/snap-subdir-test 2024-05-27T00:17:16.599 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/30 file after the snapshot 2024-05-27T00:17:16.600 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/31 file after the snapshot 2024-05-27T00:17:16.600 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/32 file after the snapshot 2024-05-27T00:17:16.601 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/33 file after the snapshot 2024-05-27T00:17:16.602 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/34 file after the snapshot 2024-05-27T00:17:16.603 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/35 file after the snapshot 2024-05-27T00:17:16.604 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/36 file after the snapshot 2024-05-27T00:17:16.605 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/37 file after the snapshot 2024-05-27T00:17:16.606 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/38 file after the snapshot 2024-05-27T00:17:16.607 INFO:tasks.workunit.client.0.smithi098.stdout: create 269/39 file after the snapshot 2024-05-27T00:17:16.608 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/.snap/snap-subdir-test 2024-05-27T00:17:16.625 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/30 file after the snapshot 2024-05-27T00:17:16.626 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/31 file after the snapshot 2024-05-27T00:17:16.627 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/32 file after the snapshot 2024-05-27T00:17:16.628 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/33 file after the snapshot 2024-05-27T00:17:16.628 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/34 file after the snapshot 2024-05-27T00:17:16.630 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/35 file after the snapshot 2024-05-27T00:17:16.630 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/36 file after the snapshot 2024-05-27T00:17:16.631 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/37 file after the snapshot 2024-05-27T00:17:16.632 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/38 file after the snapshot 2024-05-27T00:17:16.633 INFO:tasks.workunit.client.0.smithi098.stdout: create 270/39 file after the snapshot 2024-05-27T00:17:16.634 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/.snap/snap-subdir-test 2024-05-27T00:17:16.656 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/30 file after the snapshot 2024-05-27T00:17:16.657 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/31 file after the snapshot 2024-05-27T00:17:16.658 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/32 file after the snapshot 2024-05-27T00:17:16.659 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/33 file after the snapshot 2024-05-27T00:17:16.660 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/34 file after the snapshot 2024-05-27T00:17:16.660 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/35 file after the snapshot 2024-05-27T00:17:16.661 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/36 file after the snapshot 2024-05-27T00:17:16.662 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/37 file after the snapshot 2024-05-27T00:17:16.663 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/38 file after the snapshot 2024-05-27T00:17:16.664 INFO:tasks.workunit.client.0.smithi098.stdout: create 271/39 file after the snapshot 2024-05-27T00:17:16.665 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/.snap/snap-subdir-test 2024-05-27T00:17:16.683 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/30 file after the snapshot 2024-05-27T00:17:16.684 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/31 file after the snapshot 2024-05-27T00:17:16.685 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/32 file after the snapshot 2024-05-27T00:17:16.686 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/33 file after the snapshot 2024-05-27T00:17:16.687 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/34 file after the snapshot 2024-05-27T00:17:16.687 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/35 file after the snapshot 2024-05-27T00:17:16.688 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/36 file after the snapshot 2024-05-27T00:17:16.689 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/37 file after the snapshot 2024-05-27T00:17:16.690 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/38 file after the snapshot 2024-05-27T00:17:16.691 INFO:tasks.workunit.client.0.smithi098.stdout: create 272/39 file after the snapshot 2024-05-27T00:17:16.692 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/.snap/snap-subdir-test 2024-05-27T00:17:16.710 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/30 file after the snapshot 2024-05-27T00:17:16.710 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/31 file after the snapshot 2024-05-27T00:17:16.711 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/32 file after the snapshot 2024-05-27T00:17:16.712 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/33 file after the snapshot 2024-05-27T00:17:16.713 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/34 file after the snapshot 2024-05-27T00:17:16.714 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/35 file after the snapshot 2024-05-27T00:17:16.714 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/36 file after the snapshot 2024-05-27T00:17:16.716 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/37 file after the snapshot 2024-05-27T00:17:16.717 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/38 file after the snapshot 2024-05-27T00:17:16.717 INFO:tasks.workunit.client.0.smithi098.stdout: create 273/39 file after the snapshot 2024-05-27T00:17:16.718 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/.snap/snap-subdir-test 2024-05-27T00:17:16.737 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/30 file after the snapshot 2024-05-27T00:17:16.737 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/31 file after the snapshot 2024-05-27T00:17:16.738 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/32 file after the snapshot 2024-05-27T00:17:16.739 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/33 file after the snapshot 2024-05-27T00:17:16.739 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/34 file after the snapshot 2024-05-27T00:17:16.740 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/35 file after the snapshot 2024-05-27T00:17:16.741 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/36 file after the snapshot 2024-05-27T00:17:16.742 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/37 file after the snapshot 2024-05-27T00:17:16.743 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/38 file after the snapshot 2024-05-27T00:17:16.744 INFO:tasks.workunit.client.0.smithi098.stdout: create 274/39 file after the snapshot 2024-05-27T00:17:16.745 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/.snap/snap-subdir-test 2024-05-27T00:17:16.763 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/30 file after the snapshot 2024-05-27T00:17:16.763 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/31 file after the snapshot 2024-05-27T00:17:16.764 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/32 file after the snapshot 2024-05-27T00:17:16.765 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/33 file after the snapshot 2024-05-27T00:17:16.766 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/34 file after the snapshot 2024-05-27T00:17:16.767 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/35 file after the snapshot 2024-05-27T00:17:16.767 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/36 file after the snapshot 2024-05-27T00:17:16.768 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/37 file after the snapshot 2024-05-27T00:17:16.769 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/38 file after the snapshot 2024-05-27T00:17:16.770 INFO:tasks.workunit.client.0.smithi098.stdout: create 275/39 file after the snapshot 2024-05-27T00:17:16.771 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/.snap/snap-subdir-test 2024-05-27T00:17:16.789 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/30 file after the snapshot 2024-05-27T00:17:16.790 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/31 file after the snapshot 2024-05-27T00:17:16.790 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/32 file after the snapshot 2024-05-27T00:17:16.791 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/33 file after the snapshot 2024-05-27T00:17:16.792 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/34 file after the snapshot 2024-05-27T00:17:16.793 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/35 file after the snapshot 2024-05-27T00:17:16.793 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/36 file after the snapshot 2024-05-27T00:17:16.794 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/37 file after the snapshot 2024-05-27T00:17:16.795 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/38 file after the snapshot 2024-05-27T00:17:16.798 INFO:tasks.workunit.client.0.smithi098.stdout: create 276/39 file after the snapshot 2024-05-27T00:17:16.798 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/.snap/snap-subdir-test 2024-05-27T00:17:16.815 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/30 file after the snapshot 2024-05-27T00:17:16.816 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/31 file after the snapshot 2024-05-27T00:17:16.817 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/32 file after the snapshot 2024-05-27T00:17:16.817 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/33 file after the snapshot 2024-05-27T00:17:16.818 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/34 file after the snapshot 2024-05-27T00:17:16.819 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/35 file after the snapshot 2024-05-27T00:17:16.820 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/36 file after the snapshot 2024-05-27T00:17:16.821 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/37 file after the snapshot 2024-05-27T00:17:16.822 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/38 file after the snapshot 2024-05-27T00:17:16.822 INFO:tasks.workunit.client.0.smithi098.stdout: create 277/39 file after the snapshot 2024-05-27T00:17:16.823 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/.snap/snap-subdir-test 2024-05-27T00:17:16.844 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/30 file after the snapshot 2024-05-27T00:17:16.844 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/31 file after the snapshot 2024-05-27T00:17:16.844 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/32 file after the snapshot 2024-05-27T00:17:16.845 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/33 file after the snapshot 2024-05-27T00:17:16.845 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/34 file after the snapshot 2024-05-27T00:17:16.847 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/35 file after the snapshot 2024-05-27T00:17:16.847 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/36 file after the snapshot 2024-05-27T00:17:16.848 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/37 file after the snapshot 2024-05-27T00:17:16.849 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/38 file after the snapshot 2024-05-27T00:17:16.850 INFO:tasks.workunit.client.0.smithi098.stdout: create 278/39 file after the snapshot 2024-05-27T00:17:16.850 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/.snap/snap-subdir-test 2024-05-27T00:17:16.869 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/30 file after the snapshot 2024-05-27T00:17:16.869 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/31 file after the snapshot 2024-05-27T00:17:16.870 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/32 file after the snapshot 2024-05-27T00:17:16.871 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/33 file after the snapshot 2024-05-27T00:17:16.872 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/34 file after the snapshot 2024-05-27T00:17:16.873 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/35 file after the snapshot 2024-05-27T00:17:16.874 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/36 file after the snapshot 2024-05-27T00:17:16.874 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/37 file after the snapshot 2024-05-27T00:17:16.875 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/38 file after the snapshot 2024-05-27T00:17:16.876 INFO:tasks.workunit.client.0.smithi098.stdout: create 279/39 file after the snapshot 2024-05-27T00:17:16.877 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/.snap/snap-subdir-test 2024-05-27T00:17:16.896 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/30 file after the snapshot 2024-05-27T00:17:16.896 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/31 file after the snapshot 2024-05-27T00:17:16.897 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/32 file after the snapshot 2024-05-27T00:17:16.898 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/33 file after the snapshot 2024-05-27T00:17:16.899 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/34 file after the snapshot 2024-05-27T00:17:16.900 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/35 file after the snapshot 2024-05-27T00:17:16.901 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/36 file after the snapshot 2024-05-27T00:17:16.901 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/37 file after the snapshot 2024-05-27T00:17:16.902 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/38 file after the snapshot 2024-05-27T00:17:16.903 INFO:tasks.workunit.client.0.smithi098.stdout: create 280/39 file after the snapshot 2024-05-27T00:17:16.904 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/.snap/snap-subdir-test 2024-05-27T00:17:16.923 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/30 file after the snapshot 2024-05-27T00:17:16.923 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/31 file after the snapshot 2024-05-27T00:17:16.924 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/32 file after the snapshot 2024-05-27T00:17:16.925 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/33 file after the snapshot 2024-05-27T00:17:16.926 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/34 file after the snapshot 2024-05-27T00:17:16.927 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/35 file after the snapshot 2024-05-27T00:17:16.927 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/36 file after the snapshot 2024-05-27T00:17:16.928 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/37 file after the snapshot 2024-05-27T00:17:16.929 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/38 file after the snapshot 2024-05-27T00:17:16.930 INFO:tasks.workunit.client.0.smithi098.stdout: create 281/39 file after the snapshot 2024-05-27T00:17:16.931 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/.snap/snap-subdir-test 2024-05-27T00:17:16.949 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/30 file after the snapshot 2024-05-27T00:17:16.950 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/31 file after the snapshot 2024-05-27T00:17:16.951 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/32 file after the snapshot 2024-05-27T00:17:16.952 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/33 file after the snapshot 2024-05-27T00:17:16.953 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/34 file after the snapshot 2024-05-27T00:17:16.953 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/35 file after the snapshot 2024-05-27T00:17:16.954 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/36 file after the snapshot 2024-05-27T00:17:16.955 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/37 file after the snapshot 2024-05-27T00:17:16.956 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/38 file after the snapshot 2024-05-27T00:17:16.957 INFO:tasks.workunit.client.0.smithi098.stdout: create 282/39 file after the snapshot 2024-05-27T00:17:16.957 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/.snap/snap-subdir-test 2024-05-27T00:17:16.979 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/30 file after the snapshot 2024-05-27T00:17:16.979 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/31 file after the snapshot 2024-05-27T00:17:16.980 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/32 file after the snapshot 2024-05-27T00:17:16.981 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/33 file after the snapshot 2024-05-27T00:17:16.982 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/34 file after the snapshot 2024-05-27T00:17:16.982 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/35 file after the snapshot 2024-05-27T00:17:16.983 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/36 file after the snapshot 2024-05-27T00:17:16.984 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/37 file after the snapshot 2024-05-27T00:17:16.985 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/38 file after the snapshot 2024-05-27T00:17:16.986 INFO:tasks.workunit.client.0.smithi098.stdout: create 283/39 file after the snapshot 2024-05-27T00:17:16.986 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/.snap/snap-subdir-test 2024-05-27T00:17:17.004 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/30 file after the snapshot 2024-05-27T00:17:17.005 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/31 file after the snapshot 2024-05-27T00:17:17.005 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/32 file after the snapshot 2024-05-27T00:17:17.006 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/33 file after the snapshot 2024-05-27T00:17:17.007 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/34 file after the snapshot 2024-05-27T00:17:17.008 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/35 file after the snapshot 2024-05-27T00:17:17.009 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/36 file after the snapshot 2024-05-27T00:17:17.010 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/37 file after the snapshot 2024-05-27T00:17:17.011 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/38 file after the snapshot 2024-05-27T00:17:17.011 INFO:tasks.workunit.client.0.smithi098.stdout: create 284/39 file after the snapshot 2024-05-27T00:17:17.012 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/.snap/snap-subdir-test 2024-05-27T00:17:17.030 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/30 file after the snapshot 2024-05-27T00:17:17.031 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/31 file after the snapshot 2024-05-27T00:17:17.032 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/32 file after the snapshot 2024-05-27T00:17:17.032 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/33 file after the snapshot 2024-05-27T00:17:17.033 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/34 file after the snapshot 2024-05-27T00:17:17.034 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/35 file after the snapshot 2024-05-27T00:17:17.035 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/36 file after the snapshot 2024-05-27T00:17:17.036 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/37 file after the snapshot 2024-05-27T00:17:17.037 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/38 file after the snapshot 2024-05-27T00:17:17.037 INFO:tasks.workunit.client.0.smithi098.stdout: create 285/39 file after the snapshot 2024-05-27T00:17:17.038 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/.snap/snap-subdir-test 2024-05-27T00:17:17.057 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/30 file after the snapshot 2024-05-27T00:17:17.057 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/31 file after the snapshot 2024-05-27T00:17:17.058 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/32 file after the snapshot 2024-05-27T00:17:17.059 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/33 file after the snapshot 2024-05-27T00:17:17.060 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/34 file after the snapshot 2024-05-27T00:17:17.061 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/35 file after the snapshot 2024-05-27T00:17:17.062 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/36 file after the snapshot 2024-05-27T00:17:17.062 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/37 file after the snapshot 2024-05-27T00:17:17.063 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/38 file after the snapshot 2024-05-27T00:17:17.064 INFO:tasks.workunit.client.0.smithi098.stdout: create 286/39 file after the snapshot 2024-05-27T00:17:17.068 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/.snap/snap-subdir-test 2024-05-27T00:17:17.085 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/30 file after the snapshot 2024-05-27T00:17:17.086 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/31 file after the snapshot 2024-05-27T00:17:17.087 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/32 file after the snapshot 2024-05-27T00:17:17.088 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/33 file after the snapshot 2024-05-27T00:17:17.089 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/34 file after the snapshot 2024-05-27T00:17:17.090 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/35 file after the snapshot 2024-05-27T00:17:17.091 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/36 file after the snapshot 2024-05-27T00:17:17.092 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/37 file after the snapshot 2024-05-27T00:17:17.093 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/38 file after the snapshot 2024-05-27T00:17:17.093 INFO:tasks.workunit.client.0.smithi098.stdout: create 287/39 file after the snapshot 2024-05-27T00:17:17.095 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/.snap/snap-subdir-test 2024-05-27T00:17:17.112 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/30 file after the snapshot 2024-05-27T00:17:17.114 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/31 file after the snapshot 2024-05-27T00:17:17.114 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/32 file after the snapshot 2024-05-27T00:17:17.115 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/33 file after the snapshot 2024-05-27T00:17:17.116 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/34 file after the snapshot 2024-05-27T00:17:17.117 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/35 file after the snapshot 2024-05-27T00:17:17.118 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/36 file after the snapshot 2024-05-27T00:17:17.119 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/37 file after the snapshot 2024-05-27T00:17:17.119 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/38 file after the snapshot 2024-05-27T00:17:17.120 INFO:tasks.workunit.client.0.smithi098.stdout: create 288/39 file after the snapshot 2024-05-27T00:17:17.122 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/.snap/snap-subdir-test 2024-05-27T00:17:17.139 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/30 file after the snapshot 2024-05-27T00:17:17.140 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/31 file after the snapshot 2024-05-27T00:17:17.141 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/32 file after the snapshot 2024-05-27T00:17:17.142 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/33 file after the snapshot 2024-05-27T00:17:17.142 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/34 file after the snapshot 2024-05-27T00:17:17.143 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/35 file after the snapshot 2024-05-27T00:17:17.144 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/36 file after the snapshot 2024-05-27T00:17:17.145 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/37 file after the snapshot 2024-05-27T00:17:17.146 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/38 file after the snapshot 2024-05-27T00:17:17.147 INFO:tasks.workunit.client.0.smithi098.stdout: create 289/39 file after the snapshot 2024-05-27T00:17:17.148 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/.snap/snap-subdir-test 2024-05-27T00:17:17.166 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/30 file after the snapshot 2024-05-27T00:17:17.167 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/31 file after the snapshot 2024-05-27T00:17:17.168 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/32 file after the snapshot 2024-05-27T00:17:17.168 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/33 file after the snapshot 2024-05-27T00:17:17.169 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/34 file after the snapshot 2024-05-27T00:17:17.170 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/35 file after the snapshot 2024-05-27T00:17:17.171 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/36 file after the snapshot 2024-05-27T00:17:17.172 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/37 file after the snapshot 2024-05-27T00:17:17.172 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/38 file after the snapshot 2024-05-27T00:17:17.174 INFO:tasks.workunit.client.0.smithi098.stdout: create 290/39 file after the snapshot 2024-05-27T00:17:17.175 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/.snap/snap-subdir-test 2024-05-27T00:17:17.194 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/30 file after the snapshot 2024-05-27T00:17:17.194 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/31 file after the snapshot 2024-05-27T00:17:17.195 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/32 file after the snapshot 2024-05-27T00:17:17.196 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/33 file after the snapshot 2024-05-27T00:17:17.197 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/34 file after the snapshot 2024-05-27T00:17:17.198 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/35 file after the snapshot 2024-05-27T00:17:17.199 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/36 file after the snapshot 2024-05-27T00:17:17.199 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/37 file after the snapshot 2024-05-27T00:17:17.200 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/38 file after the snapshot 2024-05-27T00:17:17.201 INFO:tasks.workunit.client.0.smithi098.stdout: create 291/39 file after the snapshot 2024-05-27T00:17:17.202 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/.snap/snap-subdir-test 2024-05-27T00:17:17.220 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/30 file after the snapshot 2024-05-27T00:17:17.221 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/31 file after the snapshot 2024-05-27T00:17:17.222 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/32 file after the snapshot 2024-05-27T00:17:17.223 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/33 file after the snapshot 2024-05-27T00:17:17.224 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/34 file after the snapshot 2024-05-27T00:17:17.225 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/35 file after the snapshot 2024-05-27T00:17:17.225 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/36 file after the snapshot 2024-05-27T00:17:17.226 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/37 file after the snapshot 2024-05-27T00:17:17.227 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/38 file after the snapshot 2024-05-27T00:17:17.228 INFO:tasks.workunit.client.0.smithi098.stdout: create 292/39 file after the snapshot 2024-05-27T00:17:17.229 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/.snap/snap-subdir-test 2024-05-27T00:17:17.247 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/30 file after the snapshot 2024-05-27T00:17:17.248 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/31 file after the snapshot 2024-05-27T00:17:17.248 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/32 file after the snapshot 2024-05-27T00:17:17.249 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/33 file after the snapshot 2024-05-27T00:17:17.251 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/34 file after the snapshot 2024-05-27T00:17:17.251 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/35 file after the snapshot 2024-05-27T00:17:17.252 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/36 file after the snapshot 2024-05-27T00:17:17.253 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/37 file after the snapshot 2024-05-27T00:17:17.254 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/38 file after the snapshot 2024-05-27T00:17:17.255 INFO:tasks.workunit.client.0.smithi098.stdout: create 293/39 file after the snapshot 2024-05-27T00:17:17.255 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/.snap/snap-subdir-test 2024-05-27T00:17:17.274 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/30 file after the snapshot 2024-05-27T00:17:17.274 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/31 file after the snapshot 2024-05-27T00:17:17.275 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/32 file after the snapshot 2024-05-27T00:17:17.276 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/33 file after the snapshot 2024-05-27T00:17:17.277 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/34 file after the snapshot 2024-05-27T00:17:17.278 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/35 file after the snapshot 2024-05-27T00:17:17.279 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/36 file after the snapshot 2024-05-27T00:17:17.279 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/37 file after the snapshot 2024-05-27T00:17:17.280 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/38 file after the snapshot 2024-05-27T00:17:17.281 INFO:tasks.workunit.client.0.smithi098.stdout: create 294/39 file after the snapshot 2024-05-27T00:17:17.282 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/.snap/snap-subdir-test 2024-05-27T00:17:17.300 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/30 file after the snapshot 2024-05-27T00:17:17.301 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/31 file after the snapshot 2024-05-27T00:17:17.302 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/32 file after the snapshot 2024-05-27T00:17:17.303 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/33 file after the snapshot 2024-05-27T00:17:17.304 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/34 file after the snapshot 2024-05-27T00:17:17.305 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/35 file after the snapshot 2024-05-27T00:17:17.306 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/36 file after the snapshot 2024-05-27T00:17:17.306 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/37 file after the snapshot 2024-05-27T00:17:17.307 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/38 file after the snapshot 2024-05-27T00:17:17.308 INFO:tasks.workunit.client.0.smithi098.stdout: create 295/39 file after the snapshot 2024-05-27T00:17:17.309 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/.snap/snap-subdir-test 2024-05-27T00:17:17.327 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/30 file after the snapshot 2024-05-27T00:17:17.328 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/31 file after the snapshot 2024-05-27T00:17:17.329 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/32 file after the snapshot 2024-05-27T00:17:17.329 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/33 file after the snapshot 2024-05-27T00:17:17.330 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/34 file after the snapshot 2024-05-27T00:17:17.331 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/35 file after the snapshot 2024-05-27T00:17:17.331 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/36 file after the snapshot 2024-05-27T00:17:17.332 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/37 file after the snapshot 2024-05-27T00:17:17.333 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/38 file after the snapshot 2024-05-27T00:17:17.334 INFO:tasks.workunit.client.0.smithi098.stdout: create 296/39 file after the snapshot 2024-05-27T00:17:17.335 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/.snap/snap-subdir-test 2024-05-27T00:17:17.354 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/30 file after the snapshot 2024-05-27T00:17:17.354 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/31 file after the snapshot 2024-05-27T00:17:17.355 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/32 file after the snapshot 2024-05-27T00:17:17.356 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/33 file after the snapshot 2024-05-27T00:17:17.357 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/34 file after the snapshot 2024-05-27T00:17:17.358 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/35 file after the snapshot 2024-05-27T00:17:17.359 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/36 file after the snapshot 2024-05-27T00:17:17.359 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/37 file after the snapshot 2024-05-27T00:17:17.360 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/38 file after the snapshot 2024-05-27T00:17:17.361 INFO:tasks.workunit.client.0.smithi098.stdout: create 297/39 file after the snapshot 2024-05-27T00:17:17.362 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/.snap/snap-subdir-test 2024-05-27T00:17:17.381 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/30 file after the snapshot 2024-05-27T00:17:17.382 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/31 file after the snapshot 2024-05-27T00:17:17.383 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/32 file after the snapshot 2024-05-27T00:17:17.384 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/33 file after the snapshot 2024-05-27T00:17:17.385 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/34 file after the snapshot 2024-05-27T00:17:17.386 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/35 file after the snapshot 2024-05-27T00:17:17.386 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/36 file after the snapshot 2024-05-27T00:17:17.387 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/37 file after the snapshot 2024-05-27T00:17:17.388 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/38 file after the snapshot 2024-05-27T00:17:17.389 INFO:tasks.workunit.client.0.smithi098.stdout: create 298/39 file after the snapshot 2024-05-27T00:17:17.390 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/.snap/snap-subdir-test 2024-05-27T00:17:17.410 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/30 file after the snapshot 2024-05-27T00:17:17.411 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/31 file after the snapshot 2024-05-27T00:17:17.412 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/32 file after the snapshot 2024-05-27T00:17:17.412 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/33 file after the snapshot 2024-05-27T00:17:17.413 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/34 file after the snapshot 2024-05-27T00:17:17.414 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/35 file after the snapshot 2024-05-27T00:17:17.415 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/36 file after the snapshot 2024-05-27T00:17:17.416 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/37 file after the snapshot 2024-05-27T00:17:17.416 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/38 file after the snapshot 2024-05-27T00:17:17.417 INFO:tasks.workunit.client.0.smithi098.stdout: create 299/39 file after the snapshot 2024-05-27T00:17:17.418 INFO:tasks.workunit.client.0.smithi098.stdout:Take second snapshot .snap/test2 2024-05-27T00:17:17.464 INFO:tasks.workunit.client.0.smithi098.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2024-05-27T00:17:19.187 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2024-05-27T00:17:19.188 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2024-05-27T00:17:19.188 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2024-05-27T00:17:19.188 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2024-05-27T00:17:19.188 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2024-05-27T00:17:19.188 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2024-05-27T00:17:19.188 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2024-05-27T00:17:19.188 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2024-05-27T00:17:19.189 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2024-05-27T00:17:19.190 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2024-05-27T00:17:19.190 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2024-05-27T00:17:19.190 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2024-05-27T00:17:19.190 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2024-05-27T00:17:19.190 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2024-05-27T00:17:19.190 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2024-05-27T00:17:19.190 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2024-05-27T00:17:19.190 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2024-05-27T00:17:19.190 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2024-05-27T00:17:19.190 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2024-05-27T00:17:19.191 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2024-05-27T00:17:19.191 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2024-05-27T00:17:19.191 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2024-05-27T00:17:19.191 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2024-05-27T00:17:19.191 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2024-05-27T00:17:19.191 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2024-05-27T00:17:19.191 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2024-05-27T00:17:19.191 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2024-05-27T00:17:19.191 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2024-05-27T00:17:19.191 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2024-05-27T00:17:19.192 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2024-05-27T00:17:19.193 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2024-05-27T00:17:19.193 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2024-05-27T00:17:19.193 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2024-05-27T00:17:19.193 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2024-05-27T00:17:19.193 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2024-05-27T00:17:19.193 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2024-05-27T00:17:19.193 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2024-05-27T00:17:19.193 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2024-05-27T00:17:19.193 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2024-05-27T00:17:19.194 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2024-05-27T00:17:19.195 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2024-05-27T00:17:19.195 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2024-05-27T00:17:19.195 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2024-05-27T00:17:19.195 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2024-05-27T00:17:19.195 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2024-05-27T00:17:19.195 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2024-05-27T00:17:19.195 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2024-05-27T00:17:19.195 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2024-05-27T00:17:19.195 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2024-05-27T00:17:19.195 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2024-05-27T00:17:19.356 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2024-05-27T00:17:19.356 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2024-05-27T00:17:19.356 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2024-05-27T00:17:19.357 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2024-05-27T00:17:19.357 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2024-05-27T00:17:19.357 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2024-05-27T00:17:19.358 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2024-05-27T00:17:19.358 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2024-05-27T00:17:19.358 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2024-05-27T00:17:19.358 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2024-05-27T00:17:19.358 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2024-05-27T00:17:19.359 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2024-05-27T00:17:19.359 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2024-05-27T00:17:19.359 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2024-05-27T00:17:19.359 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2024-05-27T00:17:19.359 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2024-05-27T00:17:19.359 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2024-05-27T00:17:19.359 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2024-05-27T00:17:19.359 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2024-05-27T00:17:19.360 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2024-05-27T00:17:19.361 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2024-05-27T00:17:19.362 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2024-05-27T00:17:19.362 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2024-05-27T00:17:19.362 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2024-05-27T00:17:19.362 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2024-05-27T00:17:19.362 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2024-05-27T00:17:19.362 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2024-05-27T00:17:19.362 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2024-05-27T00:17:19.362 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2024-05-27T00:17:19.362 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2024-05-27T00:17:19.363 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2024-05-27T00:17:19.363 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2024-05-27T00:17:19.363 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2024-05-27T00:17:19.363 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2024-05-27T00:17:19.363 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2024-05-27T00:17:19.363 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2024-05-27T00:17:19.363 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2024-05-27T00:17:19.363 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2024-05-27T00:17:19.363 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2024-05-27T00:17:19.363 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2024-05-27T00:17:19.364 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2024-05-27T00:17:19.365 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2024-05-27T00:17:19.365 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2024-05-27T00:17:19.538 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2024-05-27T00:17:19.538 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2024-05-27T00:17:19.538 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2024-05-27T00:17:19.538 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2024-05-27T00:17:19.538 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2024-05-27T00:17:19.538 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2024-05-27T00:17:19.538 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2024-05-27T00:17:19.539 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2024-05-27T00:17:19.540 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2024-05-27T00:17:19.540 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2024-05-27T00:17:19.540 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2024-05-27T00:17:19.540 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2024-05-27T00:17:19.540 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2024-05-27T00:17:19.540 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2024-05-27T00:17:19.540 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2024-05-27T00:17:19.540 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2024-05-27T00:17:19.540 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2024-05-27T00:17:19.541 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2024-05-27T00:17:19.542 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2024-05-27T00:17:19.543 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2024-05-27T00:17:19.544 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2024-05-27T00:17:19.545 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2024-05-27T00:17:19.545 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2024-05-27T00:17:19.545 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2024-05-27T00:17:19.545 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2024-05-27T00:17:19.545 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2024-05-27T00:17:19.721 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2024-05-27T00:17:19.721 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2024-05-27T00:17:19.721 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2024-05-27T00:17:19.722 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2024-05-27T00:17:19.723 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2024-05-27T00:17:19.723 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2024-05-27T00:17:19.723 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2024-05-27T00:17:19.723 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2024-05-27T00:17:19.723 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2024-05-27T00:17:19.723 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2024-05-27T00:17:19.723 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2024-05-27T00:17:19.723 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2024-05-27T00:17:19.723 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2024-05-27T00:17:19.723 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2024-05-27T00:17:19.724 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2024-05-27T00:17:19.725 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2024-05-27T00:17:19.725 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2024-05-27T00:17:19.725 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2024-05-27T00:17:19.725 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2024-05-27T00:17:19.725 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2024-05-27T00:17:19.725 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2024-05-27T00:17:19.725 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2024-05-27T00:17:19.725 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2024-05-27T00:17:19.725 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2024-05-27T00:17:19.725 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2024-05-27T00:17:19.726 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2024-05-27T00:17:19.727 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2024-05-27T00:17:19.727 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2024-05-27T00:17:19.727 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2024-05-27T00:17:19.727 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2024-05-27T00:17:19.727 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2024-05-27T00:17:19.727 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2024-05-27T00:17:19.727 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2024-05-27T00:17:19.727 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2024-05-27T00:17:19.727 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2024-05-27T00:17:19.727 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2024-05-27T00:17:19.728 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2024-05-27T00:17:19.728 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2024-05-27T00:17:19.728 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2024-05-27T00:17:19.728 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2024-05-27T00:17:19.728 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2024-05-27T00:17:19.728 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2024-05-27T00:17:19.728 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2024-05-27T00:17:19.904 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2024-05-27T00:17:19.904 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2024-05-27T00:17:19.905 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2024-05-27T00:17:19.906 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2024-05-27T00:17:19.907 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2024-05-27T00:17:19.908 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2024-05-27T00:17:19.909 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2024-05-27T00:17:19.910 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2024-05-27T00:17:19.911 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2024-05-27T00:17:19.911 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2024-05-27T00:17:19.911 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2024-05-27T00:17:19.911 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2024-05-27T00:17:19.911 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2024-05-27T00:17:19.911 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2024-05-27T00:17:19.911 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2024-05-27T00:17:20.088 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2024-05-27T00:17:20.089 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2024-05-27T00:17:20.089 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2024-05-27T00:17:20.089 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2024-05-27T00:17:20.089 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2024-05-27T00:17:20.089 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2024-05-27T00:17:20.090 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2024-05-27T00:17:20.090 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2024-05-27T00:17:20.090 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2024-05-27T00:17:20.090 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2024-05-27T00:17:20.090 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2024-05-27T00:17:20.091 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2024-05-27T00:17:20.091 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2024-05-27T00:17:20.091 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2024-05-27T00:17:20.091 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2024-05-27T00:17:20.091 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2024-05-27T00:17:20.091 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2024-05-27T00:17:20.092 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2024-05-27T00:17:20.092 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2024-05-27T00:17:20.092 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2024-05-27T00:17:20.092 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2024-05-27T00:17:20.092 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2024-05-27T00:17:20.092 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2024-05-27T00:17:20.093 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2024-05-27T00:17:20.093 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2024-05-27T00:17:20.093 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2024-05-27T00:17:20.093 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2024-05-27T00:17:20.093 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2024-05-27T00:17:20.093 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2024-05-27T00:17:20.094 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2024-05-27T00:17:20.094 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2024-05-27T00:17:20.094 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2024-05-27T00:17:20.094 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2024-05-27T00:17:20.094 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2024-05-27T00:17:20.094 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2024-05-27T00:17:20.095 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2024-05-27T00:17:20.095 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2024-05-27T00:17:20.095 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2024-05-27T00:17:20.095 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2024-05-27T00:17:20.095 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2024-05-27T00:17:20.095 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2024-05-27T00:17:20.096 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2024-05-27T00:17:20.096 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2024-05-27T00:17:20.096 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2024-05-27T00:17:20.096 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2024-05-27T00:17:20.096 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2024-05-27T00:17:20.097 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2024-05-27T00:17:20.097 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2024-05-27T00:17:20.097 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2024-05-27T00:17:20.097 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2024-05-27T00:17:20.097 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2024-05-27T00:17:20.097 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2024-05-27T00:17:20.098 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2024-05-27T00:17:20.098 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2024-05-27T00:17:20.098 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2024-05-27T00:17:20.098 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2024-05-27T00:17:20.098 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2024-05-27T00:17:20.098 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2024-05-27T00:17:20.099 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2024-05-27T00:17:20.099 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2024-05-27T00:17:20.099 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2024-05-27T00:17:20.099 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2024-05-27T00:17:20.099 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2024-05-27T00:17:20.099 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2024-05-27T00:17:20.100 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2024-05-27T00:17:20.100 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2024-05-27T00:17:20.100 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2024-05-27T00:17:20.100 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2024-05-27T00:17:20.100 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2024-05-27T00:17:20.100 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2024-05-27T00:17:20.101 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2024-05-27T00:17:20.101 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2024-05-27T00:17:20.101 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2024-05-27T00:17:20.101 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2024-05-27T00:17:20.101 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2024-05-27T00:17:20.102 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2024-05-27T00:17:20.102 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2024-05-27T00:17:20.102 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2024-05-27T00:17:20.102 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2024-05-27T00:17:20.102 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2024-05-27T00:17:20.102 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2024-05-27T00:17:20.283 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2024-05-27T00:17:20.283 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2024-05-27T00:17:20.283 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2024-05-27T00:17:20.283 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2024-05-27T00:17:20.283 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2024-05-27T00:17:20.283 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2024-05-27T00:17:20.283 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2024-05-27T00:17:20.284 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2024-05-27T00:17:20.285 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2024-05-27T00:17:20.286 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2024-05-27T00:17:20.287 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2024-05-27T00:17:20.288 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2024-05-27T00:17:20.289 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2024-05-27T00:17:20.289 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2024-05-27T00:17:20.289 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2024-05-27T00:17:20.289 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2024-05-27T00:17:20.289 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2024-05-27T00:17:20.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2024-05-27T00:17:20.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2024-05-27T00:17:20.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2024-05-27T00:17:20.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2024-05-27T00:17:20.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2024-05-27T00:17:20.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2024-05-27T00:17:20.469 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2024-05-27T00:17:20.469 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2024-05-27T00:17:20.469 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2024-05-27T00:17:20.469 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2024-05-27T00:17:20.469 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2024-05-27T00:17:20.469 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2024-05-27T00:17:20.470 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2024-05-27T00:17:20.470 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2024-05-27T00:17:20.470 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2024-05-27T00:17:20.470 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2024-05-27T00:17:20.470 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2024-05-27T00:17:20.470 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2024-05-27T00:17:20.470 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2024-05-27T00:17:20.868 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2024-05-27T00:17:20.868 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2024-05-27T00:17:20.868 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2024-05-27T00:17:20.868 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2024-05-27T00:17:20.868 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2024-05-27T00:17:20.868 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2024-05-27T00:17:20.869 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2024-05-27T00:17:20.870 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2024-05-27T00:17:20.871 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2024-05-27T00:17:20.872 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2024-05-27T00:17:20.872 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2024-05-27T00:17:20.872 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2024-05-27T00:17:20.872 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2024-05-27T00:17:20.872 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2024-05-27T00:17:20.872 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2024-05-27T00:17:20.873 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2024-05-27T00:17:20.873 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2024-05-27T00:17:20.873 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2024-05-27T00:17:20.873 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2024-05-27T00:17:20.873 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2024-05-27T00:17:20.873 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2024-05-27T00:17:20.873 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2024-05-27T00:17:20.873 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2024-05-27T00:17:20.873 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2024-05-27T00:17:20.874 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2024-05-27T00:17:20.874 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2024-05-27T00:17:20.874 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2024-05-27T00:17:20.874 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2024-05-27T00:17:20.874 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2024-05-27T00:17:20.874 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2024-05-27T00:17:20.874 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2024-05-27T00:17:20.874 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2024-05-27T00:17:20.874 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2024-05-27T00:17:20.874 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2024-05-27T00:17:20.875 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2024-05-27T00:17:20.875 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2024-05-27T00:17:20.875 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2024-05-27T00:17:20.875 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2024-05-27T00:17:20.875 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2024-05-27T00:17:20.875 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2024-05-27T00:17:20.875 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2024-05-27T00:17:20.875 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2024-05-27T00:17:20.875 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2024-05-27T00:17:20.876 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2024-05-27T00:17:20.876 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2024-05-27T00:17:20.876 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2024-05-27T00:17:20.876 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2024-05-27T00:17:21.016 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2024-05-27T00:17:21.016 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2024-05-27T00:17:21.016 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2024-05-27T00:17:21.016 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2024-05-27T00:17:21.016 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2024-05-27T00:17:21.017 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2024-05-27T00:17:21.018 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2024-05-27T00:17:21.019 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2024-05-27T00:17:21.020 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2024-05-27T00:17:21.021 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2024-05-27T00:17:21.022 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2024-05-27T00:17:21.023 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2024-05-27T00:17:21.168 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2024-05-27T00:17:21.168 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2024-05-27T00:17:21.169 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2024-05-27T00:17:21.169 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2024-05-27T00:17:21.169 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2024-05-27T00:17:21.169 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2024-05-27T00:17:21.169 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2024-05-27T00:17:21.169 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2024-05-27T00:17:21.169 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2024-05-27T00:17:21.169 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2024-05-27T00:17:21.170 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2024-05-27T00:17:21.171 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2024-05-27T00:17:21.172 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2024-05-27T00:17:21.173 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2024-05-27T00:17:21.173 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2024-05-27T00:17:21.173 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2024-05-27T00:17:21.173 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2024-05-27T00:17:21.173 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2024-05-27T00:17:21.173 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2024-05-27T00:17:21.173 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2024-05-27T00:17:21.173 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2024-05-27T00:17:21.173 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2024-05-27T00:17:21.173 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2024-05-27T00:17:21.174 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2024-05-27T00:17:21.175 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2024-05-27T00:17:21.175 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2024-05-27T00:17:21.175 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2024-05-27T00:17:21.175 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2024-05-27T00:17:21.175 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2024-05-27T00:17:21.175 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2024-05-27T00:17:21.175 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2024-05-27T00:17:21.175 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2024-05-27T00:17:21.316 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2024-05-27T00:17:21.317 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2024-05-27T00:17:21.318 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2024-05-27T00:17:21.319 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2024-05-27T00:17:21.320 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2024-05-27T00:17:21.321 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2024-05-27T00:17:21.322 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2024-05-27T00:17:21.322 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2024-05-27T00:17:21.322 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2024-05-27T00:17:21.459 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2024-05-27T00:17:21.459 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2024-05-27T00:17:21.459 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2024-05-27T00:17:21.460 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2024-05-27T00:17:21.460 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2024-05-27T00:17:21.461 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2024-05-27T00:17:21.461 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2024-05-27T00:17:21.461 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2024-05-27T00:17:21.461 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2024-05-27T00:17:21.461 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2024-05-27T00:17:21.461 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2024-05-27T00:17:21.462 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2024-05-27T00:17:21.463 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2024-05-27T00:17:21.464 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2024-05-27T00:17:21.465 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2024-05-27T00:17:21.466 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2024-05-27T00:17:21.467 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2024-05-27T00:17:21.609 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2024-05-27T00:17:21.609 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2024-05-27T00:17:21.609 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2024-05-27T00:17:21.609 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2024-05-27T00:17:21.610 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2024-05-27T00:17:21.610 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2024-05-27T00:17:21.610 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2024-05-27T00:17:21.610 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2024-05-27T00:17:21.610 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2024-05-27T00:17:21.610 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2024-05-27T00:17:21.611 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2024-05-27T00:17:21.611 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2024-05-27T00:17:21.611 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2024-05-27T00:17:21.611 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2024-05-27T00:17:21.611 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2024-05-27T00:17:21.611 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2024-05-27T00:17:21.612 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2024-05-27T00:17:21.612 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2024-05-27T00:17:21.612 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2024-05-27T00:17:21.612 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2024-05-27T00:17:21.612 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2024-05-27T00:17:21.612 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2024-05-27T00:17:21.612 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2024-05-27T00:17:21.612 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2024-05-27T00:17:21.613 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2024-05-27T00:17:21.613 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2024-05-27T00:17:21.613 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2024-05-27T00:17:21.613 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2024-05-27T00:17:21.613 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2024-05-27T00:17:21.613 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2024-05-27T00:17:21.613 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2024-05-27T00:17:21.613 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2024-05-27T00:17:21.614 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2024-05-27T00:17:21.614 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2024-05-27T00:17:21.614 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2024-05-27T00:17:21.614 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2024-05-27T00:17:21.614 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2024-05-27T00:17:21.614 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2024-05-27T00:17:21.614 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2024-05-27T00:17:21.614 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2024-05-27T00:17:21.615 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2024-05-27T00:17:21.615 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2024-05-27T00:17:21.615 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2024-05-27T00:17:21.615 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2024-05-27T00:17:21.615 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2024-05-27T00:17:21.615 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2024-05-27T00:17:21.615 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2024-05-27T00:17:21.616 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2024-05-27T00:17:21.616 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2024-05-27T00:17:21.616 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2024-05-27T00:17:21.616 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2024-05-27T00:17:21.616 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2024-05-27T00:17:21.616 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2024-05-27T00:17:21.617 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2024-05-27T00:17:21.617 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2024-05-27T00:17:21.617 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2024-05-27T00:17:21.617 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2024-05-27T00:17:21.617 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2024-05-27T00:17:21.617 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2024-05-27T00:17:21.618 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2024-05-27T00:17:21.619 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2024-05-27T00:17:21.752 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2024-05-27T00:17:21.752 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2024-05-27T00:17:21.752 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2024-05-27T00:17:21.752 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2024-05-27T00:17:21.753 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2024-05-27T00:17:21.754 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2024-05-27T00:17:21.755 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2024-05-27T00:17:21.756 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2024-05-27T00:17:21.757 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2024-05-27T00:17:21.757 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2024-05-27T00:17:21.757 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2024-05-27T00:17:21.757 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2024-05-27T00:17:21.757 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2024-05-27T00:17:21.757 INFO:tasks.workunit.client.0.smithi098.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2024-05-27T00:17:21.757 INFO:tasks.workunit.client.0.smithi098.stdout:Take third snapshot .snap/test3 2024-05-27T00:17:22.004 INFO:tasks.workunit.client.0.smithi098.stdout:Delete the snapshots... 2024-05-27T00:17:25.490 INFO:tasks.workunit.client.0.smithi098.stdout:.//.snap/test1 2024-05-27T00:17:25.490 INFO:tasks.workunit.client.0.smithi098.stdout:.//.snap/test2 2024-05-27T00:17:25.490 INFO:tasks.workunit.client.0.smithi098.stdout:.//.snap/test3 2024-05-27T00:17:25.496 INFO:tasks.workunit.client.0.smithi098.stdout:./274/.snap/snap-subdir-test 2024-05-27T00:17:25.504 INFO:tasks.workunit.client.0.smithi098.stdout:./166/.snap/snap-subdir-test 2024-05-27T00:17:25.511 INFO:tasks.workunit.client.0.smithi098.stdout:./256/.snap/snap-subdir-test 2024-05-27T00:17:25.517 INFO:tasks.workunit.client.0.smithi098.stdout:./247/.snap/snap-subdir-test 2024-05-27T00:17:25.523 INFO:tasks.workunit.client.0.smithi098.stdout:./248/.snap/snap-subdir-test 2024-05-27T00:17:25.530 INFO:tasks.workunit.client.0.smithi098.stdout:./147/.snap/snap-subdir-test 2024-05-27T00:17:25.537 INFO:tasks.workunit.client.0.smithi098.stdout:./151/.snap/snap-subdir-test 2024-05-27T00:17:25.543 INFO:tasks.workunit.client.0.smithi098.stdout:./170/.snap/snap-subdir-test 2024-05-27T00:17:25.550 INFO:tasks.workunit.client.0.smithi098.stdout:./167/.snap/snap-subdir-test 2024-05-27T00:17:25.556 INFO:tasks.workunit.client.0.smithi098.stdout:./140/.snap/snap-subdir-test 2024-05-27T00:17:25.563 INFO:tasks.workunit.client.0.smithi098.stdout:./142/.snap/snap-subdir-test 2024-05-27T00:17:25.569 INFO:tasks.workunit.client.0.smithi098.stdout:./150/.snap/snap-subdir-test 2024-05-27T00:17:25.576 INFO:tasks.workunit.client.0.smithi098.stdout:./273/.snap/snap-subdir-test 2024-05-27T00:17:25.581 INFO:tasks.workunit.client.0.smithi098.stdout:./269/.snap/snap-subdir-test 2024-05-27T00:17:25.588 INFO:tasks.workunit.client.0.smithi098.stdout:./105/.snap/snap-subdir-test 2024-05-27T00:17:25.594 INFO:tasks.workunit.client.0.smithi098.stdout:./210/.snap/snap-subdir-test 2024-05-27T00:17:25.605 INFO:tasks.workunit.client.0.smithi098.stdout:./265/.snap/snap-subdir-test 2024-05-27T00:17:25.612 INFO:tasks.workunit.client.0.smithi098.stdout:./218/.snap/snap-subdir-test 2024-05-27T00:17:25.620 INFO:tasks.workunit.client.0.smithi098.stdout:./290/.snap/snap-subdir-test 2024-05-27T00:17:25.627 INFO:tasks.workunit.client.0.smithi098.stdout:./149/.snap/snap-subdir-test 2024-05-27T00:17:25.634 INFO:tasks.workunit.client.0.smithi098.stdout:./109/.snap/snap-subdir-test 2024-05-27T00:17:25.641 INFO:tasks.workunit.client.0.smithi098.stdout:./292/.snap/snap-subdir-test 2024-05-27T00:17:25.648 INFO:tasks.workunit.client.0.smithi098.stdout:./242/.snap/snap-subdir-test 2024-05-27T00:17:25.654 INFO:tasks.workunit.client.0.smithi098.stdout:./158/.snap/snap-subdir-test 2024-05-27T00:17:25.660 INFO:tasks.workunit.client.0.smithi098.stdout:./243/.snap/snap-subdir-test 2024-05-27T00:17:25.666 INFO:tasks.workunit.client.0.smithi098.stdout:./106/.snap/snap-subdir-test 2024-05-27T00:17:25.672 INFO:tasks.workunit.client.0.smithi098.stdout:./250/.snap/snap-subdir-test 2024-05-27T00:17:25.678 INFO:tasks.workunit.client.0.smithi098.stdout:./221/.snap/snap-subdir-test 2024-05-27T00:17:25.685 INFO:tasks.workunit.client.0.smithi098.stdout:./244/.snap/snap-subdir-test 2024-05-27T00:17:25.690 INFO:tasks.workunit.client.0.smithi098.stdout:./258/.snap/snap-subdir-test 2024-05-27T00:17:25.696 INFO:tasks.workunit.client.0.smithi098.stdout:./268/.snap/snap-subdir-test 2024-05-27T00:17:25.702 INFO:tasks.workunit.client.0.smithi098.stdout:./189/.snap/snap-subdir-test 2024-05-27T00:17:25.706 INFO:tasks.workunit.client.0.smithi098.stdout:./285/.snap/snap-subdir-test 2024-05-27T00:17:25.711 INFO:tasks.workunit.client.0.smithi098.stdout:./208/.snap/snap-subdir-test 2024-05-27T00:17:25.716 INFO:tasks.workunit.client.0.smithi098.stdout:./288/.snap/snap-subdir-test 2024-05-27T00:17:25.721 INFO:tasks.workunit.client.0.smithi098.stdout:./209/.snap/snap-subdir-test 2024-05-27T00:17:25.727 INFO:tasks.workunit.client.0.smithi098.stdout:./253/.snap/snap-subdir-test 2024-05-27T00:17:25.733 INFO:tasks.workunit.client.0.smithi098.stdout:./249/.snap/snap-subdir-test 2024-05-27T00:17:25.738 INFO:tasks.workunit.client.0.smithi098.stdout:./133/.snap/snap-subdir-test 2024-05-27T00:17:25.742 INFO:tasks.workunit.client.0.smithi098.stdout:./291/.snap/snap-subdir-test 2024-05-27T00:17:25.748 INFO:tasks.workunit.client.0.smithi098.stdout:./226/.snap/snap-subdir-test 2024-05-27T00:17:25.753 INFO:tasks.workunit.client.0.smithi098.stdout:./173/.snap/snap-subdir-test 2024-05-27T00:17:25.757 INFO:tasks.workunit.client.0.smithi098.stdout:./174/.snap/snap-subdir-test 2024-05-27T00:17:25.762 INFO:tasks.workunit.client.0.smithi098.stdout:./299/.snap/snap-subdir-test 2024-05-27T00:17:25.768 INFO:tasks.workunit.client.0.smithi098.stdout:./262/.snap/snap-subdir-test 2024-05-27T00:17:25.772 INFO:tasks.workunit.client.0.smithi098.stdout:./231/.snap/snap-subdir-test 2024-05-27T00:17:25.776 INFO:tasks.workunit.client.0.smithi098.stdout:./214/.snap/snap-subdir-test 2024-05-27T00:17:25.781 INFO:tasks.workunit.client.0.smithi098.stdout:./281/.snap/snap-subdir-test 2024-05-27T00:17:25.786 INFO:tasks.workunit.client.0.smithi098.stdout:./289/.snap/snap-subdir-test 2024-05-27T00:17:25.791 INFO:tasks.workunit.client.0.smithi098.stdout:./283/.snap/snap-subdir-test 2024-05-27T00:17:25.795 INFO:tasks.workunit.client.0.smithi098.stdout:./232/.snap/snap-subdir-test 2024-05-27T00:17:25.800 INFO:tasks.workunit.client.0.smithi098.stdout:./298/.snap/snap-subdir-test 2024-05-27T00:17:25.805 INFO:tasks.workunit.client.0.smithi098.stdout:./233/.snap/snap-subdir-test 2024-05-27T00:17:25.810 INFO:tasks.workunit.client.0.smithi098.stdout:./155/.snap/snap-subdir-test 2024-05-27T00:17:25.814 INFO:tasks.workunit.client.0.smithi098.stdout:./114/.snap/snap-subdir-test 2024-05-27T00:17:25.819 INFO:tasks.workunit.client.0.smithi098.stdout:./159/.snap/snap-subdir-test 2024-05-27T00:17:25.823 INFO:tasks.workunit.client.0.smithi098.stdout:./187/.snap/snap-subdir-test 2024-05-27T00:17:25.829 INFO:tasks.workunit.client.0.smithi098.stdout:./185/.snap/snap-subdir-test 2024-05-27T00:17:25.833 INFO:tasks.workunit.client.0.smithi098.stdout:./293/.snap/snap-subdir-test 2024-05-27T00:17:25.840 INFO:tasks.workunit.client.0.smithi098.stdout:./228/.snap/snap-subdir-test 2024-05-27T00:17:25.845 INFO:tasks.workunit.client.0.smithi098.stdout:./120/.snap/snap-subdir-test 2024-05-27T00:17:25.851 INFO:tasks.workunit.client.0.smithi098.stdout:./110/.snap/snap-subdir-test 2024-05-27T00:17:25.858 INFO:tasks.workunit.client.0.smithi098.stdout:./267/.snap/snap-subdir-test 2024-05-27T00:17:25.863 INFO:tasks.workunit.client.0.smithi098.stdout:./202/.snap/snap-subdir-test 2024-05-27T00:17:25.870 INFO:tasks.workunit.client.0.smithi098.stdout:./137/.snap/snap-subdir-test 2024-05-27T00:17:25.876 INFO:tasks.workunit.client.0.smithi098.stdout:./246/.snap/snap-subdir-test 2024-05-27T00:17:25.882 INFO:tasks.workunit.client.0.smithi098.stdout:./219/.snap/snap-subdir-test 2024-05-27T00:17:25.888 INFO:tasks.workunit.client.0.smithi098.stdout:./198/.snap/snap-subdir-test 2024-05-27T00:17:25.894 INFO:tasks.workunit.client.0.smithi098.stdout:./136/.snap/snap-subdir-test 2024-05-27T00:17:25.900 INFO:tasks.workunit.client.0.smithi098.stdout:./270/.snap/snap-subdir-test 2024-05-27T00:17:25.906 INFO:tasks.workunit.client.0.smithi098.stdout:./125/.snap/snap-subdir-test 2024-05-27T00:17:25.912 INFO:tasks.workunit.client.0.smithi098.stdout:./194/.snap/snap-subdir-test 2024-05-27T00:17:25.918 INFO:tasks.workunit.client.0.smithi098.stdout:./119/.snap/snap-subdir-test 2024-05-27T00:17:25.924 INFO:tasks.workunit.client.0.smithi098.stdout:./297/.snap/snap-subdir-test 2024-05-27T00:17:25.928 INFO:tasks.workunit.client.0.smithi098.stdout:./230/.snap/snap-subdir-test 2024-05-27T00:17:25.933 INFO:tasks.workunit.client.0.smithi098.stdout:./260/.snap/snap-subdir-test 2024-05-27T00:17:25.937 INFO:tasks.workunit.client.0.smithi098.stdout:./157/.snap/snap-subdir-test 2024-05-27T00:17:25.942 INFO:tasks.workunit.client.0.smithi098.stdout:./172/.snap/snap-subdir-test 2024-05-27T00:17:25.946 INFO:tasks.workunit.client.0.smithi098.stdout:./277/.snap/snap-subdir-test 2024-05-27T00:17:25.951 INFO:tasks.workunit.client.0.smithi098.stdout:./186/.snap/snap-subdir-test 2024-05-27T00:17:25.957 INFO:tasks.workunit.client.0.smithi098.stdout:./237/.snap/snap-subdir-test 2024-05-27T00:17:25.963 INFO:tasks.workunit.client.0.smithi098.stdout:./190/.snap/snap-subdir-test 2024-05-27T00:17:25.969 INFO:tasks.workunit.client.0.smithi098.stdout:./213/.snap/snap-subdir-test 2024-05-27T00:17:25.975 INFO:tasks.workunit.client.0.smithi098.stdout:./104/.snap/snap-subdir-test 2024-05-27T00:17:25.979 INFO:tasks.workunit.client.0.smithi098.stdout:./164/.snap/snap-subdir-test 2024-05-27T00:17:25.984 INFO:tasks.workunit.client.0.smithi098.stdout:./112/.snap/snap-subdir-test 2024-05-27T00:17:25.990 INFO:tasks.workunit.client.0.smithi098.stdout:./295/.snap/snap-subdir-test 2024-05-27T00:17:25.993 INFO:tasks.workunit.client.0.smithi098.stdout:./272/.snap/snap-subdir-test 2024-05-27T00:17:25.998 INFO:tasks.workunit.client.0.smithi098.stdout:./100/.snap/snap-subdir-test 2024-05-27T00:17:26.003 INFO:tasks.workunit.client.0.smithi098.stdout:./134/.snap/snap-subdir-test 2024-05-27T00:17:26.009 INFO:tasks.workunit.client.0.smithi098.stdout:./200/.snap/snap-subdir-test 2024-05-27T00:17:26.015 INFO:tasks.workunit.client.0.smithi098.stdout:./115/.snap/snap-subdir-test 2024-05-27T00:17:26.021 INFO:tasks.workunit.client.0.smithi098.stdout:./284/.snap/snap-subdir-test 2024-05-27T00:17:26.027 INFO:tasks.workunit.client.0.smithi098.stdout:./280/.snap/snap-subdir-test 2024-05-27T00:17:26.032 INFO:tasks.workunit.client.0.smithi098.stdout:./116/.snap/snap-subdir-test 2024-05-27T00:17:26.038 INFO:tasks.workunit.client.0.smithi098.stdout:./245/.snap/snap-subdir-test 2024-05-27T00:17:26.044 INFO:tasks.workunit.client.0.smithi098.stdout:./195/.snap/snap-subdir-test 2024-05-27T00:17:26.050 INFO:tasks.workunit.client.0.smithi098.stdout:./138/.snap/snap-subdir-test 2024-05-27T00:17:26.055 INFO:tasks.workunit.client.0.smithi098.stdout:./135/.snap/snap-subdir-test 2024-05-27T00:17:26.060 INFO:tasks.workunit.client.0.smithi098.stdout:./117/.snap/snap-subdir-test 2024-05-27T00:17:26.065 INFO:tasks.workunit.client.0.smithi098.stdout:./241/.snap/snap-subdir-test 2024-05-27T00:17:26.070 INFO:tasks.workunit.client.0.smithi098.stdout:./143/.snap/snap-subdir-test 2024-05-27T00:17:26.074 INFO:tasks.workunit.client.0.smithi098.stdout:./180/.snap/snap-subdir-test 2024-05-27T00:17:26.079 INFO:tasks.workunit.client.0.smithi098.stdout:./126/.snap/snap-subdir-test 2024-05-27T00:17:26.084 INFO:tasks.workunit.client.0.smithi098.stdout:./161/.snap/snap-subdir-test 2024-05-27T00:17:26.089 INFO:tasks.workunit.client.0.smithi098.stdout:./204/.snap/snap-subdir-test 2024-05-27T00:17:26.094 INFO:tasks.workunit.client.0.smithi098.stdout:./176/.snap/snap-subdir-test 2024-05-27T00:17:26.099 INFO:tasks.workunit.client.0.smithi098.stdout:./255/.snap/snap-subdir-test 2024-05-27T00:17:26.104 INFO:tasks.workunit.client.0.smithi098.stdout:./215/.snap/snap-subdir-test 2024-05-27T00:17:26.109 INFO:tasks.workunit.client.0.smithi098.stdout:./111/.snap/snap-subdir-test 2024-05-27T00:17:26.113 INFO:tasks.workunit.client.0.smithi098.stdout:./238/.snap/snap-subdir-test 2024-05-27T00:17:26.118 INFO:tasks.workunit.client.0.smithi098.stdout:./191/.snap/snap-subdir-test 2024-05-27T00:17:26.123 INFO:tasks.workunit.client.0.smithi098.stdout:./184/.snap/snap-subdir-test 2024-05-27T00:17:26.127 INFO:tasks.workunit.client.0.smithi098.stdout:./192/.snap/snap-subdir-test 2024-05-27T00:17:26.133 INFO:tasks.workunit.client.0.smithi098.stdout:./103/.snap/snap-subdir-test 2024-05-27T00:17:26.137 INFO:tasks.workunit.client.0.smithi098.stdout:./128/.snap/snap-subdir-test 2024-05-27T00:17:26.142 INFO:tasks.workunit.client.0.smithi098.stdout:./229/.snap/snap-subdir-test 2024-05-27T00:17:26.147 INFO:tasks.workunit.client.0.smithi098.stdout:./144/.snap/snap-subdir-test 2024-05-27T00:17:26.152 INFO:tasks.workunit.client.0.smithi098.stdout:./146/.snap/snap-subdir-test 2024-05-27T00:17:26.156 INFO:tasks.workunit.client.0.smithi098.stdout:./227/.snap/snap-subdir-test 2024-05-27T00:17:26.161 INFO:tasks.workunit.client.0.smithi098.stdout:./278/.snap/snap-subdir-test 2024-05-27T00:17:26.166 INFO:tasks.workunit.client.0.smithi098.stdout:./141/.snap/snap-subdir-test 2024-05-27T00:17:26.171 INFO:tasks.workunit.client.0.smithi098.stdout:./234/.snap/snap-subdir-test 2024-05-27T00:17:26.177 INFO:tasks.workunit.client.0.smithi098.stdout:./225/.snap/snap-subdir-test 2024-05-27T00:17:26.182 INFO:tasks.workunit.client.0.smithi098.stdout:./182/.snap/snap-subdir-test 2024-05-27T00:17:26.187 INFO:tasks.workunit.client.0.smithi098.stdout:./107/.snap/snap-subdir-test 2024-05-27T00:17:26.193 INFO:tasks.workunit.client.0.smithi098.stdout:./178/.snap/snap-subdir-test 2024-05-27T00:17:26.197 INFO:tasks.workunit.client.0.smithi098.stdout:./266/.snap/snap-subdir-test 2024-05-27T00:17:26.202 INFO:tasks.workunit.client.0.smithi098.stdout:./240/.snap/snap-subdir-test 2024-05-27T00:17:26.212 INFO:tasks.workunit.client.0.smithi098.stdout:./156/.snap/snap-subdir-test 2024-05-27T00:17:26.217 INFO:tasks.workunit.client.0.smithi098.stdout:./131/.snap/snap-subdir-test 2024-05-27T00:17:26.221 INFO:tasks.workunit.client.0.smithi098.stdout:./123/.snap/snap-subdir-test 2024-05-27T00:17:26.226 INFO:tasks.workunit.client.0.smithi098.stdout:./160/.snap/snap-subdir-test 2024-05-27T00:17:26.231 INFO:tasks.workunit.client.0.smithi098.stdout:./165/.snap/snap-subdir-test 2024-05-27T00:17:26.236 INFO:tasks.workunit.client.0.smithi098.stdout:./101/.snap/snap-subdir-test 2024-05-27T00:17:26.242 INFO:tasks.workunit.client.0.smithi098.stdout:./148/.snap/snap-subdir-test 2024-05-27T00:17:26.247 INFO:tasks.workunit.client.0.smithi098.stdout:./168/.snap/snap-subdir-test 2024-05-27T00:17:26.251 INFO:tasks.workunit.client.0.smithi098.stdout:./179/.snap/snap-subdir-test 2024-05-27T00:17:26.256 INFO:tasks.workunit.client.0.smithi098.stdout:./257/.snap/snap-subdir-test 2024-05-27T00:17:26.261 INFO:tasks.workunit.client.0.smithi098.stdout:./252/.snap/snap-subdir-test 2024-05-27T00:17:26.267 INFO:tasks.workunit.client.0.smithi098.stdout:./132/.snap/snap-subdir-test 2024-05-27T00:17:26.272 INFO:tasks.workunit.client.0.smithi098.stdout:./236/.snap/snap-subdir-test 2024-05-27T00:17:26.279 INFO:tasks.workunit.client.0.smithi098.stdout:./211/.snap/snap-subdir-test 2024-05-27T00:17:26.284 INFO:tasks.workunit.client.0.smithi098.stdout:./199/.snap/snap-subdir-test 2024-05-27T00:17:26.291 INFO:tasks.workunit.client.0.smithi098.stdout:./205/.snap/snap-subdir-test 2024-05-27T00:17:26.298 INFO:tasks.workunit.client.0.smithi098.stdout:./163/.snap/snap-subdir-test 2024-05-27T00:17:26.303 INFO:tasks.workunit.client.0.smithi098.stdout:./261/.snap/snap-subdir-test 2024-05-27T00:17:26.309 INFO:tasks.workunit.client.0.smithi098.stdout:./201/.snap/snap-subdir-test 2024-05-27T00:17:26.315 INFO:tasks.workunit.client.0.smithi098.stdout:./282/.snap/snap-subdir-test 2024-05-27T00:17:26.321 INFO:tasks.workunit.client.0.smithi098.stdout:./296/.snap/snap-subdir-test 2024-05-27T00:17:26.328 INFO:tasks.workunit.client.0.smithi098.stdout:./217/.snap/snap-subdir-test 2024-05-27T00:17:26.335 INFO:tasks.workunit.client.0.smithi098.stdout:./153/.snap/snap-subdir-test 2024-05-27T00:17:26.341 INFO:tasks.workunit.client.0.smithi098.stdout:./206/.snap/snap-subdir-test 2024-05-27T00:17:26.347 INFO:tasks.workunit.client.0.smithi098.stdout:./122/.snap/snap-subdir-test 2024-05-27T00:17:26.353 INFO:tasks.workunit.client.0.smithi098.stdout:./162/.snap/snap-subdir-test 2024-05-27T00:17:26.359 INFO:tasks.workunit.client.0.smithi098.stdout:./197/.snap/snap-subdir-test 2024-05-27T00:17:26.364 INFO:tasks.workunit.client.0.smithi098.stdout:./216/.snap/snap-subdir-test 2024-05-27T00:17:26.815 INFO:tasks.workunit.client.0.smithi098.stdout:./254/.snap/snap-subdir-test 2024-05-27T00:17:26.820 INFO:tasks.workunit.client.0.smithi098.stdout:./113/.snap/snap-subdir-test 2024-05-27T00:17:26.825 INFO:tasks.workunit.client.0.smithi098.stdout:./152/.snap/snap-subdir-test 2024-05-27T00:17:26.830 INFO:tasks.workunit.client.0.smithi098.stdout:./121/.snap/snap-subdir-test 2024-05-27T00:17:26.836 INFO:tasks.workunit.client.0.smithi098.stdout:./118/.snap/snap-subdir-test 2024-05-27T00:17:26.841 INFO:tasks.workunit.client.0.smithi098.stdout:./196/.snap/snap-subdir-test 2024-05-27T00:17:26.846 INFO:tasks.workunit.client.0.smithi098.stdout:./124/.snap/snap-subdir-test 2024-05-27T00:17:26.852 INFO:tasks.workunit.client.0.smithi098.stdout:./239/.snap/snap-subdir-test 2024-05-27T00:17:26.857 INFO:tasks.workunit.client.0.smithi098.stdout:./188/.snap/snap-subdir-test 2024-05-27T00:17:26.861 INFO:tasks.workunit.client.0.smithi098.stdout:./287/.snap/snap-subdir-test 2024-05-27T00:17:26.867 INFO:tasks.workunit.client.0.smithi098.stdout:./275/.snap/snap-subdir-test 2024-05-27T00:17:26.871 INFO:tasks.workunit.client.0.smithi098.stdout:./259/.snap/snap-subdir-test 2024-05-27T00:17:26.877 INFO:tasks.workunit.client.0.smithi098.stdout:./224/.snap/snap-subdir-test 2024-05-27T00:17:26.882 INFO:tasks.workunit.client.0.smithi098.stdout:./264/.snap/snap-subdir-test 2024-05-27T00:17:26.888 INFO:tasks.workunit.client.0.smithi098.stdout:./145/.snap/snap-subdir-test 2024-05-27T00:17:26.893 INFO:tasks.workunit.client.0.smithi098.stdout:./279/.snap/snap-subdir-test 2024-05-27T00:17:26.898 INFO:tasks.workunit.client.0.smithi098.stdout:./263/.snap/snap-subdir-test 2024-05-27T00:17:26.903 INFO:tasks.workunit.client.0.smithi098.stdout:./130/.snap/snap-subdir-test 2024-05-27T00:17:26.908 INFO:tasks.workunit.client.0.smithi098.stdout:./193/.snap/snap-subdir-test 2024-05-27T00:17:26.914 INFO:tasks.workunit.client.0.smithi098.stdout:./223/.snap/snap-subdir-test 2024-05-27T00:17:26.919 INFO:tasks.workunit.client.0.smithi098.stdout:./207/.snap/snap-subdir-test 2024-05-27T00:17:26.924 INFO:tasks.workunit.client.0.smithi098.stdout:./276/.snap/snap-subdir-test 2024-05-27T00:17:26.929 INFO:tasks.workunit.client.0.smithi098.stdout:./271/.snap/snap-subdir-test 2024-05-27T00:17:26.935 INFO:tasks.workunit.client.0.smithi098.stdout:./251/.snap/snap-subdir-test 2024-05-27T00:17:26.940 INFO:tasks.workunit.client.0.smithi098.stdout:./175/.snap/snap-subdir-test 2024-05-27T00:17:26.945 INFO:tasks.workunit.client.0.smithi098.stdout:./222/.snap/snap-subdir-test 2024-05-27T00:17:26.951 INFO:tasks.workunit.client.0.smithi098.stdout:./220/.snap/snap-subdir-test 2024-05-27T00:17:26.956 INFO:tasks.workunit.client.0.smithi098.stdout:./203/.snap/snap-subdir-test 2024-05-27T00:17:26.961 INFO:tasks.workunit.client.0.smithi098.stdout:./181/.snap/snap-subdir-test 2024-05-27T00:17:26.967 INFO:tasks.workunit.client.0.smithi098.stdout:./286/.snap/snap-subdir-test 2024-05-27T00:17:26.972 INFO:tasks.workunit.client.0.smithi098.stdout:./127/.snap/snap-subdir-test 2024-05-27T00:17:26.977 INFO:tasks.workunit.client.0.smithi098.stdout:./108/.snap/snap-subdir-test 2024-05-27T00:17:26.982 INFO:tasks.workunit.client.0.smithi098.stdout:./183/.snap/snap-subdir-test 2024-05-27T00:17:26.988 INFO:tasks.workunit.client.0.smithi098.stdout:./294/.snap/snap-subdir-test 2024-05-27T00:17:26.993 INFO:tasks.workunit.client.0.smithi098.stdout:./235/.snap/snap-subdir-test 2024-05-27T00:17:26.998 INFO:tasks.workunit.client.0.smithi098.stdout:./177/.snap/snap-subdir-test 2024-05-27T00:17:27.003 INFO:tasks.workunit.client.0.smithi098.stdout:./169/.snap/snap-subdir-test 2024-05-27T00:17:27.009 INFO:tasks.workunit.client.0.smithi098.stdout:./212/.snap/snap-subdir-test 2024-05-27T00:17:27.014 INFO:tasks.workunit.client.0.smithi098.stdout:./154/.snap/snap-subdir-test 2024-05-27T00:17:27.019 INFO:tasks.workunit.client.0.smithi098.stdout:./171/.snap/snap-subdir-test 2024-05-27T00:17:27.024 INFO:tasks.workunit.client.0.smithi098.stdout:./139/.snap/snap-subdir-test 2024-05-27T00:17:27.029 INFO:tasks.workunit.client.0.smithi098.stdout:./102/.snap/snap-subdir-test 2024-05-27T00:17:27.034 INFO:tasks.workunit.client.0.smithi098.stdout:./129/.snap/snap-subdir-test 2024-05-27T00:17:28.839 INFO:tasks.workunit.client.0.smithi098.stdout:Delete all the files and directories ... 2024-05-27T00:17:29.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/19' 2024-05-27T00:17:29.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/15' 2024-05-27T00:17:29.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/31' 2024-05-27T00:17:29.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/36' 2024-05-27T00:17:29.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/20' 2024-05-27T00:17:29.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/12' 2024-05-27T00:17:29.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/39' 2024-05-27T00:17:29.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/10' 2024-05-27T00:17:29.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/34' 2024-05-27T00:17:29.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/38' 2024-05-27T00:17:29.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/11' 2024-05-27T00:17:29.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/16' 2024-05-27T00:17:29.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/32' 2024-05-27T00:17:29.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/30' 2024-05-27T00:17:29.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/14' 2024-05-27T00:17:29.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/18' 2024-05-27T00:17:29.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/13' 2024-05-27T00:17:29.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/33' 2024-05-27T00:17:29.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/17' 2024-05-27T00:17:29.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/35' 2024-05-27T00:17:29.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './100/37' 2024-05-27T00:17:29.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './100' 2024-05-27T00:17:29.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/19' 2024-05-27T00:17:29.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/15' 2024-05-27T00:17:29.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/31' 2024-05-27T00:17:29.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/36' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/20' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/12' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/39' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/10' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/34' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/38' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/11' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/16' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/32' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/30' 2024-05-27T00:17:29.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/14' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/18' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/13' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/33' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/17' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/35' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './101/37' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './101' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/19' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/15' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/31' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/36' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/20' 2024-05-27T00:17:29.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/12' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/39' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/10' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/34' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/38' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/11' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/16' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/32' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/30' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/14' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/18' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/13' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/33' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/17' 2024-05-27T00:17:29.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/35' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './102/37' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './102' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/19' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/15' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/31' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/36' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/20' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/12' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/39' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/10' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/34' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/38' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/11' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/16' 2024-05-27T00:17:30.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/32' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/30' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/14' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/18' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/13' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/33' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/17' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/35' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './103/37' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './103' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/19' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/15' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/31' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/36' 2024-05-27T00:17:30.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/20' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/12' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/39' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/10' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/34' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/38' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/11' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/16' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/32' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/30' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/14' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/18' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/13' 2024-05-27T00:17:30.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/33' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/17' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/35' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './104/37' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './104' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/19' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/15' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/31' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/36' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/20' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/12' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/39' 2024-05-27T00:17:30.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/10' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/34' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/38' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/11' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/16' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/32' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/30' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/14' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/18' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/13' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/33' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/17' 2024-05-27T00:17:30.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/35' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './105/37' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './105' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/19' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/15' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/31' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/36' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/20' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/12' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/39' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/10' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/34' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/38' 2024-05-27T00:17:30.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/11' 2024-05-27T00:17:30.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/16' 2024-05-27T00:17:30.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/32' 2024-05-27T00:17:30.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/30' 2024-05-27T00:17:30.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/14' 2024-05-27T00:17:30.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/18' 2024-05-27T00:17:30.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/13' 2024-05-27T00:17:30.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/33' 2024-05-27T00:17:30.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/17' 2024-05-27T00:17:30.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/35' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './106/37' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './106' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/19' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/15' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/31' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/36' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/20' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/12' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/39' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/10' 2024-05-27T00:17:30.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/34' 2024-05-27T00:17:30.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/38' 2024-05-27T00:17:30.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/11' 2024-05-27T00:17:30.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/16' 2024-05-27T00:17:30.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/32' 2024-05-27T00:17:30.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/30' 2024-05-27T00:17:30.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/14' 2024-05-27T00:17:30.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/18' 2024-05-27T00:17:30.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/13' 2024-05-27T00:17:30.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/33' 2024-05-27T00:17:30.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/17' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/35' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './107/37' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './107' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/19' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/15' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/31' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/36' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/20' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/12' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/39' 2024-05-27T00:17:30.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/10' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/34' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/38' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/11' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/16' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/32' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/30' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/14' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/18' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/13' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/33' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/17' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/35' 2024-05-27T00:17:30.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './108/37' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './108' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/19' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/15' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/31' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/36' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/20' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/12' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/39' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/10' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/34' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/38' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/11' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/16' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/32' 2024-05-27T00:17:30.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/30' 2024-05-27T00:17:30.596 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/14' 2024-05-27T00:17:30.596 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/18' 2024-05-27T00:17:30.596 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/13' 2024-05-27T00:17:30.596 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/33' 2024-05-27T00:17:30.596 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/17' 2024-05-27T00:17:30.596 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/35' 2024-05-27T00:17:30.596 INFO:tasks.workunit.client.0.smithi098.stdout:removed './109/37' 2024-05-27T00:17:30.596 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './109' 2024-05-27T00:17:30.596 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/19' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/15' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/31' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/36' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/20' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/12' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/39' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/10' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/34' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/38' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/11' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/16' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/32' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/30' 2024-05-27T00:17:30.597 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/14' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/18' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/13' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/33' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/17' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/35' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './110/37' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './110' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/19' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/15' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/31' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/36' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/20' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/12' 2024-05-27T00:17:30.598 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/39' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/10' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/34' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/38' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/11' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/16' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/32' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/30' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/14' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/18' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/13' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/33' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/17' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/35' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed './111/37' 2024-05-27T00:17:30.599 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './111' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/19' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/15' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/31' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/36' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/20' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/12' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/39' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/10' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/34' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/38' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/11' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/16' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/32' 2024-05-27T00:17:30.600 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/30' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/14' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/18' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/13' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/33' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/17' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/35' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './112/37' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './112' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/19' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/15' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/31' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/36' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/20' 2024-05-27T00:17:30.601 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/12' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/39' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/10' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/34' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/38' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/11' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/16' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/32' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/30' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/14' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/18' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/13' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/33' 2024-05-27T00:17:30.602 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/17' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/35' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './113/37' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './113' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/19' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/15' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/31' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/36' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/20' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/12' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/39' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/10' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/34' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/38' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/11' 2024-05-27T00:17:30.603 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/16' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/32' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/30' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/14' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/18' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/13' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/33' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/17' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/35' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './114/37' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './114' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/19' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/15' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/31' 2024-05-27T00:17:30.604 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/36' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/20' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/12' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/39' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/10' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/34' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/38' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/11' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/16' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/32' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/30' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/14' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/18' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/13' 2024-05-27T00:17:30.605 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/33' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/17' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/35' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed './115/37' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './115' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/19' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/15' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/31' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/36' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/20' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/12' 2024-05-27T00:17:30.606 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/39' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/10' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/34' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/38' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/11' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/16' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/32' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/30' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/14' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/18' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/13' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/33' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/17' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/35' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed './116/37' 2024-05-27T00:17:30.607 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './116' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/19' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/15' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/31' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/36' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/20' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/12' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/39' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/10' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/34' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/38' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/11' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/16' 2024-05-27T00:17:30.608 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/32' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/30' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/14' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/18' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/13' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/33' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/17' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/35' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './117/37' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './117' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/19' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/15' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/31' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/36' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/20' 2024-05-27T00:17:30.609 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/12' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/39' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/10' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/34' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/38' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/11' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/16' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/32' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/30' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/14' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/18' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/13' 2024-05-27T00:17:30.610 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/33' 2024-05-27T00:17:30.611 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/17' 2024-05-27T00:17:30.611 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/35' 2024-05-27T00:17:30.611 INFO:tasks.workunit.client.0.smithi098.stdout:removed './118/37' 2024-05-27T00:17:30.611 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './118' 2024-05-27T00:17:30.611 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/19' 2024-05-27T00:17:30.611 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/15' 2024-05-27T00:17:30.611 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/31' 2024-05-27T00:17:30.611 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/36' 2024-05-27T00:17:30.611 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/20' 2024-05-27T00:17:30.611 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/12' 2024-05-27T00:17:31.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/39' 2024-05-27T00:17:31.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/10' 2024-05-27T00:17:31.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/34' 2024-05-27T00:17:31.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/38' 2024-05-27T00:17:31.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/11' 2024-05-27T00:17:31.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/16' 2024-05-27T00:17:31.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/32' 2024-05-27T00:17:31.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/30' 2024-05-27T00:17:31.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/14' 2024-05-27T00:17:31.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/18' 2024-05-27T00:17:31.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/13' 2024-05-27T00:17:31.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/33' 2024-05-27T00:17:31.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/17' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/35' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './119/37' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './119' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/19' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/15' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/31' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/36' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/20' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/12' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/39' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/10' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/34' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/38' 2024-05-27T00:17:31.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/11' 2024-05-27T00:17:31.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/16' 2024-05-27T00:17:31.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/32' 2024-05-27T00:17:31.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/30' 2024-05-27T00:17:31.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/14' 2024-05-27T00:17:31.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/18' 2024-05-27T00:17:31.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/13' 2024-05-27T00:17:31.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/33' 2024-05-27T00:17:31.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/17' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/35' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './120/37' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './120' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/19' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/15' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/31' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/36' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/20' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/12' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/39' 2024-05-27T00:17:31.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/10' 2024-05-27T00:17:31.035 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/34' 2024-05-27T00:17:31.035 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/38' 2024-05-27T00:17:31.035 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/11' 2024-05-27T00:17:31.035 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/16' 2024-05-27T00:17:31.035 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/32' 2024-05-27T00:17:31.035 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/30' 2024-05-27T00:17:31.035 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/14' 2024-05-27T00:17:31.035 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/18' 2024-05-27T00:17:31.035 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/13' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/33' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/17' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/35' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './121/37' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './121' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/19' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/15' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/31' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/36' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/20' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/12' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/39' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/10' 2024-05-27T00:17:31.036 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/34' 2024-05-27T00:17:31.037 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/38' 2024-05-27T00:17:31.037 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/11' 2024-05-27T00:17:31.037 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/16' 2024-05-27T00:17:31.037 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/32' 2024-05-27T00:17:31.037 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/30' 2024-05-27T00:17:31.037 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/14' 2024-05-27T00:17:31.037 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/18' 2024-05-27T00:17:31.037 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/13' 2024-05-27T00:17:31.037 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/33' 2024-05-27T00:17:31.037 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/17' 2024-05-27T00:17:31.038 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/35' 2024-05-27T00:17:31.038 INFO:tasks.workunit.client.0.smithi098.stdout:removed './122/37' 2024-05-27T00:17:31.038 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './122' 2024-05-27T00:17:31.038 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/19' 2024-05-27T00:17:31.038 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/15' 2024-05-27T00:17:31.038 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/31' 2024-05-27T00:17:31.038 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/36' 2024-05-27T00:17:31.038 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/20' 2024-05-27T00:17:31.038 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/12' 2024-05-27T00:17:31.039 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/39' 2024-05-27T00:17:31.039 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/10' 2024-05-27T00:17:31.039 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/34' 2024-05-27T00:17:31.039 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/38' 2024-05-27T00:17:31.039 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/11' 2024-05-27T00:17:31.039 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/16' 2024-05-27T00:17:31.039 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/32' 2024-05-27T00:17:31.040 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/30' 2024-05-27T00:17:31.040 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/14' 2024-05-27T00:17:31.040 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/18' 2024-05-27T00:17:31.040 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/13' 2024-05-27T00:17:31.040 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/33' 2024-05-27T00:17:31.040 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/17' 2024-05-27T00:17:31.040 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/35' 2024-05-27T00:17:31.041 INFO:tasks.workunit.client.0.smithi098.stdout:removed './123/37' 2024-05-27T00:17:31.041 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './123' 2024-05-27T00:17:31.041 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/19' 2024-05-27T00:17:31.041 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/15' 2024-05-27T00:17:31.041 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/31' 2024-05-27T00:17:31.041 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/36' 2024-05-27T00:17:31.041 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/20' 2024-05-27T00:17:31.041 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/12' 2024-05-27T00:17:31.041 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/39' 2024-05-27T00:17:31.042 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/10' 2024-05-27T00:17:31.042 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/34' 2024-05-27T00:17:31.042 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/38' 2024-05-27T00:17:31.042 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/11' 2024-05-27T00:17:31.043 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/16' 2024-05-27T00:17:31.043 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/32' 2024-05-27T00:17:31.043 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/30' 2024-05-27T00:17:31.043 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/14' 2024-05-27T00:17:31.043 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/18' 2024-05-27T00:17:31.043 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/13' 2024-05-27T00:17:31.043 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/33' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/17' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/35' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed './124/37' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './124' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/19' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/15' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/31' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/36' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/20' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/12' 2024-05-27T00:17:31.044 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/39' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/10' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/34' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/38' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/11' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/16' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/32' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/30' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/14' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/18' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/13' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/33' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/17' 2024-05-27T00:17:31.045 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/35' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './125/37' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './125' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/19' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/15' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/31' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/36' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/20' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/12' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/39' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/10' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/34' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/38' 2024-05-27T00:17:31.046 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/11' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/16' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/32' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/30' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/14' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/18' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/13' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/33' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/17' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/35' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './126/37' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './126' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/19' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/15' 2024-05-27T00:17:31.047 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/31' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/36' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/20' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/12' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/39' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/10' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/34' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/38' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/11' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/16' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/32' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/30' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/14' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/18' 2024-05-27T00:17:31.048 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/13' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/33' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/17' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/35' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './127/37' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './127' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/19' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/15' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/31' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/36' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/20' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/12' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/39' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/10' 2024-05-27T00:17:31.049 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/34' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/38' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/11' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/16' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/32' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/30' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/14' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/18' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/13' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/33' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/17' 2024-05-27T00:17:31.050 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/35' 2024-05-27T00:17:32.061 INFO:tasks.workunit.client.0.smithi098.stdout:removed './128/37' 2024-05-27T00:17:32.061 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './128' 2024-05-27T00:17:32.061 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/19' 2024-05-27T00:17:32.062 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/15' 2024-05-27T00:17:32.062 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/31' 2024-05-27T00:17:32.062 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/36' 2024-05-27T00:17:32.062 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/20' 2024-05-27T00:17:32.062 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/12' 2024-05-27T00:17:32.063 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/39' 2024-05-27T00:17:32.063 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/10' 2024-05-27T00:17:32.063 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/34' 2024-05-27T00:17:32.063 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/38' 2024-05-27T00:17:32.063 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/11' 2024-05-27T00:17:32.063 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/16' 2024-05-27T00:17:32.064 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/32' 2024-05-27T00:17:32.064 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/30' 2024-05-27T00:17:32.064 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/14' 2024-05-27T00:17:32.064 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/18' 2024-05-27T00:17:32.064 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/13' 2024-05-27T00:17:32.065 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/33' 2024-05-27T00:17:32.065 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/17' 2024-05-27T00:17:32.065 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/35' 2024-05-27T00:17:32.065 INFO:tasks.workunit.client.0.smithi098.stdout:removed './129/37' 2024-05-27T00:17:32.065 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './129' 2024-05-27T00:17:32.065 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/19' 2024-05-27T00:17:32.066 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/15' 2024-05-27T00:17:32.066 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/31' 2024-05-27T00:17:32.066 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/36' 2024-05-27T00:17:32.066 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/20' 2024-05-27T00:17:32.066 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/12' 2024-05-27T00:17:32.066 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/39' 2024-05-27T00:17:32.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/10' 2024-05-27T00:17:32.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/34' 2024-05-27T00:17:32.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/38' 2024-05-27T00:17:32.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/11' 2024-05-27T00:17:32.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/16' 2024-05-27T00:17:32.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/32' 2024-05-27T00:17:32.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/30' 2024-05-27T00:17:32.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/14' 2024-05-27T00:17:32.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/18' 2024-05-27T00:17:32.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/13' 2024-05-27T00:17:32.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/33' 2024-05-27T00:17:32.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/17' 2024-05-27T00:17:32.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/35' 2024-05-27T00:17:32.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './130/37' 2024-05-27T00:17:32.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './130' 2024-05-27T00:17:32.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/19' 2024-05-27T00:17:32.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/15' 2024-05-27T00:17:32.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/31' 2024-05-27T00:17:32.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/36' 2024-05-27T00:17:32.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/20' 2024-05-27T00:17:32.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/12' 2024-05-27T00:17:32.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/39' 2024-05-27T00:17:32.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/10' 2024-05-27T00:17:32.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/34' 2024-05-27T00:17:32.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/38' 2024-05-27T00:17:32.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/11' 2024-05-27T00:17:32.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/16' 2024-05-27T00:17:32.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/32' 2024-05-27T00:17:32.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/30' 2024-05-27T00:17:32.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/14' 2024-05-27T00:17:32.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/18' 2024-05-27T00:17:32.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/13' 2024-05-27T00:17:32.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/33' 2024-05-27T00:17:32.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/17' 2024-05-27T00:17:32.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/35' 2024-05-27T00:17:32.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './131/37' 2024-05-27T00:17:32.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './131' 2024-05-27T00:17:32.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/19' 2024-05-27T00:17:32.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/15' 2024-05-27T00:17:32.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/31' 2024-05-27T00:17:32.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/36' 2024-05-27T00:17:32.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/20' 2024-05-27T00:17:32.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/12' 2024-05-27T00:17:32.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/39' 2024-05-27T00:17:32.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/10' 2024-05-27T00:17:32.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/34' 2024-05-27T00:17:32.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/38' 2024-05-27T00:17:32.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/11' 2024-05-27T00:17:32.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/16' 2024-05-27T00:17:32.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/32' 2024-05-27T00:17:32.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/30' 2024-05-27T00:17:32.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/14' 2024-05-27T00:17:32.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/18' 2024-05-27T00:17:32.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/13' 2024-05-27T00:17:32.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/33' 2024-05-27T00:17:32.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/17' 2024-05-27T00:17:32.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/35' 2024-05-27T00:17:32.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './132/37' 2024-05-27T00:17:32.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './132' 2024-05-27T00:17:32.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/19' 2024-05-27T00:17:32.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/15' 2024-05-27T00:17:32.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/31' 2024-05-27T00:17:32.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/36' 2024-05-27T00:17:32.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/20' 2024-05-27T00:17:32.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/12' 2024-05-27T00:17:32.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/39' 2024-05-27T00:17:32.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/10' 2024-05-27T00:17:32.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/34' 2024-05-27T00:17:32.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/38' 2024-05-27T00:17:32.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/11' 2024-05-27T00:17:32.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/16' 2024-05-27T00:17:32.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/32' 2024-05-27T00:17:32.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/30' 2024-05-27T00:17:32.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/14' 2024-05-27T00:17:32.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/18' 2024-05-27T00:17:32.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/13' 2024-05-27T00:17:32.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/33' 2024-05-27T00:17:32.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/17' 2024-05-27T00:17:32.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/35' 2024-05-27T00:17:32.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './133/37' 2024-05-27T00:17:32.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './133' 2024-05-27T00:17:32.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/19' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/15' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/31' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/36' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/20' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/12' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/39' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/10' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/34' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/38' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/11' 2024-05-27T00:17:32.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/16' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/32' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/30' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/14' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/18' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/13' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/33' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/17' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/35' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './134/37' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './134' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/19' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/15' 2024-05-27T00:17:32.082 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/31' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/36' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/20' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/12' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/39' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/10' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/34' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/38' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/11' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/16' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/32' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/30' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/14' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/18' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/13' 2024-05-27T00:17:32.083 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/33' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/17' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/35' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './135/37' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './135' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/19' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/15' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/31' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/36' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/20' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/12' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/39' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/10' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/34' 2024-05-27T00:17:32.084 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/38' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/11' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/16' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/32' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/30' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/14' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/18' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/13' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/33' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/17' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/35' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed './136/37' 2024-05-27T00:17:32.085 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './136' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/19' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/15' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/31' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/36' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/20' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/12' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/39' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/10' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/34' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/38' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/11' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/16' 2024-05-27T00:17:32.086 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/32' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/30' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/14' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/18' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/13' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/33' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/17' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/35' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './137/37' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './137' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/19' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/15' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/31' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/36' 2024-05-27T00:17:32.087 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/20' 2024-05-27T00:17:32.088 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/12' 2024-05-27T00:17:32.088 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/39' 2024-05-27T00:17:32.088 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/10' 2024-05-27T00:17:32.088 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/34' 2024-05-27T00:17:32.088 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/38' 2024-05-27T00:17:32.088 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/11' 2024-05-27T00:17:32.088 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/16' 2024-05-27T00:17:33.148 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/32' 2024-05-27T00:17:33.149 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/30' 2024-05-27T00:17:33.149 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/14' 2024-05-27T00:17:33.149 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/18' 2024-05-27T00:17:33.149 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/13' 2024-05-27T00:17:33.149 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/33' 2024-05-27T00:17:33.149 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/17' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/35' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './138/37' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './138' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/19' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/15' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/31' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/36' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/20' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/12' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/39' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/10' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/34' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/38' 2024-05-27T00:17:33.150 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/11' 2024-05-27T00:17:33.151 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/16' 2024-05-27T00:17:33.151 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/32' 2024-05-27T00:17:33.151 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/30' 2024-05-27T00:17:33.151 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/14' 2024-05-27T00:17:33.151 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/18' 2024-05-27T00:17:33.151 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/13' 2024-05-27T00:17:33.151 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/33' 2024-05-27T00:17:33.151 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/17' 2024-05-27T00:17:33.151 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/35' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './139/37' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './139' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/19' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/15' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/31' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/36' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/20' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/12' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/39' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/10' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/34' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/38' 2024-05-27T00:17:33.152 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/11' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/16' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/32' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/30' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/14' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/18' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/13' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/33' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/17' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/35' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed './140/37' 2024-05-27T00:17:33.153 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './140' 2024-05-27T00:17:33.154 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/19' 2024-05-27T00:17:33.154 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/15' 2024-05-27T00:17:33.154 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/31' 2024-05-27T00:17:33.154 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/36' 2024-05-27T00:17:33.154 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/20' 2024-05-27T00:17:33.154 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/12' 2024-05-27T00:17:33.154 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/39' 2024-05-27T00:17:33.154 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/10' 2024-05-27T00:17:33.154 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/34' 2024-05-27T00:17:33.154 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/38' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/11' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/16' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/32' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/30' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/14' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/18' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/13' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/33' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/17' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/35' 2024-05-27T00:17:33.155 INFO:tasks.workunit.client.0.smithi098.stdout:removed './141/37' 2024-05-27T00:17:33.156 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './141' 2024-05-27T00:17:33.156 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/19' 2024-05-27T00:17:33.156 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/15' 2024-05-27T00:17:33.156 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/31' 2024-05-27T00:17:33.156 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/36' 2024-05-27T00:17:33.156 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/20' 2024-05-27T00:17:33.156 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/12' 2024-05-27T00:17:33.156 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/39' 2024-05-27T00:17:33.156 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/10' 2024-05-27T00:17:33.156 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/34' 2024-05-27T00:17:33.157 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/38' 2024-05-27T00:17:33.157 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/11' 2024-05-27T00:17:33.157 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/16' 2024-05-27T00:17:33.157 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/32' 2024-05-27T00:17:33.157 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/30' 2024-05-27T00:17:33.157 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/14' 2024-05-27T00:17:33.157 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/18' 2024-05-27T00:17:33.157 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/13' 2024-05-27T00:17:33.157 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/33' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/17' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/35' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './142/37' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './142' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/19' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/15' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/31' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/36' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/20' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/12' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/39' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/10' 2024-05-27T00:17:33.158 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/34' 2024-05-27T00:17:33.159 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/38' 2024-05-27T00:17:33.159 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/11' 2024-05-27T00:17:33.159 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/16' 2024-05-27T00:17:33.159 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/32' 2024-05-27T00:17:33.159 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/30' 2024-05-27T00:17:33.159 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/14' 2024-05-27T00:17:33.159 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/18' 2024-05-27T00:17:33.160 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/13' 2024-05-27T00:17:33.160 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/33' 2024-05-27T00:17:33.160 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/17' 2024-05-27T00:17:33.160 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/35' 2024-05-27T00:17:33.160 INFO:tasks.workunit.client.0.smithi098.stdout:removed './143/37' 2024-05-27T00:17:33.160 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './143' 2024-05-27T00:17:33.160 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/19' 2024-05-27T00:17:33.160 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/15' 2024-05-27T00:17:33.160 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/31' 2024-05-27T00:17:33.160 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/36' 2024-05-27T00:17:33.161 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/20' 2024-05-27T00:17:33.161 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/12' 2024-05-27T00:17:33.161 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/39' 2024-05-27T00:17:33.161 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/10' 2024-05-27T00:17:33.161 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/34' 2024-05-27T00:17:33.161 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/38' 2024-05-27T00:17:33.161 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/11' 2024-05-27T00:17:33.162 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/16' 2024-05-27T00:17:33.162 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/32' 2024-05-27T00:17:33.162 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/30' 2024-05-27T00:17:33.162 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/14' 2024-05-27T00:17:33.162 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/18' 2024-05-27T00:17:33.162 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/13' 2024-05-27T00:17:33.162 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/33' 2024-05-27T00:17:33.162 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/17' 2024-05-27T00:17:33.162 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/35' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './144/37' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './144' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/19' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/15' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/31' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/36' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/20' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/12' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/39' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/10' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/34' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/38' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/11' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/16' 2024-05-27T00:17:33.163 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/32' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/30' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/14' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/18' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/13' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/33' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/17' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/35' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './145/37' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './145' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/19' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/15' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/31' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/36' 2024-05-27T00:17:33.164 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/20' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/12' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/39' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/10' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/34' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/38' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/11' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/16' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/32' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/30' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/14' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/18' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/13' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/33' 2024-05-27T00:17:33.165 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/17' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/35' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './146/37' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './146' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/19' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/15' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/31' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/36' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/20' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/12' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/39' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/10' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/34' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/38' 2024-05-27T00:17:33.166 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/11' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/16' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/32' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/30' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/14' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/18' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/13' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/33' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/17' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/35' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './147/37' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './147' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/19' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/15' 2024-05-27T00:17:33.167 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/31' 2024-05-27T00:17:33.168 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/36' 2024-05-27T00:17:33.871 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/20' 2024-05-27T00:17:33.871 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/12' 2024-05-27T00:17:33.871 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/39' 2024-05-27T00:17:33.871 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/10' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/34' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/38' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/11' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/16' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/32' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/30' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/14' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/18' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/13' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/33' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/17' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/35' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './148/37' 2024-05-27T00:17:33.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './148' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/19' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/15' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/31' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/36' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/20' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/12' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/39' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/10' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/34' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/38' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/11' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/16' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/32' 2024-05-27T00:17:33.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/30' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/14' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/18' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/13' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/33' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/17' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/35' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './149/37' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './149' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/19' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/15' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/31' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/36' 2024-05-27T00:17:33.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/20' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/12' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/39' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/10' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/34' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/38' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/11' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/16' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/32' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/30' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/14' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/18' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/13' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/33' 2024-05-27T00:17:33.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/17' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/35' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './150/37' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './150' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/19' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/15' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/31' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/36' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/20' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/12' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/39' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/10' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/34' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/38' 2024-05-27T00:17:33.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/11' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/16' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/32' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/30' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/14' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/18' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/13' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/33' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/17' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/35' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './151/37' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './151' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/19' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/15' 2024-05-27T00:17:33.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/31' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/36' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/20' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/12' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/39' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/10' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/34' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/38' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/11' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/16' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/32' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/30' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/14' 2024-05-27T00:17:33.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/18' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/13' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/33' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/17' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/35' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './152/37' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './152' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/19' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/15' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/31' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/36' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/20' 2024-05-27T00:17:33.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/12' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/39' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/10' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/34' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/38' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/11' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/16' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/32' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/30' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/14' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/18' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/13' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/33' 2024-05-27T00:17:33.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/17' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/35' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './153/37' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './153' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/19' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/15' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/31' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/36' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/20' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/12' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/39' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/10' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/34' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/38' 2024-05-27T00:17:33.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/11' 2024-05-27T00:17:33.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/16' 2024-05-27T00:17:33.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/32' 2024-05-27T00:17:33.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/30' 2024-05-27T00:17:33.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/14' 2024-05-27T00:17:33.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/18' 2024-05-27T00:17:33.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/13' 2024-05-27T00:17:33.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/33' 2024-05-27T00:17:33.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/17' 2024-05-27T00:17:33.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/35' 2024-05-27T00:17:33.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './154/37' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './154' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/19' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/15' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/31' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/36' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/20' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/12' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/39' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/10' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/34' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/38' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/11' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/16' 2024-05-27T00:17:33.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/32' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/30' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/14' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/18' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/13' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/33' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/17' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/35' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './155/37' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './155' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/19' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/15' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/31' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/36' 2024-05-27T00:17:33.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/20' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/12' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/39' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/10' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/34' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/38' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/11' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/16' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/32' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/30' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/14' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/18' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/13' 2024-05-27T00:17:33.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/33' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/17' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/35' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './156/37' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './156' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/19' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/15' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/31' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/36' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/20' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/12' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/39' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/10' 2024-05-27T00:17:33.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/34' 2024-05-27T00:17:33.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/38' 2024-05-27T00:17:33.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/11' 2024-05-27T00:17:33.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/16' 2024-05-27T00:17:33.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/32' 2024-05-27T00:17:33.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/30' 2024-05-27T00:17:33.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/14' 2024-05-27T00:17:33.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/18' 2024-05-27T00:17:33.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/13' 2024-05-27T00:17:33.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/33' 2024-05-27T00:17:35.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/17' 2024-05-27T00:17:35.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/35' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './157/37' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './157' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/19' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/15' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/31' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/36' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/20' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/12' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/39' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/10' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/34' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/38' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/11' 2024-05-27T00:17:35.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/16' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/32' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/30' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/14' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/18' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/13' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/33' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/17' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/35' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './158/37' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './158' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/19' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/15' 2024-05-27T00:17:35.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/31' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/36' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/20' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/12' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/39' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/10' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/34' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/38' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/11' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/16' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/32' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/30' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/14' 2024-05-27T00:17:35.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/18' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/13' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/33' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/17' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/35' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './159/37' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './159' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/19' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/15' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/31' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/36' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/20' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/12' 2024-05-27T00:17:35.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/39' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/10' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/34' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/38' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/11' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/16' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/32' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/30' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/14' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/18' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/13' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/33' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/17' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/35' 2024-05-27T00:17:35.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './160/37' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './160' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/19' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/15' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/31' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/36' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/20' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/12' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/39' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/10' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/34' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/38' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/11' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/16' 2024-05-27T00:17:35.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/32' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/30' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/14' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/18' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/13' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/33' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/17' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/35' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './161/37' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './161' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/19' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/15' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/31' 2024-05-27T00:17:35.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/36' 2024-05-27T00:17:35.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/20' 2024-05-27T00:17:35.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/12' 2024-05-27T00:17:35.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/39' 2024-05-27T00:17:35.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/10' 2024-05-27T00:17:35.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/34' 2024-05-27T00:17:35.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/38' 2024-05-27T00:17:35.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/11' 2024-05-27T00:17:35.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/16' 2024-05-27T00:17:35.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/32' 2024-05-27T00:17:35.010 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/30' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/14' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/18' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/13' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/33' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/17' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/35' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './162/37' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './162' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/19' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/15' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/31' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/36' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/20' 2024-05-27T00:17:35.011 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/12' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/39' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/10' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/34' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/38' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/11' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/16' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/32' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/30' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/14' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/18' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/13' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/33' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/17' 2024-05-27T00:17:35.012 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/35' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './163/37' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './163' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/19' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/15' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/31' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/36' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/20' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/12' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/39' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/10' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/34' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/38' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/11' 2024-05-27T00:17:35.013 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/16' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/32' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/30' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/14' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/18' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/13' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/33' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/17' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/35' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './164/37' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './164' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/19' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/15' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/31' 2024-05-27T00:17:35.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/36' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/20' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/12' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/39' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/10' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/34' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/38' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/11' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/16' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/32' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/30' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/14' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/18' 2024-05-27T00:17:35.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/13' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/33' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/17' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/35' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './165/37' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './165' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/19' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/15' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/31' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/36' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/20' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/12' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/39' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/10' 2024-05-27T00:17:35.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/34' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/38' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/11' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/16' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/32' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/30' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/14' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/18' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/13' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/33' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/17' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/35' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './166/37' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './166' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/19' 2024-05-27T00:17:35.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/15' 2024-05-27T00:17:35.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/31' 2024-05-27T00:17:35.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/36' 2024-05-27T00:17:35.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/20' 2024-05-27T00:17:35.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/12' 2024-05-27T00:17:35.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/39' 2024-05-27T00:17:35.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/10' 2024-05-27T00:17:35.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/34' 2024-05-27T00:17:35.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/38' 2024-05-27T00:17:35.512 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/11' 2024-05-27T00:17:35.512 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/16' 2024-05-27T00:17:35.512 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/32' 2024-05-27T00:17:35.512 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/30' 2024-05-27T00:17:35.512 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/14' 2024-05-27T00:17:35.512 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/18' 2024-05-27T00:17:35.512 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/13' 2024-05-27T00:17:35.512 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/33' 2024-05-27T00:17:35.512 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/17' 2024-05-27T00:17:35.512 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/35' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './167/37' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './167' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/19' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/15' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/31' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/36' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/20' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/12' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/39' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/10' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/34' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/38' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/11' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/16' 2024-05-27T00:17:35.513 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/32' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/30' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/14' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/18' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/13' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/33' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/17' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/35' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './168/37' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './168' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/19' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/15' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/31' 2024-05-27T00:17:35.514 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/36' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/20' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/12' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/39' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/10' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/34' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/38' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/11' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/16' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/32' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/30' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/14' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/18' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/13' 2024-05-27T00:17:35.515 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/33' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/17' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/35' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './169/37' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './169' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/19' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/15' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/31' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/36' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/20' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/12' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/39' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/10' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/34' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/38' 2024-05-27T00:17:35.516 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/11' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/16' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/32' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/30' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/14' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/18' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/13' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/33' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/17' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/35' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './170/37' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './170' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/19' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/15' 2024-05-27T00:17:35.517 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/31' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/36' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/20' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/12' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/39' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/10' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/34' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/38' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/11' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/16' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/32' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/30' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/14' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/18' 2024-05-27T00:17:35.518 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/13' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/33' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/17' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/35' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './171/37' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './171' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/19' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/15' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/31' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/36' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/20' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/12' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/39' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/10' 2024-05-27T00:17:35.519 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/34' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/38' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/11' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/16' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/32' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/30' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/14' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/18' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/13' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/33' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/17' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/35' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './172/37' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './172' 2024-05-27T00:17:35.520 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/19' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/15' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/31' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/36' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/20' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/12' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/39' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/10' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/34' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/38' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/11' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/16' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/32' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/30' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/14' 2024-05-27T00:17:35.521 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/18' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/13' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/33' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/17' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/35' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './173/37' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './173' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/19' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/15' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/31' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/36' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/20' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/12' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/39' 2024-05-27T00:17:35.522 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/10' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/34' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/38' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/11' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/16' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/32' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/30' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/14' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/18' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/13' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/33' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/17' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/35' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed './174/37' 2024-05-27T00:17:35.523 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './174' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/19' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/15' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/31' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/36' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/20' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/12' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/39' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/10' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/34' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/38' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/11' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/16' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/32' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/30' 2024-05-27T00:17:35.524 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/14' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/18' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/13' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/33' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/17' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/35' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './175/37' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './175' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/19' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/15' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/31' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/36' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/20' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/12' 2024-05-27T00:17:35.525 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/39' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/10' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/34' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/38' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/11' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/16' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/32' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/30' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/14' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/18' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/13' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/33' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/17' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/35' 2024-05-27T00:17:35.526 INFO:tasks.workunit.client.0.smithi098.stdout:removed './176/37' 2024-05-27T00:17:35.527 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './176' 2024-05-27T00:17:35.527 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/19' 2024-05-27T00:17:35.527 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/15' 2024-05-27T00:17:36.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/31' 2024-05-27T00:17:36.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/36' 2024-05-27T00:17:36.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/20' 2024-05-27T00:17:36.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/12' 2024-05-27T00:17:36.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/39' 2024-05-27T00:17:36.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/10' 2024-05-27T00:17:36.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/34' 2024-05-27T00:17:36.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/38' 2024-05-27T00:17:36.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/11' 2024-05-27T00:17:36.014 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/16' 2024-05-27T00:17:36.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/32' 2024-05-27T00:17:36.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/30' 2024-05-27T00:17:36.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/14' 2024-05-27T00:17:36.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/18' 2024-05-27T00:17:36.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/13' 2024-05-27T00:17:36.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/33' 2024-05-27T00:17:36.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/17' 2024-05-27T00:17:36.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/35' 2024-05-27T00:17:36.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed './177/37' 2024-05-27T00:17:36.015 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './177' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/19' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/15' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/31' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/36' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/20' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/12' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/39' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/10' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/34' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/38' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/11' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/16' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/32' 2024-05-27T00:17:36.016 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/30' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/14' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/18' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/13' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/33' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/17' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/35' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './178/37' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './178' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/19' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/15' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/31' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/36' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/20' 2024-05-27T00:17:36.017 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/12' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/39' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/10' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/34' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/38' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/11' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/16' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/32' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/30' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/14' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/18' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/13' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/33' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/17' 2024-05-27T00:17:36.018 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/35' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './179/37' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './179' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/19' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/15' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/31' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/36' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/20' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/12' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/39' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/10' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/34' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/38' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/11' 2024-05-27T00:17:36.019 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/16' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/32' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/30' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/14' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/18' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/13' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/33' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/17' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/35' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './180/37' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './180' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/19' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/15' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/31' 2024-05-27T00:17:36.020 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/36' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/20' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/12' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/39' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/10' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/34' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/38' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/11' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/16' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/32' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/30' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/14' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/18' 2024-05-27T00:17:36.021 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/13' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/33' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/17' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/35' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './181/37' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './181' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/19' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/15' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/31' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/36' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/20' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/12' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/39' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/10' 2024-05-27T00:17:36.022 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/34' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/38' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/11' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/16' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/32' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/30' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/14' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/18' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/13' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/33' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/17' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/35' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './182/37' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './182' 2024-05-27T00:17:36.023 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/19' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/15' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/31' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/36' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/20' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/12' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/39' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/10' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/34' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/38' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/11' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/16' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/32' 2024-05-27T00:17:36.024 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/30' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/14' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/18' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/13' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/33' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/17' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/35' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './183/37' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './183' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/19' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/15' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/31' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/36' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/20' 2024-05-27T00:17:36.025 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/12' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/39' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/10' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/34' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/38' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/11' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/16' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/32' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/30' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/14' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/18' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/13' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/33' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/17' 2024-05-27T00:17:36.026 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/35' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './184/37' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './184' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/19' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/15' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/31' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/36' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/20' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/12' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/39' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/10' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/34' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/38' 2024-05-27T00:17:36.027 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/11' 2024-05-27T00:17:36.028 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/16' 2024-05-27T00:17:36.028 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/32' 2024-05-27T00:17:36.028 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/30' 2024-05-27T00:17:36.028 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/14' 2024-05-27T00:17:36.028 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/18' 2024-05-27T00:17:36.028 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/13' 2024-05-27T00:17:36.028 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/33' 2024-05-27T00:17:36.028 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/17' 2024-05-27T00:17:36.029 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/35' 2024-05-27T00:17:36.029 INFO:tasks.workunit.client.0.smithi098.stdout:removed './185/37' 2024-05-27T00:17:36.029 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './185' 2024-05-27T00:17:36.029 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/19' 2024-05-27T00:17:36.029 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/15' 2024-05-27T00:17:36.029 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/31' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/36' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/20' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/12' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/39' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/10' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/34' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/38' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/11' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/16' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/32' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/30' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/14' 2024-05-27T00:17:36.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/18' 2024-05-27T00:17:36.494 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/13' 2024-05-27T00:17:36.494 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/33' 2024-05-27T00:17:36.494 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/17' 2024-05-27T00:17:36.494 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/35' 2024-05-27T00:17:36.494 INFO:tasks.workunit.client.0.smithi098.stdout:removed './186/37' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './186' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/19' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/15' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/31' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/36' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/20' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/12' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/39' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/10' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/34' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/38' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/11' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/16' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/32' 2024-05-27T00:17:36.495 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/30' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/14' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/18' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/13' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/33' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/17' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/35' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './187/37' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './187' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/19' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/15' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/31' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/36' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/20' 2024-05-27T00:17:36.496 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/12' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/39' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/10' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/34' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/38' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/11' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/16' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/32' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/30' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/14' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/18' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/13' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/33' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/17' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/35' 2024-05-27T00:17:36.497 INFO:tasks.workunit.client.0.smithi098.stdout:removed './188/37' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './188' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/19' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/15' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/31' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/36' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/20' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/12' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/39' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/10' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/34' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/38' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/11' 2024-05-27T00:17:36.498 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/16' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/32' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/30' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/14' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/18' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/13' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/33' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/17' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/35' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './189/37' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './189' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/19' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/15' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/31' 2024-05-27T00:17:36.499 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/36' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/20' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/12' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/39' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/10' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/34' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/38' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/11' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/16' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/32' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/30' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/14' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/18' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/13' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/33' 2024-05-27T00:17:36.500 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/17' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/35' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './190/37' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './190' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/19' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/15' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/31' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/36' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/20' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/12' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/39' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/10' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/34' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/38' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/11' 2024-05-27T00:17:36.501 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/16' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/32' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/30' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/14' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/18' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/13' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/33' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/17' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/35' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './191/37' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './191' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/19' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/15' 2024-05-27T00:17:36.502 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/31' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/36' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/20' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/12' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/39' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/10' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/34' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/38' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/11' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/16' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/32' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/30' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/14' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/18' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/13' 2024-05-27T00:17:36.503 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/33' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/17' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/35' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './192/37' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './192' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/19' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/15' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/31' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/36' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/20' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/12' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/39' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/10' 2024-05-27T00:17:36.504 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/34' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/38' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/11' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/16' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/32' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/30' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/14' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/18' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/13' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/33' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/17' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/35' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './193/37' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './193' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/19' 2024-05-27T00:17:36.505 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/15' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/31' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/36' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/20' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/12' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/39' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/10' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/34' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/38' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/11' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/16' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/32' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/30' 2024-05-27T00:17:36.506 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/14' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/18' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/13' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/33' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/17' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/35' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './194/37' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './194' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/19' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/15' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/31' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/36' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/20' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/12' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/39' 2024-05-27T00:17:36.507 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/10' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/34' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/38' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/11' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/16' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/32' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/30' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/14' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/18' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/13' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/33' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/17' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/35' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed './195/37' 2024-05-27T00:17:36.508 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './195' 2024-05-27T00:17:36.509 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/19' 2024-05-27T00:17:36.509 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/15' 2024-05-27T00:17:36.509 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/31' 2024-05-27T00:17:36.509 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/36' 2024-05-27T00:17:36.509 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/20' 2024-05-27T00:17:36.509 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/12' 2024-05-27T00:17:36.509 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/39' 2024-05-27T00:17:36.509 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/10' 2024-05-27T00:17:37.066 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/34' 2024-05-27T00:17:37.066 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/38' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/11' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/16' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/32' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/30' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/14' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/18' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/13' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/33' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/17' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/35' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './196/37' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './196' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/19' 2024-05-27T00:17:37.067 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/15' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/31' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/36' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/20' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/12' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/39' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/10' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/34' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/38' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/11' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/16' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/32' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/30' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/14' 2024-05-27T00:17:37.068 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/18' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/13' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/33' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/17' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/35' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './197/37' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './197' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/19' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/15' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/31' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/36' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/20' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/12' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/39' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/10' 2024-05-27T00:17:37.069 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/34' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/38' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/11' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/16' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/32' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/30' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/14' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/18' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/13' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/33' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/17' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/35' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './198/37' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './198' 2024-05-27T00:17:37.070 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/19' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/15' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/31' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/36' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/20' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/12' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/39' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/10' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/34' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/38' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/11' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/16' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/32' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/30' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/14' 2024-05-27T00:17:37.071 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/18' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/13' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/33' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/17' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/35' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './199/37' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './199' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/21' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/27' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/31' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/36' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/20' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/39' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/34' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/38' 2024-05-27T00:17:37.072 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/28' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/24' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/29' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/32' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/22' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/25' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/30' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/33' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/23' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/35' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/26' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './200/37' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './200' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/21' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/27' 2024-05-27T00:17:37.073 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/31' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/36' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/20' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/39' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/34' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/38' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/28' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/24' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/29' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/32' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/22' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/25' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/30' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/33' 2024-05-27T00:17:37.074 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/23' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/35' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/26' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './201/37' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './201' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/21' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/27' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/31' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/36' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/20' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/39' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/34' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/38' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/28' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/24' 2024-05-27T00:17:37.075 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/29' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/32' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/22' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/25' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/30' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/33' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/23' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/35' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/26' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './202/37' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './202' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/21' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/27' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/31' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/36' 2024-05-27T00:17:37.076 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/20' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/39' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/34' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/38' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/28' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/24' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/29' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/32' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/22' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/25' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/30' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/33' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/23' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/35' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/26' 2024-05-27T00:17:37.077 INFO:tasks.workunit.client.0.smithi098.stdout:removed './203/37' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './203' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/21' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/27' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/31' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/36' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/20' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/39' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/34' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/38' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/28' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/24' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/29' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/32' 2024-05-27T00:17:37.078 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/22' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/25' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/30' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/33' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/23' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/35' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/26' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './204/37' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './204' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/21' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/27' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/31' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/36' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/20' 2024-05-27T00:17:37.079 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/39' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/34' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/38' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/28' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/24' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/29' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/32' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/22' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/25' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/30' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/33' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/23' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/35' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/26' 2024-05-27T00:17:37.080 INFO:tasks.workunit.client.0.smithi098.stdout:removed './205/37' 2024-05-27T00:17:37.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './205' 2024-05-27T00:17:37.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/21' 2024-05-27T00:17:37.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/27' 2024-05-27T00:17:37.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/31' 2024-05-27T00:17:37.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/36' 2024-05-27T00:17:37.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/20' 2024-05-27T00:17:37.081 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/39' 2024-05-27T00:17:37.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/34' 2024-05-27T00:17:37.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/38' 2024-05-27T00:17:37.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/28' 2024-05-27T00:17:37.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/24' 2024-05-27T00:17:37.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/29' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/32' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/22' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/25' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/30' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/33' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/23' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/35' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/26' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './206/37' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './206' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/21' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/27' 2024-05-27T00:17:37.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/31' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/36' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/20' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/39' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/34' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/38' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/28' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/24' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/29' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/32' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/22' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/25' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/30' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/33' 2024-05-27T00:17:37.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/23' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/35' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/26' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './207/37' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './207' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/21' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/27' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/31' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/36' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/20' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/39' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/34' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/38' 2024-05-27T00:17:37.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/28' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/24' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/29' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/32' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/22' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/25' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/30' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/33' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/23' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/35' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/26' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './208/37' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './208' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/21' 2024-05-27T00:17:37.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/27' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/31' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/36' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/20' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/39' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/34' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/38' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/28' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/24' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/29' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/32' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/22' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/25' 2024-05-27T00:17:37.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/30' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/33' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/23' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/35' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/26' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './209/37' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './209' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/21' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/27' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/31' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/36' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/20' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/39' 2024-05-27T00:17:37.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/34' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/38' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/28' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/24' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/29' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/32' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/22' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/25' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/30' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/33' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/23' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/35' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/26' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './210/37' 2024-05-27T00:17:37.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './210' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/21' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/27' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/31' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/36' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/20' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/39' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/34' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/38' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/28' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/24' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/29' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/32' 2024-05-27T00:17:37.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/22' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/25' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/30' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/33' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/23' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/35' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/26' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './211/37' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './211' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/21' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/27' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/31' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/36' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/20' 2024-05-27T00:17:37.474 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/39' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/34' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/38' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/28' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/24' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/29' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/32' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/22' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/25' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/30' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/33' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/23' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/35' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/26' 2024-05-27T00:17:37.475 INFO:tasks.workunit.client.0.smithi098.stdout:removed './212/37' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './212' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/21' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/27' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/31' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/36' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/20' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/39' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/34' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/38' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/28' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/24' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/29' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/32' 2024-05-27T00:17:37.476 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/22' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/25' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/30' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/33' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/23' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/35' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/26' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './213/37' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './213' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/21' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/27' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/31' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/36' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/20' 2024-05-27T00:17:37.477 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/39' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/34' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/38' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/28' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/24' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/29' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/32' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/22' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/25' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/30' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/33' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/23' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/35' 2024-05-27T00:17:37.478 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/26' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './214/37' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './214' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/21' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/27' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/31' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/36' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/20' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/39' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/34' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/38' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/28' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/24' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/29' 2024-05-27T00:17:37.479 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/32' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/22' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/25' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/30' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/33' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/23' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/35' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/26' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './215/37' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './215' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/21' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/27' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/31' 2024-05-27T00:17:37.480 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/36' 2024-05-27T00:17:37.481 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/20' 2024-05-27T00:17:37.481 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/39' 2024-05-27T00:17:37.481 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/34' 2024-05-27T00:17:37.481 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/38' 2024-05-27T00:17:37.989 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/28' 2024-05-27T00:17:37.989 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/24' 2024-05-27T00:17:37.989 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/29' 2024-05-27T00:17:37.989 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/32' 2024-05-27T00:17:37.989 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/22' 2024-05-27T00:17:37.989 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/25' 2024-05-27T00:17:37.990 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/30' 2024-05-27T00:17:37.990 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/33' 2024-05-27T00:17:37.990 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/23' 2024-05-27T00:17:37.990 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/35' 2024-05-27T00:17:37.990 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/26' 2024-05-27T00:17:37.990 INFO:tasks.workunit.client.0.smithi098.stdout:removed './216/37' 2024-05-27T00:17:37.990 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './216' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/21' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/27' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/31' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/36' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/20' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/39' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/34' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/38' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/28' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/24' 2024-05-27T00:17:37.991 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/29' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/32' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/22' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/25' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/30' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/33' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/23' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/35' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/26' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed './217/37' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './217' 2024-05-27T00:17:37.992 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/21' 2024-05-27T00:17:37.993 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/27' 2024-05-27T00:17:37.993 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/31' 2024-05-27T00:17:37.993 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/36' 2024-05-27T00:17:37.993 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/20' 2024-05-27T00:17:37.993 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/39' 2024-05-27T00:17:37.993 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/34' 2024-05-27T00:17:37.993 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/38' 2024-05-27T00:17:37.993 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/28' 2024-05-27T00:17:37.993 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/24' 2024-05-27T00:17:37.993 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/29' 2024-05-27T00:17:37.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/32' 2024-05-27T00:17:37.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/22' 2024-05-27T00:17:37.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/25' 2024-05-27T00:17:37.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/30' 2024-05-27T00:17:37.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/33' 2024-05-27T00:17:37.994 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/23' 2024-05-27T00:17:37.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/35' 2024-05-27T00:17:37.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/26' 2024-05-27T00:17:37.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './218/37' 2024-05-27T00:17:37.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './218' 2024-05-27T00:17:37.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/21' 2024-05-27T00:17:37.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/27' 2024-05-27T00:17:37.995 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/31' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/36' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/20' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/39' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/34' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/38' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/28' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/24' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/29' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/32' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/22' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/25' 2024-05-27T00:17:37.996 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/30' 2024-05-27T00:17:37.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/33' 2024-05-27T00:17:37.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/23' 2024-05-27T00:17:37.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/35' 2024-05-27T00:17:37.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/26' 2024-05-27T00:17:37.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './219/37' 2024-05-27T00:17:37.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './219' 2024-05-27T00:17:37.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/21' 2024-05-27T00:17:37.997 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/27' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/31' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/36' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/20' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/39' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/34' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/38' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/28' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/24' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/29' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/32' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/22' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/25' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/30' 2024-05-27T00:17:37.998 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/33' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/23' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/35' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/26' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './220/37' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './220' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/21' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/27' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/31' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/36' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/20' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/39' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/34' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/38' 2024-05-27T00:17:37.999 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/28' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/24' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/29' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/32' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/22' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/25' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/30' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/33' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/23' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/35' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/26' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed './221/37' 2024-05-27T00:17:38.000 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './221' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/21' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/27' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/31' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/36' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/20' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/39' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/34' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/38' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/28' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/24' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/29' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/32' 2024-05-27T00:17:38.001 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/22' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/25' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/30' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/33' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/23' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/35' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/26' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './222/37' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './222' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/21' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/27' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/31' 2024-05-27T00:17:38.002 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/36' 2024-05-27T00:17:38.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/20' 2024-05-27T00:17:38.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/39' 2024-05-27T00:17:38.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/34' 2024-05-27T00:17:38.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/38' 2024-05-27T00:17:38.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/28' 2024-05-27T00:17:38.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/24' 2024-05-27T00:17:38.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/29' 2024-05-27T00:17:38.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/32' 2024-05-27T00:17:38.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/22' 2024-05-27T00:17:38.003 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/25' 2024-05-27T00:17:38.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/30' 2024-05-27T00:17:38.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/33' 2024-05-27T00:17:38.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/23' 2024-05-27T00:17:38.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/35' 2024-05-27T00:17:38.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/26' 2024-05-27T00:17:38.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './223/37' 2024-05-27T00:17:38.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './223' 2024-05-27T00:17:38.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/21' 2024-05-27T00:17:38.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/27' 2024-05-27T00:17:38.004 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/31' 2024-05-27T00:17:38.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/36' 2024-05-27T00:17:38.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/20' 2024-05-27T00:17:38.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/39' 2024-05-27T00:17:38.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/34' 2024-05-27T00:17:38.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/38' 2024-05-27T00:17:38.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/28' 2024-05-27T00:17:38.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/24' 2024-05-27T00:17:38.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/29' 2024-05-27T00:17:38.005 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/32' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/22' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/25' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/30' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/33' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/23' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/35' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/26' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './224/37' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './224' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/21' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/27' 2024-05-27T00:17:38.006 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/31' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/36' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/20' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/39' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/34' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/38' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/28' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/24' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/29' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/32' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/22' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/25' 2024-05-27T00:17:38.007 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/30' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/33' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/23' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/35' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/26' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './225/37' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './225' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/21' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/27' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/31' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/36' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/20' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/39' 2024-05-27T00:17:38.008 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/34' 2024-05-27T00:17:38.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/38' 2024-05-27T00:17:38.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/28' 2024-05-27T00:17:38.009 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/24' 2024-05-27T00:17:38.374 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/29' 2024-05-27T00:17:38.374 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/32' 2024-05-27T00:17:38.374 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/22' 2024-05-27T00:17:38.374 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/25' 2024-05-27T00:17:38.374 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/30' 2024-05-27T00:17:38.374 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/33' 2024-05-27T00:17:38.374 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/23' 2024-05-27T00:17:38.375 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/35' 2024-05-27T00:17:38.375 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/26' 2024-05-27T00:17:38.375 INFO:tasks.workunit.client.0.smithi098.stdout:removed './226/37' 2024-05-27T00:17:38.375 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './226' 2024-05-27T00:17:38.375 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/21' 2024-05-27T00:17:38.375 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/27' 2024-05-27T00:17:38.375 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/31' 2024-05-27T00:17:38.375 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/36' 2024-05-27T00:17:38.375 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/20' 2024-05-27T00:17:38.375 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/39' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/34' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/38' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/28' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/24' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/29' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/32' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/22' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/25' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/30' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/33' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/23' 2024-05-27T00:17:38.376 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/35' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/26' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed './227/37' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './227' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/21' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/27' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/31' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/36' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/20' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/39' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/34' 2024-05-27T00:17:38.377 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/38' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/28' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/24' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/29' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/32' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/22' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/25' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/30' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/33' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/23' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/35' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/26' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed './228/37' 2024-05-27T00:17:38.378 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './228' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/21' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/27' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/31' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/36' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/20' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/39' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/34' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/38' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/28' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/24' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/29' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/32' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/22' 2024-05-27T00:17:38.379 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/25' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/30' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/33' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/23' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/35' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/26' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './229/37' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './229' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/21' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/27' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/31' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/36' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/20' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/39' 2024-05-27T00:17:38.380 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/34' 2024-05-27T00:17:38.381 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/38' 2024-05-27T00:17:38.381 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/28' 2024-05-27T00:17:38.381 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/24' 2024-05-27T00:17:38.381 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/29' 2024-05-27T00:17:38.381 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/32' 2024-05-27T00:17:38.381 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/22' 2024-05-27T00:17:38.381 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/25' 2024-05-27T00:17:38.381 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/30' 2024-05-27T00:17:38.382 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/33' 2024-05-27T00:17:38.382 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/23' 2024-05-27T00:17:38.382 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/35' 2024-05-27T00:17:38.382 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/26' 2024-05-27T00:17:38.382 INFO:tasks.workunit.client.0.smithi098.stdout:removed './230/37' 2024-05-27T00:17:38.382 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './230' 2024-05-27T00:17:38.382 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/21' 2024-05-27T00:17:38.382 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/27' 2024-05-27T00:17:38.383 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/31' 2024-05-27T00:17:38.383 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/36' 2024-05-27T00:17:38.383 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/20' 2024-05-27T00:17:38.383 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/39' 2024-05-27T00:17:38.383 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/34' 2024-05-27T00:17:38.383 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/38' 2024-05-27T00:17:38.383 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/28' 2024-05-27T00:17:38.384 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/24' 2024-05-27T00:17:38.384 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/29' 2024-05-27T00:17:38.384 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/32' 2024-05-27T00:17:38.384 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/22' 2024-05-27T00:17:38.384 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/25' 2024-05-27T00:17:38.384 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/30' 2024-05-27T00:17:38.384 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/33' 2024-05-27T00:17:38.384 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/23' 2024-05-27T00:17:38.385 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/35' 2024-05-27T00:17:38.385 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/26' 2024-05-27T00:17:38.385 INFO:tasks.workunit.client.0.smithi098.stdout:removed './231/37' 2024-05-27T00:17:38.385 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './231' 2024-05-27T00:17:38.385 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/21' 2024-05-27T00:17:38.385 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/27' 2024-05-27T00:17:38.385 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/31' 2024-05-27T00:17:38.385 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/36' 2024-05-27T00:17:38.385 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/20' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/39' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/34' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/38' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/28' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/24' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/29' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/32' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/22' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/25' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/30' 2024-05-27T00:17:38.386 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/33' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/23' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/35' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/26' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './232/37' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './232' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/21' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/27' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/31' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/36' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/20' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/39' 2024-05-27T00:17:38.387 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/34' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/38' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/28' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/24' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/29' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/32' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/22' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/25' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/30' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/33' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/23' 2024-05-27T00:17:38.388 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/35' 2024-05-27T00:17:38.389 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/26' 2024-05-27T00:17:38.389 INFO:tasks.workunit.client.0.smithi098.stdout:removed './233/37' 2024-05-27T00:17:38.389 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './233' 2024-05-27T00:17:38.389 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/21' 2024-05-27T00:17:38.389 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/27' 2024-05-27T00:17:38.389 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/31' 2024-05-27T00:17:38.390 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/36' 2024-05-27T00:17:38.390 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/20' 2024-05-27T00:17:38.390 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/39' 2024-05-27T00:17:38.390 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/34' 2024-05-27T00:17:38.390 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/38' 2024-05-27T00:17:38.390 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/28' 2024-05-27T00:17:38.390 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/24' 2024-05-27T00:17:38.390 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/29' 2024-05-27T00:17:38.390 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/32' 2024-05-27T00:17:38.391 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/22' 2024-05-27T00:17:38.391 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/25' 2024-05-27T00:17:38.391 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/30' 2024-05-27T00:17:38.391 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/33' 2024-05-27T00:17:38.391 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/23' 2024-05-27T00:17:38.391 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/35' 2024-05-27T00:17:38.391 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/26' 2024-05-27T00:17:38.392 INFO:tasks.workunit.client.0.smithi098.stdout:removed './234/37' 2024-05-27T00:17:38.392 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './234' 2024-05-27T00:17:38.392 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/21' 2024-05-27T00:17:38.392 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/27' 2024-05-27T00:17:38.392 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/31' 2024-05-27T00:17:38.392 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/36' 2024-05-27T00:17:38.392 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/20' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/39' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/34' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/38' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/28' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/24' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/29' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/32' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/22' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/25' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/30' 2024-05-27T00:17:38.393 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/33' 2024-05-27T00:17:38.394 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/23' 2024-05-27T00:17:38.394 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/35' 2024-05-27T00:17:38.394 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/26' 2024-05-27T00:17:38.394 INFO:tasks.workunit.client.0.smithi098.stdout:removed './235/37' 2024-05-27T00:17:38.394 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './235' 2024-05-27T00:17:38.394 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/21' 2024-05-27T00:17:38.394 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/27' 2024-05-27T00:17:38.394 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/31' 2024-05-27T00:17:38.394 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/36' 2024-05-27T00:17:38.394 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/20' 2024-05-27T00:17:38.395 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/39' 2024-05-27T00:17:38.395 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/34' 2024-05-27T00:17:38.395 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/38' 2024-05-27T00:17:38.395 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/28' 2024-05-27T00:17:38.395 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/24' 2024-05-27T00:17:38.395 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/29' 2024-05-27T00:17:38.395 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/32' 2024-05-27T00:17:38.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/22' 2024-05-27T00:17:38.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/25' 2024-05-27T00:17:38.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/30' 2024-05-27T00:17:38.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/33' 2024-05-27T00:17:38.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/23' 2024-05-27T00:17:38.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/35' 2024-05-27T00:17:38.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/26' 2024-05-27T00:17:38.872 INFO:tasks.workunit.client.0.smithi098.stdout:removed './236/37' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './236' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/21' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/27' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/31' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/36' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/20' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/39' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/34' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/38' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/28' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/24' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/29' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/32' 2024-05-27T00:17:38.873 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/22' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/25' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/30' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/33' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/23' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/35' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/26' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './237/37' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './237' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/21' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/27' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/31' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/36' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/20' 2024-05-27T00:17:38.874 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/39' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/34' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/38' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/28' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/24' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/29' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/32' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/22' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/25' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/30' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/33' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/23' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/35' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/26' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed './238/37' 2024-05-27T00:17:38.875 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './238' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/21' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/27' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/31' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/36' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/20' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/39' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/34' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/38' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/28' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/24' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/29' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/32' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/22' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/25' 2024-05-27T00:17:38.876 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/30' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/33' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/23' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/35' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/26' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './239/37' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './239' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/21' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/27' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/31' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/36' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/20' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/39' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/34' 2024-05-27T00:17:38.877 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/38' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/28' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/24' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/29' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/32' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/22' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/25' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/30' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/33' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/23' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/35' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/26' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed './240/37' 2024-05-27T00:17:38.878 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './240' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/21' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/27' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/31' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/36' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/20' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/39' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/34' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/38' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/28' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/24' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/29' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/32' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/22' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/25' 2024-05-27T00:17:38.879 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/30' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/33' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/23' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/35' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/26' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './241/37' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './241' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/21' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/27' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/31' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/36' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/20' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/39' 2024-05-27T00:17:38.880 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/34' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/38' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/28' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/24' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/29' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/32' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/22' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/25' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/30' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/33' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/23' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/35' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/26' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './242/37' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './242' 2024-05-27T00:17:38.881 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/21' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/27' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/31' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/36' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/20' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/39' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/34' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/38' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/28' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/24' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/29' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/32' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/22' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/25' 2024-05-27T00:17:38.882 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/30' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/33' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/23' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/35' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/26' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './243/37' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './243' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/21' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/27' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/31' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/36' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/20' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/39' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/34' 2024-05-27T00:17:38.883 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/38' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/28' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/24' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/29' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/32' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/22' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/25' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/30' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/33' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/23' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/35' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/26' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './244/37' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './244' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/21' 2024-05-27T00:17:38.884 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/27' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/31' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/36' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/20' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/39' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/34' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/38' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/28' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/24' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/29' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/32' 2024-05-27T00:17:38.885 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/22' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/25' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/30' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/33' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/23' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/35' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/26' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './245/37' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './245' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/21' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/27' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/31' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/36' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/20' 2024-05-27T00:17:38.886 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/39' 2024-05-27T00:17:38.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/34' 2024-05-27T00:17:38.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/38' 2024-05-27T00:17:38.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/28' 2024-05-27T00:17:38.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/24' 2024-05-27T00:17:38.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/29' 2024-05-27T00:17:38.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/32' 2024-05-27T00:17:38.887 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/22' 2024-05-27T00:17:39.281 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/25' 2024-05-27T00:17:39.281 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/30' 2024-05-27T00:17:39.281 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/33' 2024-05-27T00:17:39.281 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/23' 2024-05-27T00:17:39.281 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/35' 2024-05-27T00:17:39.281 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/26' 2024-05-27T00:17:39.281 INFO:tasks.workunit.client.0.smithi098.stdout:removed './246/37' 2024-05-27T00:17:39.281 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './246' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/21' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/27' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/31' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/36' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/20' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/39' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/34' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/38' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/28' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/24' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/29' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/32' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/22' 2024-05-27T00:17:39.282 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/25' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/30' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/33' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/23' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/35' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/26' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './247/37' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './247' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/21' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/27' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/31' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/36' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/20' 2024-05-27T00:17:39.283 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/39' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/34' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/38' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/28' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/24' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/29' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/32' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/22' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/25' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/30' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/33' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/23' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/35' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/26' 2024-05-27T00:17:39.284 INFO:tasks.workunit.client.0.smithi098.stdout:removed './248/37' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './248' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/21' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/27' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/31' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/36' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/20' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/39' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/34' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/38' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/28' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/24' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/29' 2024-05-27T00:17:39.285 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/32' 2024-05-27T00:17:39.286 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/22' 2024-05-27T00:17:39.286 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/25' 2024-05-27T00:17:39.286 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/30' 2024-05-27T00:17:39.286 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/33' 2024-05-27T00:17:39.286 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/23' 2024-05-27T00:17:39.286 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/35' 2024-05-27T00:17:39.286 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/26' 2024-05-27T00:17:39.286 INFO:tasks.workunit.client.0.smithi098.stdout:removed './249/37' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './249' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/21' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/27' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/31' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/36' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/20' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/39' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/34' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/38' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/28' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/24' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/29' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/32' 2024-05-27T00:17:39.287 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/22' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/25' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/30' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/33' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/23' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/35' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/26' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './250/37' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './250' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/21' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/27' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/31' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/36' 2024-05-27T00:17:39.288 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/20' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/39' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/34' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/38' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/28' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/24' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/29' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/32' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/22' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/25' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/30' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/33' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/23' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/35' 2024-05-27T00:17:39.289 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/26' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './251/37' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './251' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/21' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/27' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/31' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/36' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/20' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/39' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/34' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/38' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/28' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/24' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/29' 2024-05-27T00:17:39.290 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/32' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/22' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/25' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/30' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/33' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/23' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/35' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/26' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './252/37' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './252' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/21' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/27' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/31' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/36' 2024-05-27T00:17:39.291 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/20' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/39' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/34' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/38' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/28' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/24' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/29' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/32' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/22' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/25' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/30' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/33' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/23' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/35' 2024-05-27T00:17:39.292 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/26' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './253/37' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './253' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/21' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/27' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/31' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/36' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/20' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/39' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/34' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/38' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/28' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/24' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/29' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/32' 2024-05-27T00:17:39.293 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/22' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/25' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/30' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/33' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/23' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/35' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/26' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './254/37' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './254' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/21' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/27' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/31' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/36' 2024-05-27T00:17:39.294 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/20' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/39' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/34' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/38' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/28' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/24' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/29' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/32' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/22' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/25' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/30' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/33' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/23' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/35' 2024-05-27T00:17:39.295 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/26' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed './255/37' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './255' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/21' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/27' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/31' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/36' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/20' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/39' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/34' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/38' 2024-05-27T00:17:39.296 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/28' 2024-05-27T00:17:39.297 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/24' 2024-05-27T00:17:39.297 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/29' 2024-05-27T00:17:39.297 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/32' 2024-05-27T00:17:39.297 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/22' 2024-05-27T00:17:39.297 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/25' 2024-05-27T00:17:39.297 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/30' 2024-05-27T00:17:39.754 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/33' 2024-05-27T00:17:39.754 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/23' 2024-05-27T00:17:39.754 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/35' 2024-05-27T00:17:39.754 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/26' 2024-05-27T00:17:39.754 INFO:tasks.workunit.client.0.smithi098.stdout:removed './256/37' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './256' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/21' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/27' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/31' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/36' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/20' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/39' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/34' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/38' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/28' 2024-05-27T00:17:39.755 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/24' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/29' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/32' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/22' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/25' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/30' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/33' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/23' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/35' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/26' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './257/37' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './257' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/21' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/27' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/31' 2024-05-27T00:17:39.756 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/36' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/20' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/39' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/34' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/38' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/28' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/24' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/29' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/32' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/22' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/25' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/30' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/33' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/23' 2024-05-27T00:17:39.757 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/35' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/26' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './258/37' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './258' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/21' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/27' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/31' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/36' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/20' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/39' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/34' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/38' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/28' 2024-05-27T00:17:39.758 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/24' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/29' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/32' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/22' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/25' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/30' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/33' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/23' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/35' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/26' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed './259/37' 2024-05-27T00:17:39.759 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './259' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/21' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/27' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/31' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/36' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/20' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/39' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/34' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/38' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/28' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/24' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/29' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/32' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/22' 2024-05-27T00:17:39.760 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/25' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/30' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/33' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/23' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/35' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/26' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './260/37' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './260' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/21' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/27' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/31' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/36' 2024-05-27T00:17:39.761 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/20' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/39' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/34' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/38' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/28' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/24' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/29' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/32' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/22' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/25' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/30' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/33' 2024-05-27T00:17:39.762 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/23' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/35' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/26' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './261/37' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './261' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/21' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/27' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/31' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/36' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/20' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/39' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/34' 2024-05-27T00:17:39.763 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/38' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/28' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/24' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/29' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/32' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/22' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/25' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/30' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/33' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/23' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/35' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/26' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './262/37' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './262' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/21' 2024-05-27T00:17:39.764 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/27' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/31' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/36' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/20' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/39' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/34' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/38' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/28' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/24' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/29' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/32' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/22' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/25' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/30' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/33' 2024-05-27T00:17:39.765 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/23' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/35' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/26' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './263/37' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './263' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/21' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/27' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/31' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/36' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/20' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/39' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/34' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/38' 2024-05-27T00:17:39.766 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/28' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/24' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/29' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/32' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/22' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/25' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/30' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/33' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/23' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/35' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/26' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './264/37' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './264' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/21' 2024-05-27T00:17:39.767 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/27' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/31' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/36' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/20' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/39' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/34' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/38' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/28' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/24' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/29' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/32' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/22' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/25' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/30' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/33' 2024-05-27T00:17:39.768 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/23' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/35' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/26' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './265/37' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './265' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/21' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/27' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/31' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/36' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/20' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/39' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/34' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/38' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/28' 2024-05-27T00:17:39.769 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/24' 2024-05-27T00:17:39.770 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/29' 2024-05-27T00:17:39.770 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/32' 2024-05-27T00:17:39.770 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/22' 2024-05-27T00:17:39.770 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/25' 2024-05-27T00:17:39.770 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/30' 2024-05-27T00:17:39.770 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/33' 2024-05-27T00:17:39.770 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/23' 2024-05-27T00:17:40.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/35' 2024-05-27T00:17:40.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/26' 2024-05-27T00:17:40.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './266/37' 2024-05-27T00:17:40.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './266' 2024-05-27T00:17:40.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/21' 2024-05-27T00:17:40.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/27' 2024-05-27T00:17:40.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/31' 2024-05-27T00:17:40.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/36' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/20' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/39' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/34' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/38' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/28' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/24' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/29' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/32' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/22' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/25' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/30' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/33' 2024-05-27T00:17:40.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/23' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/35' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/26' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './267/37' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './267' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/21' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/27' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/31' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/36' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/20' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/39' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/34' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/38' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/28' 2024-05-27T00:17:40.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/24' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/29' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/32' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/22' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/25' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/30' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/33' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/23' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/35' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/26' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './268/37' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './268' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/21' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/27' 2024-05-27T00:17:40.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/31' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/36' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/20' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/39' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/34' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/38' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/28' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/24' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/29' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/32' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/22' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/25' 2024-05-27T00:17:40.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/30' 2024-05-27T00:17:40.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/33' 2024-05-27T00:17:40.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/23' 2024-05-27T00:17:40.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/35' 2024-05-27T00:17:40.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/26' 2024-05-27T00:17:40.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './269/37' 2024-05-27T00:17:40.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './269' 2024-05-27T00:17:40.219 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/21' 2024-05-27T00:17:40.219 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/27' 2024-05-27T00:17:40.219 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/31' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/36' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/20' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/39' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/34' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/38' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/28' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/24' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/29' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/32' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/22' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/25' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/30' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/33' 2024-05-27T00:17:40.220 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/23' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/35' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/26' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './270/37' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './270' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/21' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/27' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/31' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/36' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/20' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/39' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/34' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/38' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/28' 2024-05-27T00:17:40.221 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/24' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/29' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/32' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/22' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/25' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/30' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/33' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/23' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/35' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/26' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './271/37' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './271' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/21' 2024-05-27T00:17:40.222 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/27' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/31' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/36' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/20' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/39' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/34' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/38' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/28' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/24' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/29' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/32' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/22' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/25' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/30' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/33' 2024-05-27T00:17:40.223 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/23' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/35' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/26' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './272/37' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './272' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/21' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/27' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/31' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/36' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/20' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/39' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/34' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/38' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/28' 2024-05-27T00:17:40.224 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/24' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/29' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/32' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/22' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/25' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/30' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/33' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/23' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/35' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/26' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './273/37' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './273' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/21' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/27' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/31' 2024-05-27T00:17:40.225 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/36' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/20' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/39' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/34' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/38' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/28' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/24' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/29' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/32' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/22' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/25' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/30' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/33' 2024-05-27T00:17:40.226 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/23' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/35' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/26' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './274/37' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './274' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/21' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/27' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/31' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/36' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/20' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/39' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/34' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/38' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/28' 2024-05-27T00:17:40.227 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/24' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/29' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/32' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/22' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/25' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/30' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/33' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/23' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/35' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/26' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './275/37' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './275' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/21' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/27' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/31' 2024-05-27T00:17:40.228 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/36' 2024-05-27T00:17:40.229 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/20' 2024-05-27T00:17:40.229 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/39' 2024-05-27T00:17:40.229 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/34' 2024-05-27T00:17:40.229 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/38' 2024-05-27T00:17:40.229 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/28' 2024-05-27T00:17:40.229 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/24' 2024-05-27T00:17:40.230 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/29' 2024-05-27T00:17:40.230 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/32' 2024-05-27T00:17:40.230 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/22' 2024-05-27T00:17:40.230 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/25' 2024-05-27T00:17:40.230 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/30' 2024-05-27T00:17:40.230 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/33' 2024-05-27T00:17:40.230 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/23' 2024-05-27T00:17:40.230 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/35' 2024-05-27T00:17:40.230 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/26' 2024-05-27T00:17:40.617 INFO:tasks.workunit.client.0.smithi098.stdout:removed './276/37' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './276' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/21' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/27' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/31' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/36' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/20' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/39' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/34' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/38' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/28' 2024-05-27T00:17:40.618 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/24' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/29' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/32' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/22' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/25' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/30' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/33' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/23' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/35' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/26' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './277/37' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './277' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/21' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/27' 2024-05-27T00:17:40.619 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/31' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/36' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/20' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/39' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/34' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/38' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/28' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/24' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/29' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/32' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/22' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/25' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/30' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/33' 2024-05-27T00:17:40.620 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/23' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/35' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/26' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './278/37' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './278' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/21' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/27' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/31' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/36' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/20' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/39' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/34' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/38' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/28' 2024-05-27T00:17:40.621 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/24' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/29' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/32' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/22' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/25' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/30' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/33' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/23' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/35' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/26' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './279/37' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './279' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/21' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/27' 2024-05-27T00:17:40.622 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/31' 2024-05-27T00:17:40.623 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/36' 2024-05-27T00:17:40.623 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/20' 2024-05-27T00:17:40.623 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/39' 2024-05-27T00:17:40.623 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/34' 2024-05-27T00:17:40.623 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/38' 2024-05-27T00:17:40.623 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/28' 2024-05-27T00:17:40.623 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/24' 2024-05-27T00:17:40.623 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/29' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/32' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/22' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/25' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/30' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/33' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/23' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/35' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/26' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './280/37' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './280' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/21' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/27' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/31' 2024-05-27T00:17:40.624 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/36' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/20' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/39' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/34' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/38' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/28' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/24' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/29' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/32' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/22' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/25' 2024-05-27T00:17:40.625 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/30' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/33' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/23' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/35' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/26' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './281/37' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './281' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/21' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/27' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/31' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/36' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/20' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/39' 2024-05-27T00:17:40.626 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/34' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/38' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/28' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/24' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/29' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/32' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/22' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/25' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/30' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/33' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/23' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/35' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/26' 2024-05-27T00:17:40.627 INFO:tasks.workunit.client.0.smithi098.stdout:removed './282/37' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './282' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/21' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/27' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/31' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/36' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/20' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/39' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/34' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/38' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/28' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/24' 2024-05-27T00:17:40.628 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/29' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/32' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/22' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/25' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/30' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/33' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/23' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/35' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/26' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './283/37' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './283' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/21' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/27' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/31' 2024-05-27T00:17:40.629 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/36' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/20' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/39' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/34' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/38' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/28' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/24' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/29' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/32' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/22' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/25' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/30' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/33' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/23' 2024-05-27T00:17:40.630 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/35' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/26' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './284/37' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './284' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/21' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/27' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/31' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/36' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/20' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/39' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/34' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/38' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/28' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/24' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/29' 2024-05-27T00:17:40.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/32' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/22' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/25' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/30' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/33' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/23' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/35' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/26' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './285/37' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './285' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/21' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/27' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/31' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/36' 2024-05-27T00:17:40.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/20' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/39' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/34' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/38' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/28' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/24' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/29' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/32' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/22' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/25' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/30' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/33' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/23' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/35' 2024-05-27T00:17:40.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/26' 2024-05-27T00:17:40.634 INFO:tasks.workunit.client.0.smithi098.stdout:removed './286/37' 2024-05-27T00:17:40.634 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './286' 2024-05-27T00:17:41.203 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/21' 2024-05-27T00:17:41.203 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/27' 2024-05-27T00:17:41.203 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/31' 2024-05-27T00:17:41.203 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/36' 2024-05-27T00:17:41.203 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/20' 2024-05-27T00:17:41.203 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/39' 2024-05-27T00:17:41.203 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/34' 2024-05-27T00:17:41.203 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/38' 2024-05-27T00:17:41.203 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/28' 2024-05-27T00:17:41.203 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/24' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/29' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/32' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/22' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/25' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/30' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/33' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/23' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/35' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/26' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './287/37' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './287' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/21' 2024-05-27T00:17:41.204 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/27' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/31' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/36' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/20' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/39' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/34' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/38' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/28' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/24' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/29' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/32' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/22' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/25' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/30' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/33' 2024-05-27T00:17:41.205 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/23' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/35' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/26' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './288/37' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './288' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/21' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/27' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/31' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/36' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/20' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/39' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/34' 2024-05-27T00:17:41.206 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/38' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/28' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/24' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/29' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/32' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/22' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/25' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/30' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/33' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/23' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/35' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/26' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './289/37' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './289' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/21' 2024-05-27T00:17:41.207 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/27' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/31' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/36' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/20' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/39' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/34' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/38' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/28' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/24' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/29' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/32' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/22' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/25' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/30' 2024-05-27T00:17:41.208 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/33' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/23' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/35' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/26' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './290/37' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './290' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/21' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/27' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/31' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/36' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/20' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/39' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/34' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/38' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/28' 2024-05-27T00:17:41.209 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/24' 2024-05-27T00:17:41.210 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/29' 2024-05-27T00:17:41.210 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/32' 2024-05-27T00:17:41.210 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/22' 2024-05-27T00:17:41.210 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/25' 2024-05-27T00:17:41.210 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/30' 2024-05-27T00:17:41.210 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/33' 2024-05-27T00:17:41.210 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/23' 2024-05-27T00:17:41.210 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/35' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/26' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './291/37' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './291' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/21' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/27' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/31' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/36' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/20' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/39' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/34' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/38' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/28' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/24' 2024-05-27T00:17:41.211 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/29' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/32' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/22' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/25' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/30' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/33' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/23' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/35' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/26' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './292/37' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './292' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/21' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/27' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/31' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/36' 2024-05-27T00:17:41.212 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/20' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/39' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/34' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/38' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/28' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/24' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/29' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/32' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/22' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/25' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/30' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/33' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/23' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/35' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/26' 2024-05-27T00:17:41.213 INFO:tasks.workunit.client.0.smithi098.stdout:removed './293/37' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './293' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/21' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/27' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/31' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/36' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/20' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/39' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/34' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/38' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/28' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/24' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/29' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/32' 2024-05-27T00:17:41.214 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/22' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/25' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/30' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/33' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/23' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/35' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/26' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './294/37' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './294' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/21' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/27' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/31' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/36' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/20' 2024-05-27T00:17:41.215 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/39' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/34' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/38' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/28' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/24' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/29' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/32' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/22' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/25' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/30' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/33' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/23' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/35' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/26' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed './295/37' 2024-05-27T00:17:41.216 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './295' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/21' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/27' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/31' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/36' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/20' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/39' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/34' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/38' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/28' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/24' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/29' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/32' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/22' 2024-05-27T00:17:41.217 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/25' 2024-05-27T00:17:41.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/30' 2024-05-27T00:17:41.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/33' 2024-05-27T00:17:41.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/23' 2024-05-27T00:17:41.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/35' 2024-05-27T00:17:41.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/26' 2024-05-27T00:17:41.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './296/37' 2024-05-27T00:17:41.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './296' 2024-05-27T00:17:41.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/21' 2024-05-27T00:17:41.218 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/27' 2024-05-27T00:17:41.462 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/31' 2024-05-27T00:17:41.462 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/36' 2024-05-27T00:17:41.462 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/20' 2024-05-27T00:17:41.462 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/39' 2024-05-27T00:17:41.462 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/34' 2024-05-27T00:17:41.462 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/38' 2024-05-27T00:17:41.462 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/28' 2024-05-27T00:17:41.462 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/24' 2024-05-27T00:17:41.462 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/29' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/32' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/22' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/25' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/30' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/33' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/23' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/35' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/26' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './297/37' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './297' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/21' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/27' 2024-05-27T00:17:41.463 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/31' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/36' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/20' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/39' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/34' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/38' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/28' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/24' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/29' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/32' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/22' 2024-05-27T00:17:41.464 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/25' 2024-05-27T00:17:41.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/30' 2024-05-27T00:17:41.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/33' 2024-05-27T00:17:41.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/23' 2024-05-27T00:17:41.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/35' 2024-05-27T00:17:41.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/26' 2024-05-27T00:17:41.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './298/37' 2024-05-27T00:17:41.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './298' 2024-05-27T00:17:41.465 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/21' 2024-05-27T00:17:41.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/27' 2024-05-27T00:17:41.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/31' 2024-05-27T00:17:41.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/36' 2024-05-27T00:17:41.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/20' 2024-05-27T00:17:41.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/39' 2024-05-27T00:17:41.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/34' 2024-05-27T00:17:41.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/38' 2024-05-27T00:17:41.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/28' 2024-05-27T00:17:41.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/24' 2024-05-27T00:17:41.466 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/29' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/32' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/22' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/25' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/30' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/33' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/23' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/35' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/26' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './299/37' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './299' 2024-05-27T00:17:41.467 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/19' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/15' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/20' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/12' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/10' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/11' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/16' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/14' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/18' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/13' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/166/17' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/166' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/19' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/15' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/20' 2024-05-27T00:17:41.468 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/12' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/10' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/11' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/16' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/14' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/18' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/13' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/147/17' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/147' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/19' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/15' 2024-05-27T00:17:41.469 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/20' 2024-05-27T00:17:41.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/12' 2024-05-27T00:17:41.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/10' 2024-05-27T00:17:41.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/11' 2024-05-27T00:17:41.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/16' 2024-05-27T00:17:41.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/14' 2024-05-27T00:17:41.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/18' 2024-05-27T00:17:41.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/13' 2024-05-27T00:17:41.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/151/17' 2024-05-27T00:17:41.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/151' 2024-05-27T00:17:41.470 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/19' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/15' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/20' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/12' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/10' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/11' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/16' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/14' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/18' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/13' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/170/17' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/170' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/19' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/15' 2024-05-27T00:17:41.471 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/20' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/12' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/10' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/11' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/16' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/14' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/18' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/13' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/167/17' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/167' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/19' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/15' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/20' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/12' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/10' 2024-05-27T00:17:41.472 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/11' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/16' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/14' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/18' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/13' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/140/17' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/140' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/19' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/15' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/20' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/12' 2024-05-27T00:17:41.473 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/10' 2024-05-27T00:17:41.662 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/11' 2024-05-27T00:17:41.662 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/16' 2024-05-27T00:17:41.662 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/14' 2024-05-27T00:17:41.662 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/18' 2024-05-27T00:17:41.662 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/13' 2024-05-27T00:17:41.662 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/142/17' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/142' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/19' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/15' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/20' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/12' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/10' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/11' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/16' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/14' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/18' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/13' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/150/17' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/150' 2024-05-27T00:17:41.663 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/19' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/15' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/20' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/12' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/10' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/11' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/16' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/14' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/18' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/13' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/105/17' 2024-05-27T00:17:41.664 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/105' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/19' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/15' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/20' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/12' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/10' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/11' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/16' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/14' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/18' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/13' 2024-05-27T00:17:41.665 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/149/17' 2024-05-27T00:17:41.666 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/149' 2024-05-27T00:17:41.666 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/19' 2024-05-27T00:17:41.666 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/15' 2024-05-27T00:17:41.666 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/20' 2024-05-27T00:17:41.666 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/12' 2024-05-27T00:17:41.666 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/10' 2024-05-27T00:17:41.666 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/11' 2024-05-27T00:17:41.666 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/16' 2024-05-27T00:17:41.666 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/14' 2024-05-27T00:17:41.666 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/18' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/13' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/109/17' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/109' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/19' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/15' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/20' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/12' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/10' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/11' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/16' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/14' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/18' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/13' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/158/17' 2024-05-27T00:17:41.667 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/158' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/19' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/15' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/20' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/12' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/10' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/11' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/16' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/14' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/18' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/13' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/106/17' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/106' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/19' 2024-05-27T00:17:41.668 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/15' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/20' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/12' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/10' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/11' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/16' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/14' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/18' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/13' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/189/17' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/189' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/19' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/15' 2024-05-27T00:17:41.669 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/20' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/12' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/10' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/11' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/16' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/14' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/18' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/13' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/133/17' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/133' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/19' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/15' 2024-05-27T00:17:41.670 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/20' 2024-05-27T00:17:41.671 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/12' 2024-05-27T00:17:41.671 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/10' 2024-05-27T00:17:41.671 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/11' 2024-05-27T00:17:41.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/16' 2024-05-27T00:17:41.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/14' 2024-05-27T00:17:41.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/18' 2024-05-27T00:17:41.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/13' 2024-05-27T00:17:41.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/173/17' 2024-05-27T00:17:41.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/173' 2024-05-27T00:17:41.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/19' 2024-05-27T00:17:41.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/15' 2024-05-27T00:17:41.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/20' 2024-05-27T00:17:41.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/12' 2024-05-27T00:17:41.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/10' 2024-05-27T00:17:41.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/11' 2024-05-27T00:17:41.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/16' 2024-05-27T00:17:41.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/14' 2024-05-27T00:17:41.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/18' 2024-05-27T00:17:41.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/13' 2024-05-27T00:17:41.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/174/17' 2024-05-27T00:17:41.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/174' 2024-05-27T00:17:41.962 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/19' 2024-05-27T00:17:41.962 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/15' 2024-05-27T00:17:41.962 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/20' 2024-05-27T00:17:41.962 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/12' 2024-05-27T00:17:41.962 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/10' 2024-05-27T00:17:41.962 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/11' 2024-05-27T00:17:41.963 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/16' 2024-05-27T00:17:41.963 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/14' 2024-05-27T00:17:41.963 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/18' 2024-05-27T00:17:41.963 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/13' 2024-05-27T00:17:41.963 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/155/17' 2024-05-27T00:17:41.963 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/155' 2024-05-27T00:17:41.964 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/19' 2024-05-27T00:17:41.964 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/15' 2024-05-27T00:17:41.964 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/20' 2024-05-27T00:17:41.964 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/12' 2024-05-27T00:17:41.964 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/10' 2024-05-27T00:17:41.964 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/11' 2024-05-27T00:17:41.965 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/16' 2024-05-27T00:17:41.965 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/14' 2024-05-27T00:17:41.965 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/18' 2024-05-27T00:17:41.965 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/13' 2024-05-27T00:17:41.965 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/114/17' 2024-05-27T00:17:41.965 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/114' 2024-05-27T00:17:41.966 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/19' 2024-05-27T00:17:41.966 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/15' 2024-05-27T00:17:41.966 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/20' 2024-05-27T00:17:41.966 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/12' 2024-05-27T00:17:41.967 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/10' 2024-05-27T00:17:41.967 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/11' 2024-05-27T00:17:41.967 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/16' 2024-05-27T00:17:41.967 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/14' 2024-05-27T00:17:41.967 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/18' 2024-05-27T00:17:41.968 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/13' 2024-05-27T00:17:41.968 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/159/17' 2024-05-27T00:17:41.968 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/159' 2024-05-27T00:17:41.968 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/19' 2024-05-27T00:17:41.968 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/15' 2024-05-27T00:17:41.968 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/20' 2024-05-27T00:17:41.969 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/12' 2024-05-27T00:17:41.969 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/10' 2024-05-27T00:17:41.969 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/11' 2024-05-27T00:17:41.969 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/16' 2024-05-27T00:17:41.969 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/14' 2024-05-27T00:17:41.969 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/18' 2024-05-27T00:17:41.970 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/13' 2024-05-27T00:17:41.970 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/187/17' 2024-05-27T00:17:41.970 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/187' 2024-05-27T00:17:41.970 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/19' 2024-05-27T00:17:41.970 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/15' 2024-05-27T00:17:41.970 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/20' 2024-05-27T00:17:41.971 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/12' 2024-05-27T00:17:41.971 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/10' 2024-05-27T00:17:41.971 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/11' 2024-05-27T00:17:41.971 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/16' 2024-05-27T00:17:41.971 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/14' 2024-05-27T00:17:41.971 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/18' 2024-05-27T00:17:41.972 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/13' 2024-05-27T00:17:41.972 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/185/17' 2024-05-27T00:17:41.972 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/185' 2024-05-27T00:17:41.972 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/19' 2024-05-27T00:17:41.972 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/15' 2024-05-27T00:17:41.973 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/20' 2024-05-27T00:17:41.973 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/12' 2024-05-27T00:17:41.973 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/10' 2024-05-27T00:17:41.973 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/11' 2024-05-27T00:17:41.973 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/16' 2024-05-27T00:17:41.973 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/14' 2024-05-27T00:17:41.974 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/18' 2024-05-27T00:17:41.974 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/13' 2024-05-27T00:17:41.974 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/120/17' 2024-05-27T00:17:41.974 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/120' 2024-05-27T00:17:41.974 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/19' 2024-05-27T00:17:41.974 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/15' 2024-05-27T00:17:41.975 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/20' 2024-05-27T00:17:41.975 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/12' 2024-05-27T00:17:41.975 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/10' 2024-05-27T00:17:41.975 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/11' 2024-05-27T00:17:41.975 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/16' 2024-05-27T00:17:41.976 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/14' 2024-05-27T00:17:41.976 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/18' 2024-05-27T00:17:41.976 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/13' 2024-05-27T00:17:41.976 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/110/17' 2024-05-27T00:17:41.976 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/110' 2024-05-27T00:17:41.976 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/19' 2024-05-27T00:17:41.977 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/15' 2024-05-27T00:17:41.977 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/20' 2024-05-27T00:17:41.977 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/12' 2024-05-27T00:17:41.977 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/10' 2024-05-27T00:17:41.977 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/11' 2024-05-27T00:17:41.977 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/16' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/14' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/18' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/13' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/137/17' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/137' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/19' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/15' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/20' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/12' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/10' 2024-05-27T00:17:42.186 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/11' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/16' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/14' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/18' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/13' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/198/17' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/198' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/19' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/15' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/20' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/12' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/10' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/11' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/16' 2024-05-27T00:17:42.187 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/14' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/18' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/13' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/136/17' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/136' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/19' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/15' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/20' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/12' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/10' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/11' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/16' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/14' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/18' 2024-05-27T00:17:42.188 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/13' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/125/17' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/125' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/19' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/15' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/20' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/12' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/10' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/11' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/16' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/14' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/18' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/13' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/194/17' 2024-05-27T00:17:42.189 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/194' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/19' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/15' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/20' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/12' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/10' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/11' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/16' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/14' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/18' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/13' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/119/17' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/119' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/19' 2024-05-27T00:17:42.190 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/15' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/20' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/12' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/10' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/11' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/16' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/14' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/18' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/13' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/157/17' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/157' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/19' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/15' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/20' 2024-05-27T00:17:42.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/12' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/10' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/11' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/16' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/14' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/18' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/13' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/172/17' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/172' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/19' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/15' 2024-05-27T00:17:42.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/20' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/12' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/10' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/11' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/16' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/14' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/18' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/13' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/186/17' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/186' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/19' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/15' 2024-05-27T00:17:42.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/20' 2024-05-27T00:17:42.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/12' 2024-05-27T00:17:42.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/10' 2024-05-27T00:17:42.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/11' 2024-05-27T00:17:42.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/16' 2024-05-27T00:17:42.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/14' 2024-05-27T00:17:42.423 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/18' 2024-05-27T00:17:42.424 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/13' 2024-05-27T00:17:42.424 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/190/17' 2024-05-27T00:17:42.424 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/190' 2024-05-27T00:17:42.424 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/19' 2024-05-27T00:17:42.424 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/15' 2024-05-27T00:17:42.425 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/20' 2024-05-27T00:17:42.425 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/12' 2024-05-27T00:17:42.425 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/10' 2024-05-27T00:17:42.427 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/11' 2024-05-27T00:17:42.427 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/16' 2024-05-27T00:17:42.428 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/14' 2024-05-27T00:17:42.428 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/18' 2024-05-27T00:17:42.428 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/13' 2024-05-27T00:17:42.428 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/104/17' 2024-05-27T00:17:42.428 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/104' 2024-05-27T00:17:42.428 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/19' 2024-05-27T00:17:42.429 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/15' 2024-05-27T00:17:42.429 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/20' 2024-05-27T00:17:42.429 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/12' 2024-05-27T00:17:42.429 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/10' 2024-05-27T00:17:42.429 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/11' 2024-05-27T00:17:42.429 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/16' 2024-05-27T00:17:42.429 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/14' 2024-05-27T00:17:42.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/18' 2024-05-27T00:17:42.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/13' 2024-05-27T00:17:42.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/164/17' 2024-05-27T00:17:42.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/164' 2024-05-27T00:17:42.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/19' 2024-05-27T00:17:42.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/15' 2024-05-27T00:17:42.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/20' 2024-05-27T00:17:42.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/12' 2024-05-27T00:17:42.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/10' 2024-05-27T00:17:42.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/11' 2024-05-27T00:17:42.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/16' 2024-05-27T00:17:42.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/14' 2024-05-27T00:17:42.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/18' 2024-05-27T00:17:42.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/13' 2024-05-27T00:17:42.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/112/17' 2024-05-27T00:17:42.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/112' 2024-05-27T00:17:42.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/19' 2024-05-27T00:17:42.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/15' 2024-05-27T00:17:42.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/20' 2024-05-27T00:17:42.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/12' 2024-05-27T00:17:42.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/10' 2024-05-27T00:17:42.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/11' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/16' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/14' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/18' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/13' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/100/17' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/100' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/19' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/15' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/20' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/12' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/10' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/11' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/16' 2024-05-27T00:17:42.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/14' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/18' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/13' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/134/17' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/134' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/19' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/15' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/20' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/12' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/10' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/11' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/16' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/14' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/18' 2024-05-27T00:17:42.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/13' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/115/17' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/115' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/19' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/15' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/20' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/12' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/10' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/11' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/16' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/14' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/18' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/13' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/116/17' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/116' 2024-05-27T00:17:42.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/19' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/15' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/20' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/12' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/10' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/11' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/16' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/14' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/18' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/13' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/195/17' 2024-05-27T00:17:42.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/195' 2024-05-27T00:17:42.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/19' 2024-05-27T00:17:42.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/15' 2024-05-27T00:17:42.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/20' 2024-05-27T00:17:42.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/12' 2024-05-27T00:17:42.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/10' 2024-05-27T00:17:42.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/11' 2024-05-27T00:17:42.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/16' 2024-05-27T00:17:42.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/14' 2024-05-27T00:17:42.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/18' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/13' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/138/17' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/138' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/19' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/15' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/20' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/12' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/10' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/11' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/16' 2024-05-27T00:17:42.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/14' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/18' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/13' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/135/17' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/135' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/19' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/15' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/20' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/12' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/10' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/11' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/16' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/14' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/18' 2024-05-27T00:17:42.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/13' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/117/17' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/117' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/19' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/15' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/20' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/12' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/10' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/11' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/16' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/14' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/18' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/13' 2024-05-27T00:17:42.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/143/17' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/143' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/19' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/15' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/20' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/12' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/10' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/11' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/16' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/14' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/18' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/13' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/180/17' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/180' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/19' 2024-05-27T00:17:42.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/15' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/20' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/12' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/10' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/11' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/16' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/14' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/18' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/13' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/126/17' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/126' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/19' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/15' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/20' 2024-05-27T00:17:42.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/12' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/10' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/11' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/16' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/14' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/18' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/13' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/161/17' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/161' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/19' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/15' 2024-05-27T00:17:42.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/20' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/12' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/10' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/11' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/16' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/14' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/18' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/13' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/176/17' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/176' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/19' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/15' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/20' 2024-05-27T00:17:42.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/12' 2024-05-27T00:17:42.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/10' 2024-05-27T00:17:42.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/11' 2024-05-27T00:17:42.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/16' 2024-05-27T00:17:42.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/14' 2024-05-27T00:17:42.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/18' 2024-05-27T00:17:42.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/13' 2024-05-27T00:17:42.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/111/17' 2024-05-27T00:17:42.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/111' 2024-05-27T00:17:42.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/19' 2024-05-27T00:17:42.644 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/15' 2024-05-27T00:17:42.644 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/20' 2024-05-27T00:17:42.644 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/12' 2024-05-27T00:17:42.644 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/10' 2024-05-27T00:17:42.644 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/11' 2024-05-27T00:17:42.644 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/16' 2024-05-27T00:17:42.644 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/14' 2024-05-27T00:17:42.644 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/18' 2024-05-27T00:17:42.644 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/13' 2024-05-27T00:17:42.938 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/191/17' 2024-05-27T00:17:42.938 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/191' 2024-05-27T00:17:42.938 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/19' 2024-05-27T00:17:42.938 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/15' 2024-05-27T00:17:42.938 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/20' 2024-05-27T00:17:42.938 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/12' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/10' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/11' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/16' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/14' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/18' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/13' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/184/17' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/184' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/19' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/15' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/20' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/12' 2024-05-27T00:17:42.939 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/10' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/11' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/16' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/14' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/18' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/13' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/192/17' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/192' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/19' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/15' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/20' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/12' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/10' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/11' 2024-05-27T00:17:42.940 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/16' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/14' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/18' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/13' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/103/17' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/103' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/19' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/15' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/20' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/12' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/10' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/11' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/16' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/14' 2024-05-27T00:17:42.941 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/18' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/13' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/128/17' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/128' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/19' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/15' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/20' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/12' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/10' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/11' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/16' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/14' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/18' 2024-05-27T00:17:42.942 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/13' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/144/17' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/144' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/19' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/15' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/20' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/12' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/10' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/11' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/16' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/14' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/18' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/13' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/146/17' 2024-05-27T00:17:42.943 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/146' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/19' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/15' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/20' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/12' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/10' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/11' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/16' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/14' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/18' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/13' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/141/17' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/141' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/19' 2024-05-27T00:17:42.944 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/15' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/20' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/12' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/10' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/11' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/16' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/14' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/18' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/13' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/182/17' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/182' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/19' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/15' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/20' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/12' 2024-05-27T00:17:42.945 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/10' 2024-05-27T00:17:42.946 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/11' 2024-05-27T00:17:42.946 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/16' 2024-05-27T00:17:42.946 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/14' 2024-05-27T00:17:42.946 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/18' 2024-05-27T00:17:42.946 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/13' 2024-05-27T00:17:42.946 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/107/17' 2024-05-27T00:17:43.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/107' 2024-05-27T00:17:43.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/19' 2024-05-27T00:17:43.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/15' 2024-05-27T00:17:43.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/20' 2024-05-27T00:17:43.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/12' 2024-05-27T00:17:43.191 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/10' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/11' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/16' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/14' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/18' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/13' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/178/17' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/178' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/19' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/15' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/20' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/12' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/10' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/11' 2024-05-27T00:17:43.192 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/16' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/14' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/18' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/13' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/156/17' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/156' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/19' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/15' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/20' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/12' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/10' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/11' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/16' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/14' 2024-05-27T00:17:43.193 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/18' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/13' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/131/17' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/131' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/19' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/15' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/20' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/12' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/10' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/11' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/16' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/14' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/18' 2024-05-27T00:17:43.194 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/13' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/123/17' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/123' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/19' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/15' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/20' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/12' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/10' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/11' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/16' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/14' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/18' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/13' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/160/17' 2024-05-27T00:17:43.195 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/160' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/19' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/15' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/20' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/12' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/10' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/11' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/16' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/14' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/18' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/13' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/165/17' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/165' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/19' 2024-05-27T00:17:43.196 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/15' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/20' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/12' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/10' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/11' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/16' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/14' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/18' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/13' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/101/17' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/101' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/19' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/15' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/20' 2024-05-27T00:17:43.197 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/12' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/10' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/11' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/16' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/14' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/18' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/13' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/148/17' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/148' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/19' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/15' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/20' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/12' 2024-05-27T00:17:43.198 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/10' 2024-05-27T00:17:43.199 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/11' 2024-05-27T00:17:43.199 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/16' 2024-05-27T00:17:43.199 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/14' 2024-05-27T00:17:43.199 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/18' 2024-05-27T00:17:43.199 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/13' 2024-05-27T00:17:43.199 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/168/17' 2024-05-27T00:17:43.199 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/168' 2024-05-27T00:17:43.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/19' 2024-05-27T00:17:43.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/15' 2024-05-27T00:17:43.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/20' 2024-05-27T00:17:43.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/12' 2024-05-27T00:17:43.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/10' 2024-05-27T00:17:43.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/11' 2024-05-27T00:17:43.430 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/16' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/14' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/18' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/13' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/179/17' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/179' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/19' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/15' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/20' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/12' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/10' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/11' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/16' 2024-05-27T00:17:43.431 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/14' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/18' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/13' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/132/17' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/132' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/19' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/15' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/20' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/12' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/10' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/11' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/16' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/14' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/18' 2024-05-27T00:17:43.432 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/13' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/199/17' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/199' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/19' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/15' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/20' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/12' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/10' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/11' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/16' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/14' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/18' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/13' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/163/17' 2024-05-27T00:17:43.433 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/163' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/19' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/15' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/20' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/12' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/10' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/11' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/16' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/14' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/18' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/13' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/153/17' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/153' 2024-05-27T00:17:43.434 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/19' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/15' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/20' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/12' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/10' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/11' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/16' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/14' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/18' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/13' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/122/17' 2024-05-27T00:17:43.435 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/122' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/19' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/15' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/20' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/12' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/10' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/11' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/16' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/14' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/18' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/13' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/162/17' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/162' 2024-05-27T00:17:43.436 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/19' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/15' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/20' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/12' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/10' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/11' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/16' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/14' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/18' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/13' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/197/17' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/197' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/19' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/15' 2024-05-27T00:17:43.437 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/20' 2024-05-27T00:17:43.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/12' 2024-05-27T00:17:43.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/10' 2024-05-27T00:17:43.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/11' 2024-05-27T00:17:43.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/16' 2024-05-27T00:17:43.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/14' 2024-05-27T00:17:43.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/18' 2024-05-27T00:17:43.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/13' 2024-05-27T00:17:43.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/113/17' 2024-05-27T00:17:43.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/113' 2024-05-27T00:17:43.438 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/19' 2024-05-27T00:17:43.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/15' 2024-05-27T00:17:43.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/20' 2024-05-27T00:17:43.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/12' 2024-05-27T00:17:43.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/10' 2024-05-27T00:17:43.631 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/11' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/16' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/14' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/18' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/13' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/152/17' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/152' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/19' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/15' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/20' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/12' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/10' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/11' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/16' 2024-05-27T00:17:43.632 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/14' 2024-05-27T00:17:43.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/18' 2024-05-27T00:17:43.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/13' 2024-05-27T00:17:43.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/121/17' 2024-05-27T00:17:43.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/121' 2024-05-27T00:17:43.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/19' 2024-05-27T00:17:43.633 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/15' 2024-05-27T00:17:43.634 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/20' 2024-05-27T00:17:43.634 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/12' 2024-05-27T00:17:43.634 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/10' 2024-05-27T00:17:43.634 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/11' 2024-05-27T00:17:43.634 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/16' 2024-05-27T00:17:43.635 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/14' 2024-05-27T00:17:43.635 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/18' 2024-05-27T00:17:43.635 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/13' 2024-05-27T00:17:43.635 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/118/17' 2024-05-27T00:17:43.635 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/118' 2024-05-27T00:17:43.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/19' 2024-05-27T00:17:43.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/15' 2024-05-27T00:17:43.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/20' 2024-05-27T00:17:43.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/12' 2024-05-27T00:17:43.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/10' 2024-05-27T00:17:43.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/11' 2024-05-27T00:17:43.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/16' 2024-05-27T00:17:43.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/14' 2024-05-27T00:17:43.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/18' 2024-05-27T00:17:43.636 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/13' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/196/17' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/196' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/19' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/15' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/20' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/12' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/10' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/11' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/16' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/14' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/18' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/13' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/124/17' 2024-05-27T00:17:43.637 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/124' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/19' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/15' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/20' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/12' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/10' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/11' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/16' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/14' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/18' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/13' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/188/17' 2024-05-27T00:17:43.638 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/188' 2024-05-27T00:17:43.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/19' 2024-05-27T00:17:43.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/15' 2024-05-27T00:17:43.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/20' 2024-05-27T00:17:43.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/12' 2024-05-27T00:17:43.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/10' 2024-05-27T00:17:43.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/11' 2024-05-27T00:17:43.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/16' 2024-05-27T00:17:43.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/14' 2024-05-27T00:17:43.639 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/18' 2024-05-27T00:17:43.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/13' 2024-05-27T00:17:43.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/145/17' 2024-05-27T00:17:43.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/145' 2024-05-27T00:17:43.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/19' 2024-05-27T00:17:43.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/15' 2024-05-27T00:17:43.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/20' 2024-05-27T00:17:43.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/12' 2024-05-27T00:17:43.640 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/10' 2024-05-27T00:17:43.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/11' 2024-05-27T00:17:43.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/16' 2024-05-27T00:17:43.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/14' 2024-05-27T00:17:43.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/18' 2024-05-27T00:17:43.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/13' 2024-05-27T00:17:43.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/130/17' 2024-05-27T00:17:43.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/130' 2024-05-27T00:17:43.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/19' 2024-05-27T00:17:43.641 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/15' 2024-05-27T00:17:43.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/20' 2024-05-27T00:17:43.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/12' 2024-05-27T00:17:43.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/10' 2024-05-27T00:17:43.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/11' 2024-05-27T00:17:43.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/16' 2024-05-27T00:17:43.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/14' 2024-05-27T00:17:43.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/18' 2024-05-27T00:17:43.642 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/13' 2024-05-27T00:17:43.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/193/17' 2024-05-27T00:17:43.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/193' 2024-05-27T00:17:43.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/19' 2024-05-27T00:17:43.643 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/15' 2024-05-27T00:17:43.953 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/20' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/12' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/10' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/11' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/16' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/14' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/18' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/13' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/175/17' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/175' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/19' 2024-05-27T00:17:43.954 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/15' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/20' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/12' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/10' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/11' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/16' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/14' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/18' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/13' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/181/17' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/181' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/19' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/15' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/20' 2024-05-27T00:17:43.955 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/12' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/10' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/11' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/16' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/14' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/18' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/13' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/127/17' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/127' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/19' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/15' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/20' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/12' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/10' 2024-05-27T00:17:43.956 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/11' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/16' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/14' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/18' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/13' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/108/17' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/108' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/19' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/15' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/20' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/12' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/10' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/11' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/16' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/14' 2024-05-27T00:17:43.957 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/18' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/13' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/183/17' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/183' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/19' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/15' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/20' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/12' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/10' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/11' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/16' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/14' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/18' 2024-05-27T00:17:43.958 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/13' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/177/17' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/177' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/19' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/15' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/20' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/12' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/10' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/11' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/16' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/14' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/18' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/13' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/169/17' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/169' 2024-05-27T00:17:43.959 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/19' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/15' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/20' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/12' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/10' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/11' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/16' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/14' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/18' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/13' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/154/17' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/154' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/19' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/15' 2024-05-27T00:17:43.960 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/20' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/12' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/10' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/11' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/16' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/14' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/18' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/13' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/171/17' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/171' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/19' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/15' 2024-05-27T00:17:43.961 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/20' 2024-05-27T00:17:44.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:17:44.014 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:17:44.030 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/12' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/10' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/11' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/16' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/14' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/18' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/13' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/139/17' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/139' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/19' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/15' 2024-05-27T00:17:44.031 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/20' 2024-05-27T00:17:44.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/12' 2024-05-27T00:17:44.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/10' 2024-05-27T00:17:44.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/11' 2024-05-27T00:17:44.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/16' 2024-05-27T00:17:44.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/14' 2024-05-27T00:17:44.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/18' 2024-05-27T00:17:44.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/13' 2024-05-27T00:17:44.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/102/17' 2024-05-27T00:17:44.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/102' 2024-05-27T00:17:44.032 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/19' 2024-05-27T00:17:44.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/15' 2024-05-27T00:17:44.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/20' 2024-05-27T00:17:44.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/12' 2024-05-27T00:17:44.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/10' 2024-05-27T00:17:44.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/11' 2024-05-27T00:17:44.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/16' 2024-05-27T00:17:44.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/14' 2024-05-27T00:17:44.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/18' 2024-05-27T00:17:44.033 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/13' 2024-05-27T00:17:44.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed './copyofsnap1/test1/129/17' 2024-05-27T00:17:44.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1/129' 2024-05-27T00:17:44.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1/test1' 2024-05-27T00:17:44.034 INFO:tasks.workunit.client.0.smithi098.stdout:removed directory './copyofsnap1' 2024-05-27T00:17:44.034 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-05-27T00:17:44.035 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T00:17:44.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T00:17:44.103 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2024-05-27T00:17:44.103 DEBUG:teuthology.orchestra.run.smithi098:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-05-27T00:17:44.481 INFO:tasks.workunit.client.0.smithi098.stderr:+ set -e 2024-05-27T00:17:44.482 INFO:tasks.workunit.client.0.smithi098.stderr:+ touch foo 2024-05-27T00:17:44.484 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod +x foo 2024-05-27T00:17:44.485 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/s 2024-05-27T00:17:44.498 INFO:tasks.workunit.client.0.smithi098.stderr:+ find .snap/s/foo -executable 2024-05-27T00:17:44.498 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep foo 2024-05-27T00:17:44.501 INFO:tasks.workunit.client.0.smithi098.stdout:.snap/s/foo 2024-05-27T00:17:44.501 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir .snap/s 2024-05-27T00:17:44.508 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm foo 2024-05-27T00:17:44.510 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo OK 2024-05-27T00:17:44.511 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-05-27T00:17:44.511 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T00:17:44.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T00:17:44.535 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2024-05-27T00:17:44.536 DEBUG:teuthology.orchestra.run.smithi098:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-05-27T00:17:44.593 INFO:tasks.workunit.client.0.smithi098.stderr:+ set -e 2024-05-27T00:17:44.593 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir foo 2024-05-27T00:17:44.596 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo x 2024-05-27T00:17:44.596 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:47.585 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir foo/.snap/ss 2024-05-27T00:17:47.594 INFO:tasks.workunit.client.0.smithi098.stderr:+ ln foo/x foo/xx 2024-05-27T00:17:47.596 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat foo/.snap/ss/x 2024-05-27T00:17:47.604 INFO:tasks.workunit.client.0.smithi098.stdout:x 2024-05-27T00:17:47.604 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir foo/.snap/ss 2024-05-27T00:17:47.611 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo a 2024-05-27T00:17:47.612 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo b 2024-05-27T00:17:47.612 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir foo/.snap/s 2024-05-27T00:17:47.622 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat foo/.snap/s/a 2024-05-27T00:17:47.635 INFO:tasks.workunit.client.0.smithi098.stderr:+ r=a 2024-05-27T00:17:47.635 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ -z a ] 2024-05-27T00:17:47.635 INFO:tasks.workunit.client.0.smithi098.stderr:+ ln foo/b foo/b2 2024-05-27T00:17:47.637 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat foo/.snap/s/b 2024-05-27T00:17:47.645 INFO:tasks.workunit.client.0.smithi098.stdout:b 2024-05-27T00:17:47.645 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo this used to hang: 2024-05-27T00:17:47.646 INFO:tasks.workunit.client.0.smithi098.stdout:this used to hang: 2024-05-27T00:17:47.646 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo more 2024-05-27T00:17:47.646 INFO:tasks.workunit.client.0.smithi098.stdout:oh, it didn't hang! good job. 2024-05-27T00:17:47.646 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo oh, it didn't hang! good job. 2024-05-27T00:17:47.650 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat foo/b 2024-05-27T00:17:47.651 INFO:tasks.workunit.client.0.smithi098.stdout:b 2024-05-27T00:17:47.651 INFO:tasks.workunit.client.0.smithi098.stdout:more 2024-05-27T00:17:47.651 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir foo/.snap/s 2024-05-27T00:17:47.653 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm -r foo 2024-05-27T00:17:47.667 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo OK 2024-05-27T00:17:47.668 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-05-27T00:17:47.668 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T00:17:47.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T00:17:47.693 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2024-05-27T00:17:47.693 DEBUG:teuthology.orchestra.run.smithi098:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-05-27T00:17:47.752 INFO:tasks.workunit.client.0.smithi098.stderr:+ set -e 2024-05-27T00:17:47.752 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:47.754 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/bar 2024-05-27T00:17:47.764 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/foo 2024-05-27T00:17:47.771 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/foo 2024-05-27T00:17:47.776 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:47.779 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/bar/a 2024-05-27T00:17:47.781 INFO:tasks.workunit.client.0.smithi098.stdout: File: .snap/bar/a 2024-05-27T00:17:47.781 INFO:tasks.workunit.client.0.smithi098.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-27T00:17:47.782 INFO:tasks.workunit.client.0.smithi098.stdout:Device: feh/254d Inode: 1099511633797 Links: 2 2024-05-27T00:17:47.782 INFO:tasks.workunit.client.0.smithi098.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2024-05-27T00:17:47.782 INFO:tasks.workunit.client.0.smithi098.stdout:Access: 2024-05-27 00:17:47.748627918 +0000 2024-05-27T00:17:47.782 INFO:tasks.workunit.client.0.smithi098.stdout:Modify: 2024-05-27 00:17:47.748627918 +0000 2024-05-27T00:17:47.782 INFO:tasks.workunit.client.0.smithi098.stdout:Change: 2024-05-27 00:17:47.748627918 +0000 2024-05-27T00:17:47.782 INFO:tasks.workunit.client.0.smithi098.stdout: Birth: 2024-05-27 00:17:47.748627918 +0000 2024-05-27T00:17:47.782 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir .snap/bar 2024-05-27T00:17:47.790 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo OK 2024-05-27T00:17:47.791 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-05-27T00:17:47.791 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T00:17:47.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T00:17:47.859 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2024-05-27T00:17:47.859 DEBUG:teuthology.orchestra.run.smithi098:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-05-27T00:17:47.915 INFO:tasks.workunit.client.0.smithi098.stderr:+ set -e 2024-05-27T00:17:47.916 INFO:tasks.workunit.client.0.smithi098.stderr:+ seq 1 20 2024-05-27T00:17:47.917 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:47.919 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:47.920 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:47.928 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:47.933 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:47.934 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:48.273 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:48.353 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:48.353 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:48.353 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:48.353 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:48.353 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:48.353 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:48.353 INFO:tasks.workunit.client.0.smithi098.stdout: 37997 ? 00:00:00 ps 2024-05-27T00:17:48.354 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:48.363 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:48.365 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:48.370 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:48.373 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:48.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:48.382 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:48.388 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:48.389 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:48.457 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:48.537 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:48.537 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:48.537 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:48.537 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:48.537 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:48.537 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:48.537 INFO:tasks.workunit.client.0.smithi098.stdout: 38007 ? 00:00:00 ps 2024-05-27T00:17:48.538 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:48.548 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:48.550 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:48.555 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:48.557 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:48.558 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:48.567 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:48.572 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:48.573 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:48.649 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:48.728 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:48.729 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:48.729 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:48.729 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:48.729 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:48.729 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:48.729 INFO:tasks.workunit.client.0.smithi098.stdout: 38017 ? 00:00:00 ps 2024-05-27T00:17:48.729 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:48.741 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:48.743 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:48.748 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:48.751 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:48.752 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:48.760 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:48.766 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:48.767 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:48.849 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:48.929 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:48.929 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:48.929 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:48.929 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:48.929 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:48.929 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:48.929 INFO:tasks.workunit.client.0.smithi098.stdout: 38027 ? 00:00:00 ps 2024-05-27T00:17:48.930 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:48.941 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:48.943 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:48.948 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:48.951 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:48.952 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:48.960 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:48.967 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:48.968 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:49.016 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:49.095 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:49.095 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:49.096 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:49.096 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:49.096 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:49.096 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:49.096 INFO:tasks.workunit.client.0.smithi098.stdout: 38037 ? 00:00:00 ps 2024-05-27T00:17:49.096 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:49.107 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:49.109 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:49.115 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:49.117 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:49.118 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:49.127 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:49.133 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:49.134 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:49.183 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:49.262 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:49.262 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:49.262 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:49.262 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:49.262 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:49.263 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:49.263 INFO:tasks.workunit.client.0.smithi098.stdout: 38047 ? 00:00:00 ps 2024-05-27T00:17:49.263 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:49.274 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:49.276 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:49.282 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:49.284 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:49.284 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:49.293 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:49.299 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:49.300 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:49.367 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:49.445 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:49.445 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:49.445 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:49.445 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:49.446 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:49.446 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:49.446 INFO:tasks.workunit.client.0.smithi098.stdout: 38057 ? 00:00:00 ps 2024-05-27T00:17:49.446 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:49.456 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:49.458 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:49.463 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:49.466 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:49.466 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:49.474 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:49.480 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:49.481 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:49.526 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:49.605 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:49.605 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:49.605 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:49.605 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:49.605 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:49.606 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:49.606 INFO:tasks.workunit.client.0.smithi098.stdout: 38067 ? 00:00:00 ps 2024-05-27T00:17:49.606 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:49.616 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:49.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:49.625 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:49.628 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:49.629 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:49.638 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:49.644 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:49.645 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:49.717 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:49.796 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:49.796 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:49.796 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:49.796 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:49.797 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:49.797 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:49.797 INFO:tasks.workunit.client.0.smithi098.stdout: 38077 ? 00:00:00 ps 2024-05-27T00:17:49.797 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:49.809 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:49.812 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:49.819 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:49.821 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:49.823 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:49.831 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:49.838 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:49.839 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:49.909 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:49.987 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:49.987 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:49.987 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:49.987 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:49.987 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:49.987 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:49.987 INFO:tasks.workunit.client.0.smithi098.stdout: 38087 ? 00:00:00 ps 2024-05-27T00:17:49.988 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:49.999 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:50.001 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:50.007 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:50.009 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:50.010 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:50.020 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:50.026 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:50.027 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:50.085 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:50.164 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:50.164 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:50.164 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:50.164 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:50.164 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:50.164 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:50.164 INFO:tasks.workunit.client.0.smithi098.stdout: 38097 ? 00:00:00 ps 2024-05-27T00:17:50.165 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:50.176 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:50.178 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:50.183 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:50.186 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:50.187 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:50.195 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:50.209 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:50.210 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:50.297 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:50.377 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:50.377 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:50.377 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:50.377 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:50.377 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:50.377 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:50.377 INFO:tasks.workunit.client.0.smithi098.stdout: 38107 ? 00:00:00 ps 2024-05-27T00:17:50.377 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:50.388 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:50.390 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:50.396 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:50.398 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:50.399 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:50.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:50.414 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:50.415 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:50.477 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:50.556 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:50.557 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:50.557 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:50.557 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:50.557 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:50.557 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:50.557 INFO:tasks.workunit.client.0.smithi098.stdout: 38117 ? 00:00:00 ps 2024-05-27T00:17:50.557 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:50.569 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:50.571 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:50.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:50.579 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:50.580 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:50.590 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:50.596 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:50.597 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:50.669 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:50.752 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:50.753 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:50.753 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:50.753 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:50.753 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:50.753 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:50.753 INFO:tasks.workunit.client.0.smithi098.stdout: 38127 ? 00:00:00 ps 2024-05-27T00:17:50.753 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:50.770 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:50.773 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:50.781 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:50.784 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:50.785 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:50.797 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:50.805 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:50.806 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:50.886 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:50.964 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:50.964 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:50.964 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:50.964 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:50.964 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:50.964 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:50.964 INFO:tasks.workunit.client.0.smithi098.stdout: 38137 ? 00:00:00 ps 2024-05-27T00:17:50.965 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:50.979 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:50.982 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:50.988 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:50.991 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:50.992 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:51.003 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:51.011 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:51.012 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:51.070 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:51.150 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:51.150 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:51.150 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:51.151 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:51.151 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:51.151 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:51.151 INFO:tasks.workunit.client.0.smithi098.stdout: 38147 ? 00:00:00 ps 2024-05-27T00:17:51.151 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:51.165 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:51.167 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:51.174 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:51.177 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:51.178 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:51.190 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:51.199 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:51.200 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:51.244 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:51.374 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:51.374 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:51.374 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:51.374 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:51.374 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:51.374 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:51.374 INFO:tasks.workunit.client.0.smithi098.stdout: 38157 ? 00:00:00 ps 2024-05-27T00:17:51.375 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:51.420 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:51.427 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:51.436 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:51.439 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:51.441 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:51.459 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:51.469 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:51.470 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:51.812 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:51.890 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:51.890 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:51.890 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:51.890 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:51.891 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:51.891 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:51.891 INFO:tasks.workunit.client.0.smithi098.stdout: 38167 ? 00:00:00 ps 2024-05-27T00:17:51.891 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:51.906 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:51.908 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:51.915 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:51.917 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:51.918 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:51.930 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:51.939 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:51.939 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:52.037 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:52.114 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:52.115 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:52.115 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:52.115 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:52.115 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:52.115 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:52.115 INFO:tasks.workunit.client.0.smithi098.stdout: 38177 ? 00:00:00 ps 2024-05-27T00:17:52.115 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:52.131 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:52.133 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:52.140 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a 2024-05-27T00:17:52.142 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/one 2024-05-27T00:17:52.144 INFO:tasks.workunit.client.0.smithi098.stderr:+ cat 2024-05-27T00:17:52.156 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir a/.snap/two 2024-05-27T00:17:52.165 INFO:tasks.workunit.client.0.smithi098.stderr:+ chmod 777 a/foo 2024-05-27T00:17:52.166 INFO:tasks.workunit.client.0.smithi098.stderr:+ sync 2024-05-27T00:17:52.221 INFO:tasks.workunit.client.0.smithi098.stderr:+ ps 2024-05-27T00:17:52.311 INFO:tasks.workunit.client.0.smithi098.stdout: PID TTY TIME CMD 2024-05-27T00:17:52.311 INFO:tasks.workunit.client.0.smithi098.stdout: 1322 ? 00:00:00 systemd 2024-05-27T00:17:52.312 INFO:tasks.workunit.client.0.smithi098.stdout: 1323 ? 00:00:00 (sd-pam) 2024-05-27T00:17:52.312 INFO:tasks.workunit.client.0.smithi098.stdout: 17586 ? 00:00:01 sshd 2024-05-27T00:17:52.312 INFO:tasks.workunit.client.0.smithi098.stdout: 37986 ? 00:00:00 timeout 2024-05-27T00:17:52.312 INFO:tasks.workunit.client.0.smithi098.stdout: 37989 ? 00:00:00 snaptest-double 2024-05-27T00:17:52.312 INFO:tasks.workunit.client.0.smithi098.stdout: 38187 ? 00:00:00 ps 2024-05-27T00:17:52.313 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T00:17:52.347 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm a/foo 2024-05-27T00:17:52.348 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir a 2024-05-27T00:17:52.354 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo OK 2024-05-27T00:17:52.355 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-05-27T00:17:52.355 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T00:17:52.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T00:17:52.423 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2024-05-27T00:17:52.423 DEBUG:teuthology.orchestra.run.smithi098:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2024-05-27T00:17:52.480 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/foo 2024-05-27T00:17:52.490 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo We want ENOENT, not ESTALE, here. 2024-05-27T00:17:52.490 INFO:tasks.workunit.client.0.smithi098.stdout:We want ENOENT, not ESTALE, here. 2024-05-27T00:17:52.490 INFO:tasks.workunit.client.0.smithi098.stderr:+ seq 1 100 2024-05-27T00:17:52.492 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/1 2024-05-27T00:17:52.492 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.494 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2024-05-27T00:17:52.494 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/2 2024-05-27T00:17:52.495 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.496 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2024-05-27T00:17:52.497 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/3 2024-05-27T00:17:52.497 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.498 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2024-05-27T00:17:52.499 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/4 2024-05-27T00:17:52.499 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.500 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2024-05-27T00:17:52.501 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/5 2024-05-27T00:17:52.501 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.503 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2024-05-27T00:17:52.503 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/6 2024-05-27T00:17:52.504 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.505 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2024-05-27T00:17:52.506 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/7 2024-05-27T00:17:52.506 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.507 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2024-05-27T00:17:52.508 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/8 2024-05-27T00:17:52.508 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.509 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2024-05-27T00:17:52.510 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/9 2024-05-27T00:17:52.510 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.512 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2024-05-27T00:17:52.512 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/10 2024-05-27T00:17:52.513 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.514 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2024-05-27T00:17:52.515 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/11 2024-05-27T00:17:52.515 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.516 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2024-05-27T00:17:52.517 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/12 2024-05-27T00:17:52.517 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.518 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2024-05-27T00:17:52.519 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/13 2024-05-27T00:17:52.519 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.520 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2024-05-27T00:17:52.521 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/14 2024-05-27T00:17:52.521 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.523 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2024-05-27T00:17:52.524 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/15 2024-05-27T00:17:52.524 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.525 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2024-05-27T00:17:52.526 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/16 2024-05-27T00:17:52.526 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.527 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2024-05-27T00:17:52.528 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/17 2024-05-27T00:17:52.528 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.532 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2024-05-27T00:17:52.533 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/18 2024-05-27T00:17:52.533 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.536 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2024-05-27T00:17:52.537 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/19 2024-05-27T00:17:52.538 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.540 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2024-05-27T00:17:52.541 INFO:tasks.workunit.client.0.smithi098.stderr:+ + stat .snap/foo/20 2024-05-27T00:17:52.541 INFO:tasks.workunit.client.0.smithi098.stderr:grep No such file 2024-05-27T00:17:52.545 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2024-05-27T00:17:52.545 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/21 2024-05-27T00:17:52.546 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.547 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2024-05-27T00:17:52.549 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/22 2024-05-27T00:17:52.549 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.551 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2024-05-27T00:17:52.553 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.555 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/23 2024-05-27T00:17:52.555 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2024-05-27T00:17:52.558 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/24 2024-05-27T00:17:52.558 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.559 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2024-05-27T00:17:52.560 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/25 2024-05-27T00:17:52.560 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.563 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2024-05-27T00:17:52.563 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/26 2024-05-27T00:17:52.564 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.565 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2024-05-27T00:17:52.566 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/27 2024-05-27T00:17:52.566 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.567 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2024-05-27T00:17:52.568 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/28 2024-05-27T00:17:52.568 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.569 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2024-05-27T00:17:52.570 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/29 2024-05-27T00:17:52.570 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.571 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2024-05-27T00:17:52.572 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/30 2024-05-27T00:17:52.572 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.574 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2024-05-27T00:17:52.574 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/31 2024-05-27T00:17:52.574 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.576 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2024-05-27T00:17:52.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/32 2024-05-27T00:17:52.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.579 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2024-05-27T00:17:52.580 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/33 2024-05-27T00:17:52.580 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.583 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2024-05-27T00:17:52.584 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/34 2024-05-27T00:17:52.584 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.586 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2024-05-27T00:17:52.587 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/35 2024-05-27T00:17:52.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.590 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2024-05-27T00:17:52.591 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.591 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/36 2024-05-27T00:17:52.594 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2024-05-27T00:17:52.595 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/37 2024-05-27T00:17:52.595 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.597 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2024-05-27T00:17:52.598 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/38 2024-05-27T00:17:52.598 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.600 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2024-05-27T00:17:52.601 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/39 2024-05-27T00:17:52.601 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.603 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2024-05-27T00:17:52.603 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/40 2024-05-27T00:17:52.604 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.605 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2024-05-27T00:17:52.606 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/41 2024-05-27T00:17:52.606 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.608 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2024-05-27T00:17:52.609 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/42 2024-05-27T00:17:52.609 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.610 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2024-05-27T00:17:52.611 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/43 2024-05-27T00:17:52.611 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.613 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2024-05-27T00:17:52.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/44 2024-05-27T00:17:52.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.615 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2024-05-27T00:17:52.616 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/45 2024-05-27T00:17:52.616 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.618 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2024-05-27T00:17:52.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/46 2024-05-27T00:17:52.619 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.621 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2024-05-27T00:17:52.622 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/47 2024-05-27T00:17:52.622 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.623 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2024-05-27T00:17:52.624 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/48 2024-05-27T00:17:52.624 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.626 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2024-05-27T00:17:52.627 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/49 2024-05-27T00:17:52.627 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.628 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2024-05-27T00:17:52.629 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/50 2024-05-27T00:17:52.629 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.630 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2024-05-27T00:17:52.631 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/51 2024-05-27T00:17:52.631 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.633 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2024-05-27T00:17:52.633 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/52 2024-05-27T00:17:52.634 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.635 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2024-05-27T00:17:52.636 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/53 2024-05-27T00:17:52.636 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.637 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2024-05-27T00:17:52.638 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/54 2024-05-27T00:17:52.638 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.639 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2024-05-27T00:17:52.640 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/55 2024-05-27T00:17:52.640 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.641 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2024-05-27T00:17:52.642 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/56 2024-05-27T00:17:52.642 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.643 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2024-05-27T00:17:52.644 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/57 2024-05-27T00:17:52.644 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.646 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2024-05-27T00:17:52.647 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/58 2024-05-27T00:17:52.647 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.648 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2024-05-27T00:17:52.649 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/59 2024-05-27T00:17:52.649 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.650 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2024-05-27T00:17:52.651 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/60 2024-05-27T00:17:52.651 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.652 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2024-05-27T00:17:52.654 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/61 2024-05-27T00:17:52.654 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.655 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2024-05-27T00:17:52.655 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/62 2024-05-27T00:17:52.655 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.656 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2024-05-27T00:17:52.657 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/63 2024-05-27T00:17:52.657 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.659 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2024-05-27T00:17:52.659 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/64 2024-05-27T00:17:52.660 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.661 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2024-05-27T00:17:52.662 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/65 2024-05-27T00:17:52.662 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.663 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2024-05-27T00:17:52.664 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/66 2024-05-27T00:17:52.664 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.665 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2024-05-27T00:17:52.666 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/67 2024-05-27T00:17:52.666 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.667 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2024-05-27T00:17:52.668 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/68 2024-05-27T00:17:52.668 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.670 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2024-05-27T00:17:52.671 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/69 2024-05-27T00:17:52.671 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.672 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2024-05-27T00:17:52.673 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/70 2024-05-27T00:17:52.673 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.675 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2024-05-27T00:17:52.675 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/71 2024-05-27T00:17:52.675 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.677 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2024-05-27T00:17:52.678 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/72 2024-05-27T00:17:52.678 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.679 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2024-05-27T00:17:52.680 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/73 2024-05-27T00:17:52.680 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.681 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2024-05-27T00:17:52.682 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/74 2024-05-27T00:17:52.682 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.684 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2024-05-27T00:17:52.684 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/75 2024-05-27T00:17:52.685 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.686 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2024-05-27T00:17:52.687 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/76 2024-05-27T00:17:52.687 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.688 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2024-05-27T00:17:52.689 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/77 2024-05-27T00:17:52.689 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.690 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2024-05-27T00:17:52.691 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/78 2024-05-27T00:17:52.691 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.693 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2024-05-27T00:17:52.693 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/79 2024-05-27T00:17:52.694 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.695 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2024-05-27T00:17:52.696 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/80 2024-05-27T00:17:52.696 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.697 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2024-05-27T00:17:52.698 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/81 2024-05-27T00:17:52.698 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.699 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2024-05-27T00:17:52.700 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/82 2024-05-27T00:17:52.700 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.701 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2024-05-27T00:17:52.702 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/83 2024-05-27T00:17:52.702 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.704 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2024-05-27T00:17:52.704 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/84 2024-05-27T00:17:52.704 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.706 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2024-05-27T00:17:52.707 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/85 2024-05-27T00:17:52.707 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.708 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2024-05-27T00:17:52.709 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/86 2024-05-27T00:17:52.709 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.710 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2024-05-27T00:17:52.711 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/87 2024-05-27T00:17:52.711 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.713 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2024-05-27T00:17:52.714 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/88 2024-05-27T00:17:52.714 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.715 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2024-05-27T00:17:52.716 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/89 2024-05-27T00:17:52.716 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.717 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2024-05-27T00:17:52.718 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/90 2024-05-27T00:17:52.718 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.720 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2024-05-27T00:17:52.721 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/91 2024-05-27T00:17:52.721 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.722 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2024-05-27T00:17:52.723 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/92 2024-05-27T00:17:52.723 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.724 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2024-05-27T00:17:52.725 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/93 2024-05-27T00:17:52.725 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.726 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2024-05-27T00:17:52.727 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/94 2024-05-27T00:17:52.727 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.729 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2024-05-27T00:17:52.729 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/95 2024-05-27T00:17:52.730 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.731 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2024-05-27T00:17:52.732 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/96 2024-05-27T00:17:52.732 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.733 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2024-05-27T00:17:52.734 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/97 2024-05-27T00:17:52.734 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.735 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2024-05-27T00:17:52.736 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/98 2024-05-27T00:17:52.736 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.737 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2024-05-27T00:17:52.738 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/99 2024-05-27T00:17:52.738 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.740 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2024-05-27T00:17:52.741 INFO:tasks.workunit.client.0.smithi098.stderr:+ stat .snap/foo/100 2024-05-27T00:17:52.741 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep No such file 2024-05-27T00:17:52.742 INFO:tasks.workunit.client.0.smithi098.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2024-05-27T00:17:52.742 INFO:tasks.workunit.client.0.smithi098.stderr:+ rmdir .snap/foo 2024-05-27T00:17:52.750 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo OK 2024-05-27T00:17:52.755 INFO:tasks.workunit.client.0.smithi098.stdout:OK 2024-05-27T00:17:52.755 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T00:17:52.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T00:17:52.819 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2024-05-27T00:17:52.820 DEBUG:teuthology.orchestra.run.smithi098:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-05-27T00:17:52.876 INFO:tasks.workunit.client.0.smithi098.stderr:+ set -e 2024-05-27T00:17:52.876 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo git config --global http.sslVerify false 2024-05-27T00:17:52.890 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo git config --global http.postBuffer 1048576000 2024-05-27T00:17:52.904 INFO:tasks.workunit.client.0.smithi098.stderr:+ retried=false 2024-05-27T00:17:52.904 INFO:tasks.workunit.client.0.smithi098.stderr:+ trap -- retry EXIT 2024-05-27T00:17:52.904 INFO:tasks.workunit.client.0.smithi098.stderr:+ rm -rf ceph 2024-05-27T00:17:52.904 INFO:tasks.workunit.client.0.smithi098.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2024-05-27T00:17:52.908 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into 'ceph'... 2024-05-27T00:18:08.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30ccc00640 1 Processor -- start 2024-05-27T00:18:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30ccc00640 1 -- start start 2024-05-27T00:18:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30ccc00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f30c80cc1f0 0x7f30c81592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30ccc00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30c80c1f60 con 0x7f30c80cbe10 2024-05-27T00:18:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30ccc00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30c80c6dd0 con 0x7f30c8073570 2024-05-27T00:18:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30ccc00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f30c8159810 con 0x7f30c80cc1f0 2024-05-27T00:18:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f30c80cbe10 0x7f30c80c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41160/0 (socket says 172.21.15.98:41160) 2024-05-27T00:18:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5e00640 1 -- 172.21.15.98:0/3314726708 learned_addr learned my addr 172.21.15.98:0/3314726708 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30bd400640 1 --2- 172.21.15.98:0/3314726708 >> v2:172.21.15.204:3300/0 conn(0x7f30c80cc1f0 0x7f30c81592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3160321008 0 0) 0x7f30c80c1f60 con 0x7f30c80cbe10 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30ac003530 con 0x7f30c80cbe10 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 200202698 0 0) 0x7f30ac003530 con 0x7f30c80cbe10 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 >> v2:172.21.15.204:3300/0 conn(0x7f30c80cc1f0 msgr2=0x7f30c81592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 --2- 172.21.15.98:0/3314726708 >> v2:172.21.15.204:3300/0 conn(0x7f30c80cc1f0 0x7f30c81592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 >> v1:172.21.15.204:6789/0 conn(0x7f30c8073570 legacy=0x7f30c81559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c80c70b0 con 0x7f30c80cbe10 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f30b8001680 con 0x7f30c80cbe10 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 >> v1:172.21.15.98:6789/0 conn(0x7f30c80cbe10 legacy=0x7f30c80c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30ac003a10 0x7f30ac003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30ac004810 0x7f30ac004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac0051d0 0x7f30ac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30c8159810 con 0x7f30ac0051d0 2024-05-27T00:18:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac00c0b0 con 0x7f30ac004810 2024-05-27T00:18:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f30ac00c2b0 con 0x7f30ac003a10 2024-05-27T00:18:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c6800640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac0051d0 0x7f30ac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c5e00640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30ac003a10 0x7f30ac003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30bd400640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30ac004810 0x7f30ac004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c6800640 1 -- 172.21.15.98:0/3314726708 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30ac003a10 msgr2=0x7f30ac003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c6800640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30ac003a10 0x7f30ac003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c6800640 1 -- 172.21.15.98:0/3314726708 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30ac004810 msgr2=0x7f30ac004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c6800640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30ac004810 0x7f30ac004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.064+0000 7f30c6800640 1 -- 172.21.15.98:0/3314726708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ac003530 con 0x7f30ac0051d0 2024-05-27T00:18:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30c6800640 1 -- 172.21.15.98:0/3314726708 reap_dead start 2024-05-27T00:18:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30c6800640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac0051d0 0x7f30ac00b9d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f30c0001e70 tx=0x7f30c001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d465d5b581e24d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0001a90 con 0x7f30ac0051d0 2024-05-27T00:18:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c0003bc0 con 0x7f30ac0051d0 2024-05-27T00:18:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30c5400640 1 -- 172.21.15.98:0/3314726708 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0005020 con 0x7f30ac0051d0 2024-05-27T00:18:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30ccc00640 1 -- 172.21.15.98:0/3314726708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac0051d0 msgr2=0x7f30ac00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30ccc00640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac0051d0 0x7f30ac00b9d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f30c0001e70 tx=0x7f30c001c780 comp rx=0 tx=0).stop 2024-05-27T00:18:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30ccc00640 1 -- 172.21.15.98:0/3314726708 shutdown_connections 2024-05-27T00:18:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30ccc00640 1 --2- 172.21.15.98:0/3314726708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30ac0051d0 0x7f30ac00b9d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30ccc00640 1 -- 172.21.15.98:0/3314726708 >> 172.21.15.98:0/3314726708 conn(0x7f30c8067530 msgr2=0x7f30c806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30ccc00640 1 -- 172.21.15.98:0/3314726708 shutdown_connections 2024-05-27T00:18:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.068+0000 7f30ccc00640 1 -- 172.21.15.98:0/3314726708 wait complete. 2024-05-27T00:18:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30ccc00640 1 Processor -- start 2024-05-27T00:18:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30ccc00640 1 -- start start 2024-05-27T00:18:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30ccc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30ac0051d0 0x7f30c81f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30ccc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c80b1ec0 0x7f30c814b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30ccc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c80c3680 0x7f30c8149790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30ccc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30c80c1330 con 0x7f30c80c3680 2024-05-27T00:18:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30ccc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f30c80c1050 con 0x7f30ac0051d0 2024-05-27T00:18:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30ccc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f30c80c79d0 con 0x7f30c80b1ec0 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30c6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c80c3680 0x7f30c8149790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30c6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c80c3680 0x7f30c8149790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36416/0 (socket says 172.21.15.98:36416) 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30c6800640 1 -- 172.21.15.98:0/2500531120 learned_addr learned my addr 172.21.15.98:0/2500531120 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30bd400640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c80b1ec0 0x7f30c814b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30c5e00640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30ac0051d0 0x7f30c81f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30c6800640 1 -- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c80b1ec0 msgr2=0x7f30c814b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30c6800640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c80b1ec0 0x7f30c814b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30c6800640 1 -- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30ac0051d0 msgr2=0x7f30c81f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30c6800640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30ac0051d0 0x7f30c81f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30c6800640 1 -- 172.21.15.98:0/2500531120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c80cb040 con 0x7f30c80c3680 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30c6800640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c80c3680 0x7f30c8149790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f30c001d020 tx=0x7f30ac005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30bea00640 1 -- 172.21.15.98:0/2500531120 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0007dc0 con 0x7f30c80c3680 2024-05-27T00:18:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30bea00640 1 -- 172.21.15.98:0/2500531120 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c00012e0 con 0x7f30c80c3680 2024-05-27T00:18:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30ccc00640 1 -- 172.21.15.98:0/2500531120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30c80cad60 con 0x7f30c80c3680 2024-05-27T00:18:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30bea00640 1 -- 172.21.15.98:0/2500531120 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0006d70 con 0x7f30c80c3680 2024-05-27T00:18:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.072+0000 7f30ccc00640 1 -- 172.21.15.98:0/2500531120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30c81f4620 con 0x7f30c80c3680 2024-05-27T00:18:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.076+0000 7f30ccc00640 1 -- 172.21.15.98:0/2500531120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30c8075cb0 con 0x7f30c80c3680 2024-05-27T00:18:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.076+0000 7f30bea00640 1 -- 172.21.15.98:0/2500531120 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f30c00a6480 con 0x7f30c80c3680 2024-05-27T00:18:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.076+0000 7f30bea00640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f30a40678c0 0x7f30a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.076+0000 7f30bea00640 1 -- 172.21.15.98:0/2500531120 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5939+0+0 (secure 0 0 0) 0x7f30c011f6e0 con 0x7f30c80c3680 2024-05-27T00:18:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.076+0000 7f30c5e00640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f30a40678c0 0x7f30a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.076+0000 7f30c5e00640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f30a40678c0 0x7f30a4069d80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f30b8000c10 tx=0x7f30b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:09.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.080+0000 7f30bea00640 1 -- 172.21.15.98:0/2500531120 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f30c0016490 con 0x7f30c80c3680 2024-05-27T00:18:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 -- 172.21.15.98:0/2500531120 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30c806f380 con 0x7f30c80c3680 2024-05-27T00:18:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30bea00640 1 -- 172.21.15.98:0/2500531120 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6838 (secure 0 0 0) 0x7f30c00f0000 con 0x7f30c80c3680 2024-05-27T00:18:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:09.276 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:18:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-05-27T00:18:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 -- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f30a40678c0 msgr2=0x7f30a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f30a40678c0 0x7f30a4069d80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f30b8000c10 tx=0x7f30b8009040 comp rx=0 tx=0).stop 2024-05-27T00:18:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 -- 172.21.15.98:0/2500531120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c80c3680 msgr2=0x7f30c8149790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c80c3680 0x7f30c8149790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f30c001d020 tx=0x7f30ac005840 comp rx=0 tx=0).stop 2024-05-27T00:18:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 -- 172.21.15.98:0/2500531120 shutdown_connections 2024-05-27T00:18:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30c80c3680 0x7f30c8149790 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30c80b1ec0 0x7f30c814b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f30a40678c0 0x7f30a4069d80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 --2- 172.21.15.98:0/2500531120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30ac0051d0 0x7f30c81f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 -- 172.21.15.98:0/2500531120 >> 172.21.15.98:0/2500531120 conn(0x7f30c8067530 msgr2=0x7f30c8152b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 -- 172.21.15.98:0/2500531120 shutdown_connections 2024-05-27T00:18:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.272+0000 7f30ccc00640 1 -- 172.21.15.98:0/2500531120 wait complete. 2024-05-27T00:18:09.297 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2024-05-27T00:18:09.297 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-27T00:18:09.297 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:18:09.320 INFO:tasks.ceph.mds.a:Stopped 2024-05-27T00:18:09.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:09.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.492+0000 7f2047600640 1 Processor -- start 2024-05-27T00:18:09.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.492+0000 7f2047600640 1 -- start start 2024-05-27T00:18:09.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f2047600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400735d0 0x7f2040151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f2047600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f20400c79f0 0x7f2040151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f2047600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20400c17d0 con 0x7f20400731f0 2024-05-27T00:18:09.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f2047600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f2040071990 con 0x7f20400c79f0 2024-05-27T00:18:09.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f2047600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2040070e10 con 0x7f20400735d0 2024-05-27T00:18:09.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f2045600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f20400c79f0 0x7f2040151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f203fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400735d0 0x7f2040151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f203fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400735d0 0x7f2040151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:58430/0 (socket says 172.21.15.98:58430) 2024-05-27T00:18:09.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f203fe00640 1 -- 172.21.15.98:0/3822456992 learned_addr learned my addr 172.21.15.98:0/3822456992 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:09.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f2044c00640 1 --1- 172.21.15.98:0/3822456992 >> v1:172.21.15.204:6789/0 conn(0x7f20400731f0 0x7f20400c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:58418/0 (socket says 172.21.15.98:58418) 2024-05-27T00:18:09.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f203fe00640 1 -- 172.21.15.98:0/3822456992 >> v2:172.21.15.204:3300/0 conn(0x7f20400c79f0 msgr2=0x7f2040151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:09.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f203fe00640 1 --2- 172.21.15.98:0/3822456992 >> v2:172.21.15.204:3300/0 conn(0x7f20400c79f0 0x7f2040151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f203fe00640 1 -- 172.21.15.98:0/3822456992 >> v1:172.21.15.204:6789/0 conn(0x7f20400731f0 legacy=0x7f20400c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.496+0000 7f203fe00640 1 -- 172.21.15.98:0/3822456992 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20400710f0 con 0x7f20400735d0 2024-05-27T00:18:09.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f203fe00640 1 --2- 172.21.15.98:0/3822456992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400735d0 0x7f2040151430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f202c009870 tx=0x7f202c01ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78530959c5cb85af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:09.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f203f400640 1 -- 172.21.15.98:0/3822456992 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c0b5070 con 0x7f20400735d0 2024-05-27T00:18:09.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f203f400640 1 -- 172.21.15.98:0/3822456992 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202c0a8a50 con 0x7f20400735d0 2024-05-27T00:18:09.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f203f400640 1 -- 172.21.15.98:0/3822456992 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c0a8e20 con 0x7f20400735d0 2024-05-27T00:18:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f2047600640 1 -- 172.21.15.98:0/3822456992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400735d0 msgr2=0x7f2040151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f2047600640 1 --2- 172.21.15.98:0/3822456992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400735d0 0x7f2040151430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f202c009870 tx=0x7f202c01ed50 comp rx=0 tx=0).stop 2024-05-27T00:18:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f2047600640 1 -- 172.21.15.98:0/3822456992 shutdown_connections 2024-05-27T00:18:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f2047600640 1 --2- 172.21.15.98:0/3822456992 >> v2:172.21.15.204:3300/0 conn(0x7f20400c79f0 0x7f2040151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f2047600640 1 --2- 172.21.15.98:0/3822456992 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400735d0 0x7f2040151430 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f2047600640 1 -- 172.21.15.98:0/3822456992 >> 172.21.15.98:0/3822456992 conn(0x7f2040067530 msgr2=0x7f204006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f2047600640 1 -- 172.21.15.98:0/3822456992 shutdown_connections 2024-05-27T00:18:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f2047600640 1 -- 172.21.15.98:0/3822456992 wait complete. 2024-05-27T00:18:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.500+0000 7f2047600640 1 Processor -- start 2024-05-27T00:18:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2047600640 1 -- start start 2024-05-27T00:18:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2047600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400731f0 0x7f2040151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2047600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20400735d0 0x7f20401477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2047600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20400c79f0 0x7f20401483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2047600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f20400c8400 con 0x7f20400c79f0 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2047600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f20400716b0 con 0x7f20400735d0 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2047600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2040157970 con 0x7f20400731f0 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2045600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20400c79f0 0x7f20401483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2045600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20400c79f0 0x7f20401483c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36420/0 (socket says 172.21.15.98:36420) 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2045600640 1 -- 172.21.15.98:0/4067829269 learned_addr learned my addr 172.21.15.98:0/4067829269 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2044c00640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400731f0 0x7f2040151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f203fe00640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20400735d0 0x7f20401477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2045600640 1 -- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400731f0 msgr2=0x7f2040151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2045600640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400731f0 0x7f2040151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2045600640 1 -- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20400735d0 msgr2=0x7f20401477f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2045600640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20400735d0 0x7f20401477f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2045600640 1 -- 172.21.15.98:0/4067829269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20401530c0 con 0x7f20400c79f0 2024-05-27T00:18:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2045600640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20400c79f0 0x7f20401483c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f203800e7f0 tx=0x7f203800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f203cc00640 1 -- 172.21.15.98:0/4067829269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038017070 con 0x7f20400c79f0 2024-05-27T00:18:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f203cc00640 1 -- 172.21.15.98:0/4067829269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f203800c940 con 0x7f20400c79f0 2024-05-27T00:18:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2047600640 1 -- 172.21.15.98:0/4067829269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2040154900 con 0x7f20400c79f0 2024-05-27T00:18:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f203cc00640 1 -- 172.21.15.98:0/4067829269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203800eea0 con 0x7f20400c79f0 2024-05-27T00:18:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2047600640 1 -- 172.21.15.98:0/4067829269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f204015ad80 con 0x7f20400c79f0 2024-05-27T00:18:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.504+0000 7f2047600640 1 -- 172.21.15.98:0/4067829269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2040075cb0 con 0x7f20400c79f0 2024-05-27T00:18:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.508+0000 7f203cc00640 1 -- 172.21.15.98:0/4067829269 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f203801a070 con 0x7f20400c79f0 2024-05-27T00:18:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.508+0000 7f203cc00640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f20200678c0 0x7f2020069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.508+0000 7f203cc00640 1 -- 172.21.15.98:0/4067829269 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5939+0+0 (secure 0 0 0) 0x7f2038085bd0 con 0x7f20400c79f0 2024-05-27T00:18:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.508+0000 7f2044c00640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f20200678c0 0x7f2020069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:09.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.508+0000 7f2044c00640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f20200678c0 0x7f2020069d80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f20300012f0 tx=0x7f20300016e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:09.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.512+0000 7f203cc00640 1 -- 172.21.15.98:0/4067829269 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f2038055fc0 con 0x7f20400c79f0 2024-05-27T00:18:09.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.700+0000 7f2047600640 1 -- 172.21.15.98:0/4067829269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f204014e2e0 con 0x7f20400c79f0 2024-05-27T00:18:09.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.700+0000 7f203cc00640 1 -- 172.21.15.98:0/4067829269 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6838 (secure 0 0 0) 0x7f203805a2c0 con 0x7f20400c79f0 2024-05-27T00:18:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:09.706 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:18:09.706 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-05-27T00:18:09.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 -- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f20200678c0 msgr2=0x7f2020069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f20200678c0 0x7f2020069d80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f20300012f0 tx=0x7f20300016e0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 -- 172.21.15.98:0/4067829269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20400c79f0 msgr2=0x7f20401483c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:09.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20400c79f0 0x7f20401483c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f203800e7f0 tx=0x7f203800ecb0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 -- 172.21.15.98:0/4067829269 shutdown_connections 2024-05-27T00:18:09.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20400c79f0 0x7f20401483c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f20200678c0 0x7f2020069d80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20400735d0 0x7f20401477f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 --2- 172.21.15.98:0/4067829269 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f20400731f0 0x7f2040151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:09.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 -- 172.21.15.98:0/4067829269 >> 172.21.15.98:0/4067829269 conn(0x7f2040067530 msgr2=0x7f20400c6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:09.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 -- 172.21.15.98:0/4067829269 shutdown_connections 2024-05-27T00:18:09.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:09.704+0000 7f2047600640 1 -- 172.21.15.98:0/4067829269 wait complete. 2024-05-27T00:18:09.726 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:18:11.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.868+0000 7f3ce6000640 1 Processor -- start 2024-05-27T00:18:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.868+0000 7f3ce6000640 1 -- start start 2024-05-27T00:18:11.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3ce6000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3ce0070e10 0x7f3ce01627e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:11.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3ce6000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce00c1000 con 0x7f3ce01511d0 2024-05-27T00:18:11.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3ce6000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce01515b0 con 0x7f3ce00cc8f0 2024-05-27T00:18:11.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3ce6000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce0071240 con 0x7f3ce0070e10 2024-05-27T00:18:11.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3cdfe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3ce0070e10 0x7f3ce01627e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:11.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3cdf400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3ce01511d0 0x7f3ce00c1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41174/0 (socket says 172.21.15.98:41174) 2024-05-27T00:18:11.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3cdf400640 1 -- 172.21.15.98:0/3933263054 learned_addr learned my addr 172.21.15.98:0/3933263054 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:11.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3cdea00640 1 -- 172.21.15.98:0/3933263054 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3885492525 0 0) 0x7f3ce00c1000 con 0x7f3ce01511d0 2024-05-27T00:18:11.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3cdea00640 1 -- 172.21.15.98:0/3933263054 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc4003530 con 0x7f3ce01511d0 2024-05-27T00:18:11.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3cdfe00640 1 -- 172.21.15.98:0/3933263054 >> v1:172.21.15.204:6789/0 conn(0x7f3ce00cc8f0 legacy=0x7f3ce00c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:11.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3cdfe00640 1 -- 172.21.15.98:0/3933263054 >> v1:172.21.15.98:6789/0 conn(0x7f3ce01511d0 legacy=0x7f3ce00c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:11.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3cdfe00640 1 -- 172.21.15.98:0/3933263054 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00c83c0 con 0x7f3ce0070e10 2024-05-27T00:18:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.872+0000 7f3cdfe00640 1 --2- 172.21.15.98:0/3933263054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 0x7f3ce01627e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80097c0 tx=0x7f3cd801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c59f268393782b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3cdea00640 1 -- 172.21.15.98:0/3933263054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd80b0070 con 0x7f3ce0070e10 2024-05-27T00:18:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3cdea00640 1 -- 172.21.15.98:0/3933263054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd80ab600 con 0x7f3ce0070e10 2024-05-27T00:18:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3cdea00640 1 -- 172.21.15.98:0/3933263054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd80ab9d0 con 0x7f3ce0070e10 2024-05-27T00:18:11.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 -- 172.21.15.98:0/3933263054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 msgr2=0x7f3ce01627e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:11.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 --2- 172.21.15.98:0/3933263054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 0x7f3ce01627e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80097c0 tx=0x7f3cd801c410 comp rx=0 tx=0).stop 2024-05-27T00:18:11.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 -- 172.21.15.98:0/3933263054 shutdown_connections 2024-05-27T00:18:11.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 --2- 172.21.15.98:0/3933263054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 0x7f3ce01627e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:11.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 -- 172.21.15.98:0/3933263054 >> 172.21.15.98:0/3933263054 conn(0x7f3ce0067530 msgr2=0x7f3ce006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:11.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 -- 172.21.15.98:0/3933263054 shutdown_connections 2024-05-27T00:18:11.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 -- 172.21.15.98:0/3933263054 wait complete. 2024-05-27T00:18:11.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 Processor -- start 2024-05-27T00:18:11.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 -- start start 2024-05-27T00:18:11.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 0x7f3ce00c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:11.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ce00cc8f0 0x7f3ce0147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:11.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ce01511d0 0x7f3ce00c3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00c1a20 con 0x7f3ce0070e10 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00b1dd0 con 0x7f3ce01511d0 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3ce6000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce00c2150 con 0x7f3ce00cc8f0 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3cdf400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 0x7f3ce00c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3cdf400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 0x7f3ce00c5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36444/0 (socket says 172.21.15.98:36444) 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3cdf400640 1 -- 172.21.15.98:0/2470997338 learned_addr learned my addr 172.21.15.98:0/2470997338 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3cdfe00640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ce01511d0 0x7f3ce00c3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.876+0000 7f3cd6a00640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ce00cc8f0 0x7f3ce0147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cdf400640 1 -- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ce00cc8f0 msgr2=0x7f3ce0147da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cdf400640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ce00cc8f0 0x7f3ce0147da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cdf400640 1 -- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ce01511d0 msgr2=0x7f3ce00c3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cdf400640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ce01511d0 0x7f3ce00c3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cdf400640 1 -- 172.21.15.98:0/2470997338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce014eb60 con 0x7f3ce0070e10 2024-05-27T00:18:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cd6a00640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ce00cc8f0 0x7f3ce0147da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:18:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cdf400640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 0x7f3ce00c5aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000c420 tx=0x7f3cd000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cd7e00640 1 -- 172.21.15.98:0/2470997338 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0017070 con 0x7f3ce0070e10 2024-05-27T00:18:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cd7e00640 1 -- 172.21.15.98:0/2470997338 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd0005cf0 con 0x7f3ce0070e10 2024-05-27T00:18:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cd7e00640 1 -- 172.21.15.98:0/2470997338 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd00072d0 con 0x7f3ce0070e10 2024-05-27T00:18:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3ce6000640 1 -- 172.21.15.98:0/2470997338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce00c4460 con 0x7f3ce0070e10 2024-05-27T00:18:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3ce6000640 1 -- 172.21.15.98:0/2470997338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce00c49e0 con 0x7f3ce0070e10 2024-05-27T00:18:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3ce6000640 1 -- 172.21.15.98:0/2470997338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce0075cb0 con 0x7f3ce0070e10 2024-05-27T00:18:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cd7e00640 1 -- 172.21.15.98:0/2470997338 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3cd001a070 con 0x7f3ce0070e10 2024-05-27T00:18:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cd7e00640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 0x7f3cbc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cd7e00640 1 -- 172.21.15.98:0/2470997338 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5939+0+0 (secure 0 0 0) 0x7f3cd0085190 con 0x7f3ce0070e10 2024-05-27T00:18:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.880+0000 7f3cd6a00640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 0x7f3cbc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.884+0000 7f3cd6a00640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 0x7f3cbc069d80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3ce01624c0 tx=0x7f3cc8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:11.888+0000 7f3cd7e00640 1 -- 172.21.15.98:0/2470997338 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f3cd00554f0 con 0x7f3ce0070e10 2024-05-27T00:18:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 -- 172.21.15.98:0/2470997338 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ce01f25e0 con 0x7f3ce0070e10 2024-05-27T00:18:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3cd7e00640 1 -- 172.21.15.98:0/2470997338 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6838 (secure 0 0 0) 0x7f3cd00597f0 con 0x7f3ce0070e10 2024-05-27T00:18:12.079 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:12.079 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:18:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 -- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 msgr2=0x7f3cbc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 0x7f3cbc069d80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3ce01624c0 tx=0x7f3cc8009040 comp rx=0 tx=0).stop 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 -- 172.21.15.98:0/2470997338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 msgr2=0x7f3ce00c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 0x7f3ce00c5aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000c420 tx=0x7f3cd000c8e0 comp rx=0 tx=0).stop 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 -- 172.21.15.98:0/2470997338 shutdown_connections 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 0x7f3cbc069d80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ce01511d0 0x7f3ce00c3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ce00cc8f0 0x7f3ce0147da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 --2- 172.21.15.98:0/2470997338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ce0070e10 0x7f3ce00c5aa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 -- 172.21.15.98:0/2470997338 >> 172.21.15.98:0/2470997338 conn(0x7f3ce0067530 msgr2=0x7f3ce01440a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 -- 172.21.15.98:0/2470997338 shutdown_connections 2024-05-27T00:18:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:12.076+0000 7f3ce6000640 1 -- 172.21.15.98:0/2470997338 wait complete. 2024-05-27T00:18:12.100 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:18:14.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:14.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.244+0000 7f7862600640 1 Processor -- start 2024-05-27T00:18:14.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.244+0000 7f7862600640 1 -- start start 2024-05-27T00:18:14.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.244+0000 7f7862600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f785c0731f0 0x7f785c0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:14.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.244+0000 7f7862600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f785c0c17d0 con 0x7f785c0c79f0 2024-05-27T00:18:14.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.244+0000 7f7862600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f785c070e10 con 0x7f785c0735d0 2024-05-27T00:18:14.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.244+0000 7f7862600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f785c071990 con 0x7f785c0731f0 2024-05-27T00:18:14.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785be00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f785c0c79f0 0x7f785c157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55542/0 (socket says 172.21.15.98:55542) 2024-05-27T00:18:14.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785be00640 1 -- 172.21.15.98:0/2592816775 learned_addr learned my addr 172.21.15.98:0/2592816775 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:14.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785b400640 1 --2- 172.21.15.98:0/2592816775 >> v2:172.21.15.204:3300/0 conn(0x7f785c0731f0 0x7f785c0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3494151534 0 0) 0x7f785c0c17d0 con 0x7f785c0c79f0 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7840003530 con 0x7f785c0c79f0 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 495614866 0 0) 0x7f785c070e10 con 0x7f785c0735d0 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f785c0c17d0 con 0x7f785c0735d0 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2345779293 0 0) 0x7f7840003530 con 0x7f785c0c79f0 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 >> v2:172.21.15.204:3300/0 conn(0x7f785c0731f0 msgr2=0x7f785c0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 --2- 172.21.15.98:0/2592816775 >> v2:172.21.15.204:3300/0 conn(0x7f785c0731f0 0x7f785c0c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 >> v1:172.21.15.204:6789/0 conn(0x7f785c0735d0 legacy=0x7f785c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785c0710f0 con 0x7f785c0c79f0 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f7854001a00 con 0x7f785c0c79f0 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 >> v1:172.21.15.98:6789/0 conn(0x7f785c0c79f0 legacy=0x7f785c157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7840003b50 0x7f7840003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:14.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7840004810 0x7f7840004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78400051d0 0x7f784000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f785c071990 con 0x7f7840004810 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f784000c070 con 0x7f78400051d0 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f784000c270 con 0x7f7840003b50 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f7852a00640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7840004810 0x7f7840004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785b400640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7840003b50 0x7f7840003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785be00640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78400051d0 0x7f784000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f7852a00640 1 -- 172.21.15.98:0/2592816775 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7840003b50 msgr2=0x7f7840003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f7852a00640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7840003b50 0x7f7840003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f7852a00640 1 -- 172.21.15.98:0/2592816775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78400051d0 msgr2=0x7f784000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f7852a00640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78400051d0 0x7f784000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f7852a00640 1 -- 172.21.15.98:0/2592816775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7840003530 con 0x7f7840004810 2024-05-27T00:18:14.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.248+0000 7f785be00640 1 -- 172.21.15.98:0/2592816775 reap_dead start 2024-05-27T00:18:14.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f7852a00640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7840004810 0x7f7840004c90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f785c0739f0 tx=0x7f784401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe573c05a8f242c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:14.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7844001520 con 0x7f7840004810 2024-05-27T00:18:14.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78440052f0 con 0x7f7840004810 2024-05-27T00:18:14.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f785aa00640 1 -- 172.21.15.98:0/2592816775 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78440a8910 con 0x7f7840004810 2024-05-27T00:18:14.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f7862600640 1 -- 172.21.15.98:0/2592816775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7840004810 msgr2=0x7f7840004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:14.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f7862600640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7840004810 0x7f7840004c90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f785c0739f0 tx=0x7f784401ecb0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f7862600640 1 -- 172.21.15.98:0/2592816775 shutdown_connections 2024-05-27T00:18:14.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f7862600640 1 --2- 172.21.15.98:0/2592816775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7840004810 0x7f7840004c90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f7862600640 1 -- 172.21.15.98:0/2592816775 >> 172.21.15.98:0/2592816775 conn(0x7f785c067530 msgr2=0x7f785c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:14.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f7862600640 1 -- 172.21.15.98:0/2592816775 shutdown_connections 2024-05-27T00:18:14.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f7862600640 1 -- 172.21.15.98:0/2592816775 wait complete. 2024-05-27T00:18:14.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f7862600640 1 Processor -- start 2024-05-27T00:18:14.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.252+0000 7f7862600640 1 -- start start 2024-05-27T00:18:14.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7862600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7840004810 0x7f785c0c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:14.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7862600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c4fa0 0x7f785c156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:14.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7862600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f785c1ede80 0x7f785c1ee300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7862600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f785c0c8400 con 0x7f785c0c4fa0 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7862600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f785c0716b0 con 0x7f785c1ede80 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7862600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f785c0c0eb0 con 0x7f7840004810 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7852a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c4fa0 0x7f785c156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7852a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c4fa0 0x7f785c156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38310/0 (socket says 172.21.15.98:38310) 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7852a00640 1 -- 172.21.15.98:0/2861483612 learned_addr learned my addr 172.21.15.98:0/2861483612 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f785b400640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7840004810 0x7f785c0c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f785be00640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f785c1ede80 0x7f785c1ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7852a00640 1 -- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7840004810 msgr2=0x7f785c0c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7852a00640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7840004810 0x7f785c0c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7852a00640 1 -- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f785c1ede80 msgr2=0x7f785c1ee300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7852a00640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f785c1ede80 0x7f785c1ee300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7852a00640 1 -- 172.21.15.98:0/2861483612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785c1f22a0 con 0x7f785c0c4fa0 2024-05-27T00:18:14.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7852a00640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c4fa0 0x7f785c156b80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f784401eae0 tx=0x7f7840005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:14.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7853e00640 1 -- 172.21.15.98:0/2861483612 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78440bd070 con 0x7f785c0c4fa0 2024-05-27T00:18:14.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7853e00640 1 -- 172.21.15.98:0/2861483612 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78440054a0 con 0x7f785c0c4fa0 2024-05-27T00:18:14.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7862600640 1 -- 172.21.15.98:0/2861483612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785c0c62d0 con 0x7f785c0c4fa0 2024-05-27T00:18:14.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7853e00640 1 -- 172.21.15.98:0/2861483612 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7844001500 con 0x7f785c0c4fa0 2024-05-27T00:18:14.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7862600640 1 -- 172.21.15.98:0/2861483612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f785c0c6010 con 0x7f785c0c4fa0 2024-05-27T00:18:14.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7862600640 1 -- 172.21.15.98:0/2861483612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f785c075cb0 con 0x7f785c0c4fa0 2024-05-27T00:18:14.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.256+0000 7f7853e00640 1 -- 172.21.15.98:0/2861483612 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f78440c0070 con 0x7f785c0c4fa0 2024-05-27T00:18:14.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.260+0000 7f7853e00640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78380678f0 0x7f7838069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:14.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.260+0000 7f7853e00640 1 -- 172.21.15.98:0/2861483612 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5939+0+0 (secure 0 0 0) 0x7f7844122370 con 0x7f785c0c4fa0 2024-05-27T00:18:14.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.260+0000 7f785b400640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78380678f0 0x7f7838069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:14.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.260+0000 7f785b400640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78380678f0 0x7f7838069db0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f784c009b80 tx=0x7f784c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:14.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.264+0000 7f7853e00640 1 -- 172.21.15.98:0/2861483612 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f78440189c0 con 0x7f785c0c4fa0 2024-05-27T00:18:14.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.452+0000 7f7862600640 1 -- 172.21.15.98:0/2861483612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f785c06eca0 con 0x7f785c0c4fa0 2024-05-27T00:18:14.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.452+0000 7f7853e00640 1 -- 172.21.15.98:0/2861483612 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6838 (secure 0 0 0) 0x7f78440f2db0 con 0x7f785c0c4fa0 2024-05-27T00:18:14.457 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:14.457 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:18:14.457 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-05-27T00:18:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 -- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78380678f0 msgr2=0x7f7838069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78380678f0 0x7f7838069db0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f784c009b80 tx=0x7f784c009190 comp rx=0 tx=0).stop 2024-05-27T00:18:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 -- 172.21.15.98:0/2861483612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c4fa0 msgr2=0x7f785c156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c4fa0 0x7f785c156b80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f784401eae0 tx=0x7f7840005840 comp rx=0 tx=0).stop 2024-05-27T00:18:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 -- 172.21.15.98:0/2861483612 shutdown_connections 2024-05-27T00:18:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78380678f0 0x7f7838069db0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f785c1ede80 0x7f785c1ee300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f785c0c4fa0 0x7f785c156b80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 --2- 172.21.15.98:0/2861483612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7840004810 0x7f785c0c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 -- 172.21.15.98:0/2861483612 >> 172.21.15.98:0/2861483612 conn(0x7f785c067530 msgr2=0x7f785c150ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 -- 172.21.15.98:0/2861483612 shutdown_connections 2024-05-27T00:18:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:14.456+0000 7f7862600640 1 -- 172.21.15.98:0/2861483612 wait complete. 2024-05-27T00:18:14.478 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:18:14.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:18:14.748 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:18:16.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:16.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.620+0000 7f35ee800640 1 Processor -- start 2024-05-27T00:18:16.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.620+0000 7f35ee800640 1 -- start start 2024-05-27T00:18:16.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35ee800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35e80c0eb0 0x7f35e80c26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35ee800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35e80c7dc0 0x7f35e815ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35ee800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e8159170 con 0x7f35e8158d90 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35ee800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f35e80c8210 con 0x7f35e80c0eb0 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35ee800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f35e815f840 con 0x7f35e80c7dc0 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35e6a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f35e80c0eb0 0x7f35e80c26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35e7400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f35e8158d90 0x7f35e80c68c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55556/0 (socket says 172.21.15.98:55556) 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35e7400640 1 -- 172.21.15.98:0/2031679184 learned_addr learned my addr 172.21.15.98:0/2031679184 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35e7e00640 1 --2- 172.21.15.98:0/2031679184 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35e80c7dc0 0x7f35e815ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35e6a00640 1 -- 172.21.15.98:0/2031679184 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35e80c7dc0 msgr2=0x7f35e815ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35e6a00640 1 --2- 172.21.15.98:0/2031679184 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35e80c7dc0 0x7f35e815ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35e6a00640 1 -- 172.21.15.98:0/2031679184 >> v1:172.21.15.98:6789/0 conn(0x7f35e8158d90 legacy=0x7f35e80c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35e6a00640 1 -- 172.21.15.98:0/2031679184 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e815fb20 con 0x7f35e80c0eb0 2024-05-27T00:18:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.624+0000 7f35e6000640 1 -- 172.21.15.98:0/2031679184 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3397134025 0 0) 0x7f35e8159170 con 0x7f35e8158d90 2024-05-27T00:18:16.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35e6a00640 1 --2- 172.21.15.98:0/2031679184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e80c0eb0 0x7f35e80c26a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f35d40097c0 tx=0x7f35d401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e86d8f95404d2c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:16.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35e6000640 1 -- 172.21.15.98:0/2031679184 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d40b5070 con 0x7f35e80c0eb0 2024-05-27T00:18:16.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35e6000640 1 -- 172.21.15.98:0/2031679184 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d40a8990 con 0x7f35e80c0eb0 2024-05-27T00:18:16.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35e6000640 1 -- 172.21.15.98:0/2031679184 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d40a8d60 con 0x7f35e80c0eb0 2024-05-27T00:18:16.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35ee800640 1 -- 172.21.15.98:0/2031679184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e80c0eb0 msgr2=0x7f35e80c26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:16.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35ee800640 1 --2- 172.21.15.98:0/2031679184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e80c0eb0 0x7f35e80c26a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f35d40097c0 tx=0x7f35d401ec90 comp rx=0 tx=0).stop 2024-05-27T00:18:16.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35ee800640 1 -- 172.21.15.98:0/2031679184 shutdown_connections 2024-05-27T00:18:16.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35ee800640 1 --2- 172.21.15.98:0/2031679184 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35e80c7dc0 0x7f35e815ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:16.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35ee800640 1 --2- 172.21.15.98:0/2031679184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e80c0eb0 0x7f35e80c26a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:16.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35ee800640 1 -- 172.21.15.98:0/2031679184 >> 172.21.15.98:0/2031679184 conn(0x7f35e8067530 msgr2=0x7f35e806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:16.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35ee800640 1 -- 172.21.15.98:0/2031679184 shutdown_connections 2024-05-27T00:18:16.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35ee800640 1 -- 172.21.15.98:0/2031679184 wait complete. 2024-05-27T00:18:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35ee800640 1 Processor -- start 2024-05-27T00:18:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.628+0000 7f35ee800640 1 -- start start 2024-05-27T00:18:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35ee800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35e80c0eb0 0x7f35e81e89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35ee800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35e80c7dc0 0x7f35e80c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35ee800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e8158d90 0x7f35e81f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35ee800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35e80c1ea0 con 0x7f35e8158d90 2024-05-27T00:18:16.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35ee800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f35e80c1bc0 con 0x7f35e80c7dc0 2024-05-27T00:18:16.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35ee800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f35e80c2180 con 0x7f35e80c0eb0 2024-05-27T00:18:16.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e8158d90 0x7f35e81f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:16.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e8158d90 0x7f35e81f49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38328/0 (socket says 172.21.15.98:38328) 2024-05-27T00:18:16.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e7e00640 1 -- 172.21.15.98:0/335965724 learned_addr learned my addr 172.21.15.98:0/335965724 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e7400640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35e80c0eb0 0x7f35e81e89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e6a00640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35e80c7dc0 0x7f35e80c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e7e00640 1 -- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35e80c0eb0 msgr2=0x7f35e81e89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e7e00640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35e80c0eb0 0x7f35e81e89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e7e00640 1 -- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35e80c7dc0 msgr2=0x7f35e80c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e7e00640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35e80c7dc0 0x7f35e80c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e7e00640 1 -- 172.21.15.98:0/335965724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e81e8fc0 con 0x7f35e8158d90 2024-05-27T00:18:16.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e6a00640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35e80c7dc0 0x7f35e80c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:18:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35e7e00640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e8158d90 0x7f35e81f49d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f35e0007e70 tx=0x7f35e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35df400640 1 -- 172.21.15.98:0/335965724 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e0014620 con 0x7f35e8158d90 2024-05-27T00:18:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35df400640 1 -- 172.21.15.98:0/335965724 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35e0019070 con 0x7f35e8158d90 2024-05-27T00:18:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35ee800640 1 -- 172.21.15.98:0/335965724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e815b7f0 con 0x7f35e8158d90 2024-05-27T00:18:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35df400640 1 -- 172.21.15.98:0/335965724 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e0014a20 con 0x7f35e8158d90 2024-05-27T00:18:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35ee800640 1 -- 172.21.15.98:0/335965724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e815bdb0 con 0x7f35e8158d90 2024-05-27T00:18:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35ee800640 1 -- 172.21.15.98:0/335965724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e815c0c0 con 0x7f35e8158d90 2024-05-27T00:18:16.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.632+0000 7f35df400640 1 -- 172.21.15.98:0/335965724 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f35e0014bc0 con 0x7f35e8158d90 2024-05-27T00:18:16.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.636+0000 7f35df400640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35c80678f0 0x7f35c8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.636+0000 7f35df400640 1 -- 172.21.15.98:0/335965724 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5939+0+0 (secure 0 0 0) 0x7f35e00866f0 con 0x7f35e8158d90 2024-05-27T00:18:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.636+0000 7f35e7400640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35c80678f0 0x7f35c8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:16.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.636+0000 7f35e7400640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35c80678f0 0x7f35c8069db0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f35d8000f70 tx=0x7f35d8007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:16.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.640+0000 7f35df400640 1 -- 172.21.15.98:0/335965724 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f35e0011260 con 0x7f35e8158d90 2024-05-27T00:18:16.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 -- 172.21.15.98:0/335965724 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35e8075cb0 con 0x7f35e8158d90 2024-05-27T00:18:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35df400640 1 -- 172.21.15.98:0/335965724 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6838 (secure 0 0 0) 0x7f35e00570a0 con 0x7f35e8158d90 2024-05-27T00:18:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:16.831 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:18:16.831 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-05-27T00:18:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 -- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35c80678f0 msgr2=0x7f35c8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35c80678f0 0x7f35c8069db0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f35d8000f70 tx=0x7f35d8007c10 comp rx=0 tx=0).stop 2024-05-27T00:18:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 -- 172.21.15.98:0/335965724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e8158d90 msgr2=0x7f35e81f49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e8158d90 0x7f35e81f49d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f35e0007e70 tx=0x7f35e000e4b0 comp rx=0 tx=0).stop 2024-05-27T00:18:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 -- 172.21.15.98:0/335965724 shutdown_connections 2024-05-27T00:18:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35e8158d90 0x7f35e81f49d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35c80678f0 0x7f35c8069db0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35e80c7dc0 0x7f35e80c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:16.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 --2- 172.21.15.98:0/335965724 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35e80c0eb0 0x7f35e81e89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 -- 172.21.15.98:0/335965724 >> 172.21.15.98:0/335965724 conn(0x7f35e8067530 msgr2=0x7f35e8071eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 -- 172.21.15.98:0/335965724 shutdown_connections 2024-05-27T00:18:16.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:16.828+0000 7f35ee800640 1 -- 172.21.15.98:0/335965724 wait complete. 2024-05-27T00:18:16.852 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:18:18.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:18.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.984+0000 7eff09e00640 1 Processor -- start 2024-05-27T00:18:18.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff09e00640 1 -- start start 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff09e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7eff040cac80 0x7eff040c9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff09e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7eff0415bb00 0x7eff041523c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff09e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff0415bf90 con 0x7eff040c9640 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff09e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7eff040c84f0 con 0x7eff040cac80 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff09e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7eff040c8990 con 0x7eff0415bb00 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff08a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7eff040cac80 0x7eff040c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff08a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7eff040cac80 0x7eff040c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38340/0 (socket says 172.21.15.98:38340) 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff08a00640 1 -- 172.21.15.98:0/836812582 learned_addr learned my addr 172.21.15.98:0/836812582 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7efefbe00640 1 --2- 172.21.15.98:0/836812582 >> v2:172.21.15.204:3300/0 conn(0x7eff0415bb00 0x7eff041523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff08a00640 1 -- 172.21.15.98:0/836812582 >> v2:172.21.15.204:3300/0 conn(0x7eff0415bb00 msgr2=0x7eff041523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff08a00640 1 --2- 172.21.15.98:0/836812582 >> v2:172.21.15.204:3300/0 conn(0x7eff0415bb00 0x7eff041523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff08a00640 1 -- 172.21.15.98:0/836812582 >> v1:172.21.15.204:6789/0 conn(0x7eff040c9640 legacy=0x7eff0406d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:18.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.988+0000 7eff08a00640 1 -- 172.21.15.98:0/836812582 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0415c270 con 0x7eff040cac80 2024-05-27T00:18:18.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff08a00640 1 --2- 172.21.15.98:0/836812582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040cac80 0x7eff040c9100 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efef4009870 tx=0x7efef401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2230b260cd5eed4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:18.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff03e00640 1 -- 172.21.15.98:0/836812582 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef40b0070 con 0x7eff040cac80 2024-05-27T00:18:18.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff03e00640 1 -- 172.21.15.98:0/836812582 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef40ab600 con 0x7eff040cac80 2024-05-27T00:18:18.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff03e00640 1 -- 172.21.15.98:0/836812582 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef40ab9d0 con 0x7eff040cac80 2024-05-27T00:18:18.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff09e00640 1 -- 172.21.15.98:0/836812582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040cac80 msgr2=0x7eff040c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:18.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff09e00640 1 --2- 172.21.15.98:0/836812582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040cac80 0x7eff040c9100 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efef4009870 tx=0x7efef401c450 comp rx=0 tx=0).stop 2024-05-27T00:18:18.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff09e00640 1 -- 172.21.15.98:0/836812582 shutdown_connections 2024-05-27T00:18:18.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff09e00640 1 --2- 172.21.15.98:0/836812582 >> v2:172.21.15.204:3300/0 conn(0x7eff0415bb00 0x7eff041523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:18.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff09e00640 1 --2- 172.21.15.98:0/836812582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040cac80 0x7eff040c9100 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:18.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff09e00640 1 -- 172.21.15.98:0/836812582 >> 172.21.15.98:0/836812582 conn(0x7eff04070b30 msgr2=0x7eff04070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:18.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff09e00640 1 -- 172.21.15.98:0/836812582 shutdown_connections 2024-05-27T00:18:18.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff09e00640 1 -- 172.21.15.98:0/836812582 wait complete. 2024-05-27T00:18:18.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff09e00640 1 Processor -- start 2024-05-27T00:18:18.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.992+0000 7eff09e00640 1 -- start start 2024-05-27T00:18:18.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040c9640 0x7eff04146250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:18.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff040cac80 0x7eff0414f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:18.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff04155480 0x7eff04143770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:18.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff04158590 con 0x7eff040c9640 2024-05-27T00:18:18.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff040b40f0 con 0x7eff040cac80 2024-05-27T00:18:18.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7eff0406e990 con 0x7eff04155480 2024-05-27T00:18:18.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff08a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040c9640 0x7eff04146250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:18.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff08a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040c9640 0x7eff04146250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38342/0 (socket says 172.21.15.98:38342) 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff08a00640 1 -- 172.21.15.98:0/730663399 learned_addr learned my addr 172.21.15.98:0/730663399 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7efefbe00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff040cac80 0x7eff0414f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09400640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff04155480 0x7eff04143770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff08a00640 1 -- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff04155480 msgr2=0x7eff04143770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff08a00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff04155480 0x7eff04143770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff08a00640 1 -- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff040cac80 msgr2=0x7eff0414f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff08a00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff040cac80 0x7eff0414f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff08a00640 1 -- 172.21.15.98:0/730663399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff040c6130 con 0x7eff040c9640 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7efefbe00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff040cac80 0x7eff0414f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09400640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff04155480 0x7eff04143770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff08a00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040c9640 0x7eff04146250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7efef401a080 tx=0x7efef40191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff01600640 1 -- 172.21.15.98:0/730663399 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef40b9070 con 0x7eff040c9640 2024-05-27T00:18:19.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff01600640 1 -- 172.21.15.98:0/730663399 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef40b4420 con 0x7eff040c9640 2024-05-27T00:18:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff01600640 1 -- 172.21.15.98:0/730663399 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef40b0040 con 0x7eff040c9640 2024-05-27T00:18:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09e00640 1 -- 172.21.15.98:0/730663399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff040c63e0 con 0x7eff040c9640 2024-05-27T00:18:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09e00640 1 -- 172.21.15.98:0/730663399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff040c69a0 con 0x7eff040c9640 2024-05-27T00:18:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff09e00640 1 -- 172.21.15.98:0/730663399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff04075c40 con 0x7eff040c9640 2024-05-27T00:18:19.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:18.996+0000 7eff01600640 1 -- 172.21.15.98:0/730663399 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efef40aa3a0 con 0x7eff040c9640 2024-05-27T00:18:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.000+0000 7eff01600640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efee00678f0 0x7efee0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.000+0000 7eff01600640 1 -- 172.21.15.98:0/730663399 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5939+0+0 (secure 0 0 0) 0x7efef411ef70 con 0x7eff040c9640 2024-05-27T00:18:19.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.000+0000 7efefbe00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efee00678f0 0x7efee0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:19.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.000+0000 7efefbe00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efee00678f0 0x7efee0069db0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7eff04147970 tx=0x7efeec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:19.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.004+0000 7eff01600640 1 -- 172.21.15.98:0/730663399 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7efef40160d0 con 0x7eff040c9640 2024-05-27T00:18:19.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.192+0000 7eff09e00640 1 -- 172.21.15.98:0/730663399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff040b3e40 con 0x7eff040c9640 2024-05-27T00:18:19.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.192+0000 7eff01600640 1 -- 172.21.15.98:0/730663399 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6838 (secure 0 0 0) 0x7efef40ef9b0 con 0x7eff040c9640 2024-05-27T00:18:19.197 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:19.197 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:18:19.197 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-05-27T00:18:19.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 -- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efee00678f0 msgr2=0x7efee0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:19.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efee00678f0 0x7efee0069db0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7eff04147970 tx=0x7efeec008040 comp rx=0 tx=0).stop 2024-05-27T00:18:19.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 -- 172.21.15.98:0/730663399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040c9640 msgr2=0x7eff04146250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:19.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040c9640 0x7eff04146250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7efef401a080 tx=0x7efef40191f0 comp rx=0 tx=0).stop 2024-05-27T00:18:19.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 -- 172.21.15.98:0/730663399 shutdown_connections 2024-05-27T00:18:19.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff04155480 0x7eff04143770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:19.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efee00678f0 0x7efee0069db0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:19.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff040cac80 0x7eff0414f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:19.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 --2- 172.21.15.98:0/730663399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff040c9640 0x7eff04146250 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:19.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 -- 172.21.15.98:0/730663399 >> 172.21.15.98:0/730663399 conn(0x7eff04070b30 msgr2=0x7eff040ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:19.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 -- 172.21.15.98:0/730663399 shutdown_connections 2024-05-27T00:18:19.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:19.196+0000 7eff09e00640 1 -- 172.21.15.98:0/730663399 wait complete. 2024-05-27T00:18:19.219 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:18:21.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:21.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f906e000640 1 Processor -- start 2024-05-27T00:18:21.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f906e000640 1 -- start start 2024-05-27T00:18:21.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f906e000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f90681511d0 0x7f9068070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:21.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f906e000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f90680cc8f0 0x7f90680c1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:21.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f906e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90680c8670 0x7f90680c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:21.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f906e000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f90680b1dd0 con 0x7f90680cc8f0 2024-05-27T00:18:21.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f906e000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f90680c1000 con 0x7f90681511d0 2024-05-27T00:18:21.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f906e000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f90681515b0 con 0x7f90680c8670 2024-05-27T00:18:21.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f9066a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f90680cc8f0 0x7f90680c1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:21.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f9066a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f90680cc8f0 0x7f90680c1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38350/0 (socket says 172.21.15.98:38350) 2024-05-27T00:18:21.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f9066a00640 1 -- 172.21.15.98:0/1771523225 learned_addr learned my addr 172.21.15.98:0/1771523225 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:21.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f9067e00640 1 --2- 172.21.15.98:0/1771523225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90680c8670 0x7f90680c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:21.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.364+0000 7f9067400640 1 --2- 172.21.15.98:0/1771523225 >> v2:172.21.15.204:3300/0 conn(0x7f90681511d0 0x7f9068070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:21.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.368+0000 7f9066a00640 1 -- 172.21.15.98:0/1771523225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90680c8670 msgr2=0x7f90680c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:21.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.368+0000 7f9066a00640 1 --2- 172.21.15.98:0/1771523225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90680c8670 0x7f90680c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.368+0000 7f9066a00640 1 -- 172.21.15.98:0/1771523225 >> v2:172.21.15.204:3300/0 conn(0x7f90681511d0 msgr2=0x7f9068070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:21.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.368+0000 7f9066a00640 1 --2- 172.21.15.98:0/1771523225 >> v2:172.21.15.204:3300/0 conn(0x7f90681511d0 0x7f9068070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.368+0000 7f9066a00640 1 -- 172.21.15.98:0/1771523225 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90680c83c0 con 0x7f90680cc8f0 2024-05-27T00:18:21.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.368+0000 7f9067400640 1 --2- 172.21.15.98:0/1771523225 >> v2:172.21.15.204:3300/0 conn(0x7f90681511d0 0x7f9068070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:18:21.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.368+0000 7f9066a00640 1 --2- 172.21.15.98:0/1771523225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90680cc8f0 0x7f90680c1310 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f90680cc880 tx=0x7f905401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c75c5385df840dc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:21.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.368+0000 7f9066000640 1 -- 172.21.15.98:0/1771523225 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90540b0070 con 0x7f90680cc8f0 2024-05-27T00:18:21.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.368+0000 7f9066000640 1 -- 172.21.15.98:0/1771523225 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90540ab600 con 0x7f90680cc8f0 2024-05-27T00:18:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.368+0000 7f9066000640 1 -- 172.21.15.98:0/1771523225 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90540a4e60 con 0x7f90680cc8f0 2024-05-27T00:18:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 -- 172.21.15.98:0/1771523225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90680cc8f0 msgr2=0x7f90680c1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 --2- 172.21.15.98:0/1771523225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90680cc8f0 0x7f90680c1310 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f90680cc880 tx=0x7f905401c490 comp rx=0 tx=0).stop 2024-05-27T00:18:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 -- 172.21.15.98:0/1771523225 shutdown_connections 2024-05-27T00:18:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 --2- 172.21.15.98:0/1771523225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90680c8670 0x7f90680c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 --2- 172.21.15.98:0/1771523225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90680cc8f0 0x7f90680c1310 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 --2- 172.21.15.98:0/1771523225 >> v2:172.21.15.204:3300/0 conn(0x7f90681511d0 0x7f9068070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 -- 172.21.15.98:0/1771523225 >> 172.21.15.98:0/1771523225 conn(0x7f9068067530 msgr2=0x7f906806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:21.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 -- 172.21.15.98:0/1771523225 shutdown_connections 2024-05-27T00:18:21.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 -- 172.21.15.98:0/1771523225 wait complete. 2024-05-27T00:18:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 Processor -- start 2024-05-27T00:18:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 -- start start 2024-05-27T00:18:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90680c8670 0x7f90680ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90680cc8f0 0x7f9068152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90681511d0 0x7f906813e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:21.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90680c22d0 con 0x7f90681511d0 2024-05-27T00:18:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f90680c2150 con 0x7f90680cc8f0 2024-05-27T00:18:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f906e000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f90680c1850 con 0x7f90680c8670 2024-05-27T00:18:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9067e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90681511d0 0x7f906813e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9067e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90681511d0 0x7f906813e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38366/0 (socket says 172.21.15.98:38366) 2024-05-27T00:18:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9067e00640 1 -- 172.21.15.98:0/292069263 learned_addr learned my addr 172.21.15.98:0/292069263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9066a00640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90680cc8f0 0x7f9068152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9067400640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90680c8670 0x7f90680ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:21.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9067e00640 1 -- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90680c8670 msgr2=0x7f90680ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9067e00640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90680c8670 0x7f90680ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9067e00640 1 -- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90680cc8f0 msgr2=0x7f9068152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9067e00640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90680cc8f0 0x7f9068152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9067e00640 1 -- 172.21.15.98:0/292069263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90681f4220 con 0x7f90681511d0 2024-05-27T00:18:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.372+0000 7f9067e00640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90681511d0 0x7f906813e390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f906000ef00 tx=0x7f906000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:21.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.376+0000 7f905f400640 1 -- 172.21.15.98:0/292069263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060018020 con 0x7f90681511d0 2024-05-27T00:18:21.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.376+0000 7f905f400640 1 -- 172.21.15.98:0/292069263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9060014400 con 0x7f90681511d0 2024-05-27T00:18:21.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.376+0000 7f905f400640 1 -- 172.21.15.98:0/292069263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90600147f0 con 0x7f90681511d0 2024-05-27T00:18:21.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.376+0000 7f906e000640 1 -- 172.21.15.98:0/292069263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90681f44c0 con 0x7f90681511d0 2024-05-27T00:18:21.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.376+0000 7f906e000640 1 -- 172.21.15.98:0/292069263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9068151d60 con 0x7f90681511d0 2024-05-27T00:18:21.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.376+0000 7f906e000640 1 -- 172.21.15.98:0/292069263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9068075cb0 con 0x7f90681511d0 2024-05-27T00:18:21.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.376+0000 7f905f400640 1 -- 172.21.15.98:0/292069263 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f906001b020 con 0x7f90681511d0 2024-05-27T00:18:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.376+0000 7f905f400640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f90480678c0 0x7f9048069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.376+0000 7f905f400640 1 -- 172.21.15.98:0/292069263 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5939+0+0 (secure 0 0 0) 0x7f90600866b0 con 0x7f90681511d0 2024-05-27T00:18:21.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.376+0000 7f9067400640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f90480678c0 0x7f9048069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:21.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.380+0000 7f9067400640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f90480678c0 0x7f9048069d80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f905800ba10 tx=0x7f9058008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:21.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.380+0000 7f905f400640 1 -- 172.21.15.98:0/292069263 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9060022030 con 0x7f90681511d0 2024-05-27T00:18:21.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 -- 172.21.15.98:0/292069263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9068154040 con 0x7f90681511d0 2024-05-27T00:18:21.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f905f400640 1 -- 172.21.15.98:0/292069263 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6838 (secure 0 0 0) 0x7f9060057060 con 0x7f90681511d0 2024-05-27T00:18:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:21.579 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:18:21.579 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-05-27T00:18:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 -- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f90480678c0 msgr2=0x7f9048069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f90480678c0 0x7f9048069d80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f905800ba10 tx=0x7f9058008040 comp rx=0 tx=0).stop 2024-05-27T00:18:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 -- 172.21.15.98:0/292069263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90681511d0 msgr2=0x7f906813e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90681511d0 0x7f906813e390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f906000ef00 tx=0x7f906000b480 comp rx=0 tx=0).stop 2024-05-27T00:18:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 -- 172.21.15.98:0/292069263 shutdown_connections 2024-05-27T00:18:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90681511d0 0x7f906813e390 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f90480678c0 0x7f9048069d80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90680cc8f0 0x7f9068152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 --2- 172.21.15.98:0/292069263 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90680c8670 0x7f90680ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 -- 172.21.15.98:0/292069263 >> 172.21.15.98:0/292069263 conn(0x7f9068067530 msgr2=0x7f90680c62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 -- 172.21.15.98:0/292069263 shutdown_connections 2024-05-27T00:18:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:21.576+0000 7f906e000640 1 -- 172.21.15.98:0/292069263 wait complete. 2024-05-27T00:18:21.601 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:18:23.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:23.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.744+0000 7fb0c2a00640 1 Processor -- start 2024-05-27T00:18:23.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.744+0000 7fb0c2a00640 1 -- start start 2024-05-27T00:18:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc0c6a60 0x7fb0bc073340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c2a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb0bc0c6e40 0x7fb0bc158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c2a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0bc073880 con 0x7fb0bc151f00 2024-05-27T00:18:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c2a00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb0bc159300 con 0x7fb0bc0c6e40 2024-05-27T00:18:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c2a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb0bc15bb10 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c2000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb0bc151f00 0x7fb0bc070e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35520/0 (socket says 172.21.15.98:35520) 2024-05-27T00:18:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c2000640 1 -- 172.21.15.98:0/736711424 learned_addr learned my addr 172.21.15.98:0/736711424 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:23.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c0c00640 1 --2- 172.21.15.98:0/736711424 >> v2:172.21.15.204:3300/0 conn(0x7fb0bc0c6e40 0x7fb0bc158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c1600640 1 --2- 172.21.15.98:0/736711424 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc0c6a60 0x7fb0bc073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0bbe00640 1 -- 172.21.15.98:0/736711424 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 52930311 0 0) 0x7fb0bc073880 con 0x7fb0bc151f00 2024-05-27T00:18:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0bbe00640 1 -- 172.21.15.98:0/736711424 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0a4003530 con 0x7fb0bc151f00 2024-05-27T00:18:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c0c00640 1 -- 172.21.15.98:0/736711424 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc0c6a60 msgr2=0x7fb0bc073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c0c00640 1 --2- 172.21.15.98:0/736711424 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc0c6a60 0x7fb0bc073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c0c00640 1 -- 172.21.15.98:0/736711424 >> v1:172.21.15.98:6789/0 conn(0x7fb0bc151f00 legacy=0x7fb0bc070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c0c00640 1 -- 172.21.15.98:0/736711424 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0bc15bdf0 con 0x7fb0bc0c6e40 2024-05-27T00:18:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.748+0000 7fb0c1600640 1 --2- 172.21.15.98:0/736711424 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc0c6a60 0x7fb0bc073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:18:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c0c00640 1 --2- 172.21.15.98:0/736711424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0bc0c6e40 0x7fb0bc158dc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80097c0 tx=0x7fb0a801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba15716b433f97a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:23.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0bbe00640 1 -- 172.21.15.98:0/736711424 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80b0070 con 0x7fb0bc0c6e40 2024-05-27T00:18:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0bbe00640 1 -- 172.21.15.98:0/736711424 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a80ab600 con 0x7fb0bc0c6e40 2024-05-27T00:18:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0bbe00640 1 -- 172.21.15.98:0/736711424 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80ab9b0 con 0x7fb0bc0c6e40 2024-05-27T00:18:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c2a00640 1 -- 172.21.15.98:0/736711424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0bc0c6e40 msgr2=0x7fb0bc158dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c2a00640 1 --2- 172.21.15.98:0/736711424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0bc0c6e40 0x7fb0bc158dc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80097c0 tx=0x7fb0a801c4a0 comp rx=0 tx=0).stop 2024-05-27T00:18:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c2a00640 1 -- 172.21.15.98:0/736711424 shutdown_connections 2024-05-27T00:18:23.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c2a00640 1 --2- 172.21.15.98:0/736711424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0bc0c6e40 0x7fb0bc158dc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:23.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c2a00640 1 --2- 172.21.15.98:0/736711424 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc0c6a60 0x7fb0bc073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:23.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c2a00640 1 -- 172.21.15.98:0/736711424 >> 172.21.15.98:0/736711424 conn(0x7fb0bc0cbe00 msgr2=0x7fb0bc0cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:23.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c2a00640 1 -- 172.21.15.98:0/736711424 shutdown_connections 2024-05-27T00:18:23.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c2a00640 1 -- 172.21.15.98:0/736711424 wait complete. 2024-05-27T00:18:23.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c2a00640 1 Processor -- start 2024-05-27T00:18:23.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.752+0000 7fb0c2a00640 1 -- start start 2024-05-27T00:18:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c2a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc0c6a60 0x7fb0bc1f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c2a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0bc0c6e40 0x7fb0bc142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc151f00 0x7fb0bc15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c2a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc1528a0 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c2a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc152720 con 0x7fb0bc0c6e40 2024-05-27T00:18:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c2a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb0bc165520 con 0x7fb0bc151f00 2024-05-27T00:18:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c1600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc0c6a60 0x7fb0bc1f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c1600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc0c6a60 0x7fb0bc1f4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37580/0 (socket says 172.21.15.98:37580) 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c1600640 1 -- 172.21.15.98:0/3354803045 learned_addr learned my addr 172.21.15.98:0/3354803045 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c0c00640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0bc0c6e40 0x7fb0bc142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c2000640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc151f00 0x7fb0bc15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c1600640 1 -- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc151f00 msgr2=0x7fb0bc15a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c1600640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc151f00 0x7fb0bc15a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c1600640 1 -- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0bc0c6e40 msgr2=0x7fb0bc142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c1600640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0bc0c6e40 0x7fb0bc142150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c1600640 1 -- 172.21.15.98:0/3354803045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0bc1f5150 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c0c00640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0bc0c6e40 0x7fb0bc142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c1600640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc0c6a60 0x7fb0bc1f4c10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000e9b0 tx=0x7fb0b000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0b9600640 1 -- 172.21.15.98:0/3354803045 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0017070 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0b9600640 1 -- 172.21.15.98:0/3354803045 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b000ca00 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0b9600640 1 -- 172.21.15.98:0/3354803045 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b000cdd0 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c2a00640 1 -- 172.21.15.98:0/3354803045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0bc0cb7b0 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c2a00640 1 -- 172.21.15.98:0/3354803045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0bc147bb0 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0c2a00640 1 -- 172.21.15.98:0/3354803045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0bc075c40 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.756+0000 7fb0b9600640 1 -- 172.21.15.98:0/3354803045 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb0b001a070 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.760+0000 7fb0b9600640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb09c0678c0 0x7fb09c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.760+0000 7fb0b9600640 1 -- 172.21.15.98:0/3354803045 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5939+0+0 (secure 0 0 0) 0x7fb0b0085fe0 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.760+0000 7fb0c0c00640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb09c0678c0 0x7fb09c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.760+0000 7fb0c0c00640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb09c0678c0 0x7fb09c069d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80ad000 tx=0x7fb0a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.764+0000 7fb0b9600640 1 -- 172.21.15.98:0/3354803045 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb0b0056a40 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.948+0000 7fb0c2a00640 1 -- 172.21.15.98:0/3354803045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0bc0c6060 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0b9600640 1 -- 172.21.15.98:0/3354803045 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+6838 (secure 0 0 0) 0x7fb0b0056340 con 0x7fb0bc0c6a60 2024-05-27T00:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4894,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:16:40.435698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6834/3903574515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3903574515},{"type":"v1","addr":"172.21.15.98:6834","nonce":3903574515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-05-27T00:18:23.954 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-05-27T00:18:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 -- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb09c0678c0 msgr2=0x7fb09c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb09c0678c0 0x7fb09c069d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80ad000 tx=0x7fb0a8008040 comp rx=0 tx=0).stop 2024-05-27T00:18:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 -- 172.21.15.98:0/3354803045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc0c6a60 msgr2=0x7fb0bc1f4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc0c6a60 0x7fb0bc1f4c10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000e9b0 tx=0x7fb0b000ee70 comp rx=0 tx=0).stop 2024-05-27T00:18:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 -- 172.21.15.98:0/3354803045 shutdown_connections 2024-05-27T00:18:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0bc151f00 0x7fb0bc15a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb09c0678c0 0x7fb09c069d80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0bc0c6e40 0x7fb0bc142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:23.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 --2- 172.21.15.98:0/3354803045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc0c6a60 0x7fb0bc1f4c10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:23.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 -- 172.21.15.98:0/3354803045 >> 172.21.15.98:0/3354803045 conn(0x7fb0bc0cbe00 msgr2=0x7fb0bc070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:23.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 -- 172.21.15.98:0/3354803045 shutdown_connections 2024-05-27T00:18:23.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:23.952+0000 7fb0c2a00640 1 -- 172.21.15.98:0/3354803045 wait complete. 2024-05-27T00:18:23.975 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:18:25.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:26.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f2455200640 1 Processor -- start 2024-05-27T00:18:26.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f2455200640 1 -- start start 2024-05-27T00:18:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f2455200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f24500735d0 0x7f2450151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f2455200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f2450151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f2455200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24500c17d0 con 0x7f24500731f0 2024-05-27T00:18:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f2455200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f2450071990 con 0x7f24500735d0 2024-05-27T00:18:26.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f2455200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2450070e10 con 0x7f24500c79f0 2024-05-27T00:18:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244d400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f24500735d0 0x7f2450151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244e800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f2450151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244de00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f24500731f0 0x7f24500c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:37268/0 (socket says 172.21.15.98:37268) 2024-05-27T00:18:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244e800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f2450151970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:47220/0 (socket says 172.21.15.98:47220) 2024-05-27T00:18:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244de00640 1 -- 172.21.15.98:0/2381513727 learned_addr learned my addr 172.21.15.98:0/2381513727 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244d400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f24500735d0 0x7f2450151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:39202/0 (socket says 172.21.15.98:39202) 2024-05-27T00:18:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244e800640 1 -- 172.21.15.98:0/2381513727 >> v2:172.21.15.204:3300/0 conn(0x7f24500735d0 msgr2=0x7f2450151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244e800640 1 --2- 172.21.15.98:0/2381513727 >> v2:172.21.15.204:3300/0 conn(0x7f24500735d0 0x7f2450151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244e800640 1 -- 172.21.15.98:0/2381513727 >> v1:172.21.15.204:6789/0 conn(0x7f24500731f0 legacy=0x7f24500c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244e800640 1 -- 172.21.15.98:0/2381513727 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24500710f0 con 0x7f24500c79f0 2024-05-27T00:18:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.120+0000 7f244d400640 1 --2- 172.21.15.98:0/2381513727 >> v2:172.21.15.204:3300/0 conn(0x7f24500735d0 0x7f2450151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:18:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f244e800640 1 --2- 172.21.15.98:0/2381513727 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f2450151970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f24480097c0 tx=0x7f244801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdae91caa914dc08 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f244ca00640 1 -- 172.21.15.98:0/2381513727 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24480b5070 con 0x7f24500c79f0 2024-05-27T00:18:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f244ca00640 1 -- 172.21.15.98:0/2381513727 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24480a8990 con 0x7f24500c79f0 2024-05-27T00:18:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f244ca00640 1 -- 172.21.15.98:0/2381513727 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24480a8d60 con 0x7f24500c79f0 2024-05-27T00:18:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f2455200640 1 -- 172.21.15.98:0/2381513727 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 msgr2=0x7f2450151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f2455200640 1 --2- 172.21.15.98:0/2381513727 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f2450151970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f24480097c0 tx=0x7f244801ec90 comp rx=0 tx=0).stop 2024-05-27T00:18:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f2455200640 1 -- 172.21.15.98:0/2381513727 shutdown_connections 2024-05-27T00:18:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f2455200640 1 --2- 172.21.15.98:0/2381513727 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f2450151970 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f2455200640 1 --2- 172.21.15.98:0/2381513727 >> v2:172.21.15.204:3300/0 conn(0x7f24500735d0 0x7f2450151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f2455200640 1 -- 172.21.15.98:0/2381513727 >> 172.21.15.98:0/2381513727 conn(0x7f2450067530 msgr2=0x7f245006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f2455200640 1 -- 172.21.15.98:0/2381513727 shutdown_connections 2024-05-27T00:18:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.124+0000 7f2455200640 1 -- 172.21.15.98:0/2381513727 wait complete. 2024-05-27T00:18:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2455200640 1 Processor -- start 2024-05-27T00:18:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2455200640 1 -- start start 2024-05-27T00:18:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2455200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24500731f0 0x7f2450159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2455200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24500735d0 0x7f2450159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2455200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f24501434b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2455200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24500c8400 con 0x7f24500731f0 2024-05-27T00:18:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2455200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f24500716b0 con 0x7f24500735d0 2024-05-27T00:18:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2455200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2450157970 con 0x7f24500c79f0 2024-05-27T00:18:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24500731f0 0x7f2450159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24500731f0 0x7f2450159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37588/0 (socket says 172.21.15.98:37588) 2024-05-27T00:18:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244de00640 1 -- 172.21.15.98:0/1005231668 learned_addr learned my addr 172.21.15.98:0/1005231668 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244e800640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f24501434b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244d400640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24500735d0 0x7f2450159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244de00640 1 -- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 msgr2=0x7f24501434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244de00640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f24501434b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244de00640 1 -- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24500735d0 msgr2=0x7f2450159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244de00640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24500735d0 0x7f2450159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244de00640 1 -- 172.21.15.98:0/1005231668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2450144e90 con 0x7f24500731f0 2024-05-27T00:18:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244e800640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f24501434b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:18:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244d400640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24500735d0 0x7f2450159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:18:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f244de00640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24500731f0 0x7f2450159440 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f244000cbd0 tx=0x7f244000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2446000640 1 -- 172.21.15.98:0/1005231668 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440017070 con 0x7f24500731f0 2024-05-27T00:18:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2446000640 1 -- 172.21.15.98:0/1005231668 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f244000ad40 con 0x7f24500731f0 2024-05-27T00:18:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2455200640 1 -- 172.21.15.98:0/1005231668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2450145130 con 0x7f24500731f0 2024-05-27T00:18:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2446000640 1 -- 172.21.15.98:0/1005231668 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440012600 con 0x7f24500731f0 2024-05-27T00:18:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.128+0000 7f2455200640 1 -- 172.21.15.98:0/1005231668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24501ecc10 con 0x7f24500731f0 2024-05-27T00:18:26.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.132+0000 7f2455200640 1 -- 172.21.15.98:0/1005231668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2450075cb0 con 0x7f24500731f0 2024-05-27T00:18:26.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.132+0000 7f2446000640 1 -- 172.21.15.98:0/1005231668 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f244001a070 con 0x7f24500731f0 2024-05-27T00:18:26.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.132+0000 7f2446000640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24300678f0 0x7f2430069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.132+0000 7f2446000640 1 -- 172.21.15.98:0/1005231668 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f2440085e20 con 0x7f24500731f0 2024-05-27T00:18:26.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.132+0000 7f244d400640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24300678f0 0x7f2430069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.136+0000 7f244d400640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24300678f0 0x7f2430069db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24500b2370 tx=0x7f243c00b3a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:26.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.136+0000 7f2446000640 1 -- 172.21.15.98:0/1005231668 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f24400560c0 con 0x7f24500731f0 2024-05-27T00:18:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.328+0000 7f2455200640 1 -- 172.21.15.98:0/1005231668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f245006eba0 con 0x7f24500731f0 2024-05-27T00:18:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.328+0000 7f2446000640 1 -- 172.21.15.98:0/1005231668 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6006 (secure 0 0 0) 0x7f244005a3c0 con 0x7f24500731f0 2024-05-27T00:18:26.333 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:26.333 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:25.179272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:18:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 14 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 -- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24300678f0 msgr2=0x7f2430069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24300678f0 0x7f2430069db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24500b2370 tx=0x7f243c00b3a0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 -- 172.21.15.98:0/1005231668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24500731f0 msgr2=0x7f2450159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24500731f0 0x7f2450159440 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f244000cbd0 tx=0x7f244000a430 comp rx=0 tx=0).stop 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 -- 172.21.15.98:0/1005231668 shutdown_connections 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24500c79f0 0x7f24501434b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24300678f0 0x7f2430069db0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24500735d0 0x7f2450159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 --2- 172.21.15.98:0/1005231668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24500731f0 0x7f2450159440 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 -- 172.21.15.98:0/1005231668 >> 172.21.15.98:0/1005231668 conn(0x7f2450067530 msgr2=0x7f245014f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 -- 172.21.15.98:0/1005231668 shutdown_connections 2024-05-27T00:18:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.332+0000 7f2455200640 1 -- 172.21.15.98:0/1005231668 wait complete. 2024-05-27T00:18:26.354 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-05-27T00:18:26.354 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:18:26.355 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:26.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.488+0000 7f01c5c00640 1 Processor -- start 2024-05-27T00:18:26.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.488+0000 7f01c5c00640 1 -- start start 2024-05-27T00:18:26.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.488+0000 7f01c5c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01c00731f0 0x7f01c00c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.488+0000 7f01c5c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01c00735d0 0x7f01c0151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.488+0000 7f01c5c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f01c00c79f0 0x7f01c0151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.488+0000 7f01c5c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f01c0071990 con 0x7f01c00735d0 2024-05-27T00:18:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.488+0000 7f01c5c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f01c00c17d0 con 0x7f01c00c79f0 2024-05-27T00:18:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.488+0000 7f01c5c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f01c0070e10 con 0x7f01c00731f0 2024-05-27T00:18:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.488+0000 7f01b7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01c00735d0 0x7f01c0151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01b7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f01c00735d0 0x7f01c0151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37594/0 (socket says 172.21.15.98:37594) 2024-05-27T00:18:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01b7e00640 1 -- 172.21.15.98:0/1530549682 learned_addr learned my addr 172.21.15.98:0/1530549682 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01bea00640 1 --2- 172.21.15.98:0/1530549682 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01c00731f0 0x7f01c00c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01bf400640 1 --2- 172.21.15.98:0/1530549682 >> v2:172.21.15.204:3300/0 conn(0x7f01c00c79f0 0x7f01c0151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01b7e00640 1 -- 172.21.15.98:0/1530549682 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01c00731f0 msgr2=0x7f01c00c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:26.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01b7e00640 1 --2- 172.21.15.98:0/1530549682 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01c00731f0 0x7f01c00c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01b7e00640 1 -- 172.21.15.98:0/1530549682 >> v2:172.21.15.204:3300/0 conn(0x7f01c00c79f0 msgr2=0x7f01c0151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:26.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01b7e00640 1 --2- 172.21.15.98:0/1530549682 >> v2:172.21.15.204:3300/0 conn(0x7f01c00c79f0 0x7f01c0151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01b7e00640 1 -- 172.21.15.98:0/1530549682 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c00710f0 con 0x7f01c00735d0 2024-05-27T00:18:26.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01b7e00640 1 --2- 172.21.15.98:0/1530549682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00735d0 0x7f01c0151430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f01c00c7e80 tx=0x7f01a80213e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4bb2fec87a4a670 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:26.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01be000640 1 -- 172.21.15.98:0/1530549682 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a80b6070 con 0x7f01c00735d0 2024-05-27T00:18:26.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01be000640 1 -- 172.21.15.98:0/1530549682 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01a80b1600 con 0x7f01c00735d0 2024-05-27T00:18:26.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.492+0000 7f01be000640 1 -- 172.21.15.98:0/1530549682 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a80b19d0 con 0x7f01c00735d0 2024-05-27T00:18:26.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 -- 172.21.15.98:0/1530549682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00735d0 msgr2=0x7f01c0151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 --2- 172.21.15.98:0/1530549682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00735d0 0x7f01c0151430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f01c00c7e80 tx=0x7f01a80213e0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 -- 172.21.15.98:0/1530549682 shutdown_connections 2024-05-27T00:18:26.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 --2- 172.21.15.98:0/1530549682 >> v2:172.21.15.204:3300/0 conn(0x7f01c00c79f0 0x7f01c0151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 --2- 172.21.15.98:0/1530549682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00735d0 0x7f01c0151430 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 --2- 172.21.15.98:0/1530549682 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01c00731f0 0x7f01c00c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 -- 172.21.15.98:0/1530549682 >> 172.21.15.98:0/1530549682 conn(0x7f01c0067530 msgr2=0x7f01c006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:26.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 -- 172.21.15.98:0/1530549682 shutdown_connections 2024-05-27T00:18:26.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 -- 172.21.15.98:0/1530549682 wait complete. 2024-05-27T00:18:26.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 Processor -- start 2024-05-27T00:18:26.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 -- start start 2024-05-27T00:18:26.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c00731f0 0x7f01c01527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01c00735d0 0x7f01c01ef5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00c79f0 0x7f01c0153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01c00c8400 con 0x7f01c00c79f0 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f01c00716b0 con 0x7f01c00731f0 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01c5c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f01c00c13f0 con 0x7f01c00735d0 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01bf400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00c79f0 0x7f01c0153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01bf400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00c79f0 0x7f01c0153db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37608/0 (socket says 172.21.15.98:37608) 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01bf400640 1 -- 172.21.15.98:0/2296698798 learned_addr learned my addr 172.21.15.98:0/2296698798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01bea00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c00731f0 0x7f01c01527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01b7e00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01c00735d0 0x7f01c01ef5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01bf400640 1 -- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01c00735d0 msgr2=0x7f01c01ef5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01bf400640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01c00735d0 0x7f01c01ef5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01bf400640 1 -- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c00731f0 msgr2=0x7f01c01527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01bf400640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c00731f0 0x7f01c01527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01bf400640 1 -- 172.21.15.98:0/2296698798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c01efed0 con 0x7f01c00c79f0 2024-05-27T00:18:26.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01bf400640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00c79f0 0x7f01c0153db0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f01b8007e60 tx=0x7f01b800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:26.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.496+0000 7f01b6a00640 1 -- 172.21.15.98:0/2296698798 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8014420 con 0x7f01c00c79f0 2024-05-27T00:18:26.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.500+0000 7f01b6a00640 1 -- 172.21.15.98:0/2296698798 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01b8019070 con 0x7f01c00c79f0 2024-05-27T00:18:26.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.500+0000 7f01b6a00640 1 -- 172.21.15.98:0/2296698798 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b8014820 con 0x7f01c00c79f0 2024-05-27T00:18:26.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.500+0000 7f01c5c00640 1 -- 172.21.15.98:0/2296698798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c01f01b0 con 0x7f01c00c79f0 2024-05-27T00:18:26.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.500+0000 7f01c5c00640 1 -- 172.21.15.98:0/2296698798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c01532d0 con 0x7f01c00c79f0 2024-05-27T00:18:26.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.500+0000 7f01c5c00640 1 -- 172.21.15.98:0/2296698798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c0075cb0 con 0x7f01c00c79f0 2024-05-27T00:18:26.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.500+0000 7f01b6a00640 1 -- 172.21.15.98:0/2296698798 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f01b80149c0 con 0x7f01c00c79f0 2024-05-27T00:18:26.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.500+0000 7f01b6a00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f019c0678f0 0x7f019c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:26.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.500+0000 7f01b6a00640 1 -- 172.21.15.98:0/2296698798 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f01b8087400 con 0x7f01c00c79f0 2024-05-27T00:18:26.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.500+0000 7f01bea00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f019c0678f0 0x7f019c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:26.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.504+0000 7f01bea00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f019c0678f0 0x7f019c069db0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f01b000ba30 tx=0x7f01b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.504+0000 7f01b6a00640 1 -- 172.21.15.98:0/2296698798 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f01b8057f00 con 0x7f01c00c79f0 2024-05-27T00:18:26.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 -- 172.21.15.98:0/2296698798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01c0158510 con 0x7f01c00c79f0 2024-05-27T00:18:26.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01b6a00640 1 -- 172.21.15.98:0/2296698798 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6006 (secure 0 0 0) 0x7f01b805e0d0 con 0x7f01c00c79f0 2024-05-27T00:18:26.699 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:26.699 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:25.179272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:18:26.700 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 14 2024-05-27T00:18:26.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 -- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f019c0678f0 msgr2=0x7f019c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f019c0678f0 0x7f019c069db0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f01b000ba30 tx=0x7f01b0008040 comp rx=0 tx=0).stop 2024-05-27T00:18:26.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 -- 172.21.15.98:0/2296698798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00c79f0 msgr2=0x7f01c0153db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:26.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00c79f0 0x7f01c0153db0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f01b8007e60 tx=0x7f01b800e4b0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 -- 172.21.15.98:0/2296698798 shutdown_connections 2024-05-27T00:18:26.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01c00c79f0 0x7f01c0153db0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01c00735d0 0x7f01c01ef5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f019c0678f0 0x7f019c069db0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 --2- 172.21.15.98:0/2296698798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c00731f0 0x7f01c01527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:26.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 -- 172.21.15.98:0/2296698798 >> 172.21.15.98:0/2296698798 conn(0x7f01c0067530 msgr2=0x7f01c0151200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:26.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 -- 172.21.15.98:0/2296698798 shutdown_connections 2024-05-27T00:18:26.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:26.696+0000 7f01c5c00640 1 -- 172.21.15.98:0/2296698798 wait complete. 2024-05-27T00:18:26.720 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 14, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 14, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4894": { "addr": "172.21.15.98:6837/1781395818", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 1781395818, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 1781395818, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4894, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:18:25.179272+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4894 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:18:28.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:28.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.864+0000 7fe1fea00640 1 Processor -- start 2024-05-27T00:18:28.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.864+0000 7fe1fea00640 1 -- start start 2024-05-27T00:18:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1fea00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe1f80731f0 0x7fe1f80c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1fea00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1f80735d0 0x7fe1f8151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1fea00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1f80c17d0 con 0x7fe1f80c79f0 2024-05-27T00:18:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1fea00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fe1f8071990 con 0x7fe1f80731f0 2024-05-27T00:18:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1fea00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe1f8070e10 con 0x7fe1f80735d0 2024-05-27T00:18:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1fca00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe1f80c79f0 0x7fe1f8157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35534/0 (socket says 172.21.15.98:35534) 2024-05-27T00:18:28.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1fca00640 1 -- 172.21.15.98:0/3346318654 learned_addr learned my addr 172.21.15.98:0/3346318654 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1f7e00640 1 --2- 172.21.15.98:0/3346318654 >> v2:172.21.15.204:3300/0 conn(0x7fe1f80731f0 0x7fe1f80c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1f7400640 1 --2- 172.21.15.98:0/3346318654 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1f80735d0 0x7fe1f8151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1f6a00640 1 -- 172.21.15.98:0/3346318654 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2166756724 0 0) 0x7fe1f80c17d0 con 0x7fe1f80c79f0 2024-05-27T00:18:28.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1f6a00640 1 -- 172.21.15.98:0/3346318654 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1dc003530 con 0x7fe1f80c79f0 2024-05-27T00:18:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1f7e00640 1 -- 172.21.15.98:0/3346318654 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1f80735d0 msgr2=0x7fe1f8151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1f7e00640 1 --2- 172.21.15.98:0/3346318654 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1f80735d0 0x7fe1f8151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1f7e00640 1 -- 172.21.15.98:0/3346318654 >> v1:172.21.15.98:6789/0 conn(0x7fe1f80c79f0 legacy=0x7fe1f8157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.868+0000 7fe1f7e00640 1 -- 172.21.15.98:0/3346318654 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f80710f0 con 0x7fe1f80731f0 2024-05-27T00:18:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1f7e00640 1 --2- 172.21.15.98:0/3346318654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1f80731f0 0x7fe1f80c0eb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00097c0 tx=0x7fe1e001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edc9a2376a4f1097 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1f6a00640 1 -- 172.21.15.98:0/3346318654 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e00b5070 con 0x7fe1f80731f0 2024-05-27T00:18:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1f6a00640 1 -- 172.21.15.98:0/3346318654 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1e00a8990 con 0x7fe1f80731f0 2024-05-27T00:18:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1f6a00640 1 -- 172.21.15.98:0/3346318654 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e00a8d60 con 0x7fe1f80731f0 2024-05-27T00:18:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1fea00640 1 -- 172.21.15.98:0/3346318654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1f80731f0 msgr2=0x7fe1f80c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1fea00640 1 --2- 172.21.15.98:0/3346318654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1f80731f0 0x7fe1f80c0eb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00097c0 tx=0x7fe1e001ec90 comp rx=0 tx=0).stop 2024-05-27T00:18:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1fea00640 1 -- 172.21.15.98:0/3346318654 shutdown_connections 2024-05-27T00:18:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1fea00640 1 --2- 172.21.15.98:0/3346318654 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1f80735d0 0x7fe1f8151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1fea00640 1 --2- 172.21.15.98:0/3346318654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1f80731f0 0x7fe1f80c0eb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1fea00640 1 -- 172.21.15.98:0/3346318654 >> 172.21.15.98:0/3346318654 conn(0x7fe1f8067530 msgr2=0x7fe1f806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1fea00640 1 -- 172.21.15.98:0/3346318654 shutdown_connections 2024-05-27T00:18:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1fea00640 1 -- 172.21.15.98:0/3346318654 wait complete. 2024-05-27T00:18:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1fea00640 1 Processor -- start 2024-05-27T00:18:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.872+0000 7fe1fea00640 1 -- start start 2024-05-27T00:18:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fea00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1f80731f0 0x7fe1f80c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f80735d0 0x7fe1f8147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1f80c79f0 0x7fe1f8151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fea00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f80c8400 con 0x7fe1f80735d0 2024-05-27T00:18:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fea00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f80716b0 con 0x7fe1f80c79f0 2024-05-27T00:18:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fea00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe1f80c13f0 con 0x7fe1f80731f0 2024-05-27T00:18:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1f7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f80735d0 0x7fe1f8147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1f7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f80735d0 0x7fe1f8147a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37612/0 (socket says 172.21.15.98:37612) 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1f7400640 1 -- 172.21.15.98:0/3426922476 learned_addr learned my addr 172.21.15.98:0/3426922476 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1f7e00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1f80731f0 0x7fe1f80c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fca00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1f80c79f0 0x7fe1f8151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1f7400640 1 -- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1f80731f0 msgr2=0x7fe1f80c2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1f7400640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1f80731f0 0x7fe1f80c2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1f7400640 1 -- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1f80c79f0 msgr2=0x7fe1f8151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1f7400640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1f80c79f0 0x7fe1f8151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1f7400640 1 -- 172.21.15.98:0/3426922476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f81493b0 con 0x7fe1f80735d0 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fca00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1f80c79f0 0x7fe1f8151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1f7400640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f80735d0 0x7fe1f8147a40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800e930 tx=0x7fe1e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1efe00640 1 -- 172.21.15.98:0/3426922476 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8017070 con 0x7fe1f80735d0 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1efe00640 1 -- 172.21.15.98:0/3426922476 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1e800ca00 con 0x7fe1f80735d0 2024-05-27T00:18:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1efe00640 1 -- 172.21.15.98:0/3426922476 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e800cdf0 con 0x7fe1f80735d0 2024-05-27T00:18:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fea00640 1 -- 172.21.15.98:0/3426922476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f80c5390 con 0x7fe1f80735d0 2024-05-27T00:18:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fea00640 1 -- 172.21.15.98:0/3426922476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f8149db0 con 0x7fe1f80735d0 2024-05-27T00:18:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1fea00640 1 -- 172.21.15.98:0/3426922476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1f8075cb0 con 0x7fe1f80735d0 2024-05-27T00:18:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.876+0000 7fe1efe00640 1 -- 172.21.15.98:0/3426922476 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe1e801a070 con 0x7fe1f80735d0 2024-05-27T00:18:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.880+0000 7fe1efe00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1d40678f0 0x7fe1d4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.880+0000 7fe1efe00640 1 -- 172.21.15.98:0/3426922476 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7fe1e8086100 con 0x7fe1f80735d0 2024-05-27T00:18:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.880+0000 7fe1f7e00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1d40678f0 0x7fe1d4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.880+0000 7fe1f7e00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1d40678f0 0x7fe1d4069db0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00b2000 tx=0x7fe1e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:28.884+0000 7fe1efe00640 1 -- 172.21.15.98:0/3426922476 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fe1e8056580 con 0x7fe1f80735d0 2024-05-27T00:18:29.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 -- 172.21.15.98:0/3426922476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1f8158510 con 0x7fe1f80735d0 2024-05-27T00:18:29.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1efe00640 1 -- 172.21.15.98:0/3426922476 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6006 (secure 0 0 0) 0x7fe1e805a880 con 0x7fe1f80735d0 2024-05-27T00:18:29.075 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:29.075 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:25.179272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:18:29.075 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 14 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 -- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1d40678f0 msgr2=0x7fe1d4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1d40678f0 0x7fe1d4069db0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00b2000 tx=0x7fe1e0008040 comp rx=0 tx=0).stop 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 -- 172.21.15.98:0/3426922476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f80735d0 msgr2=0x7fe1f8147a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f80735d0 0x7fe1f8147a40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800e930 tx=0x7fe1e800edf0 comp rx=0 tx=0).stop 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 -- 172.21.15.98:0/3426922476 shutdown_connections 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1d40678f0 0x7fe1d4069db0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1f80c79f0 0x7fe1f8151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1f80735d0 0x7fe1f8147a40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 --2- 172.21.15.98:0/3426922476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1f80731f0 0x7fe1f80c2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 -- 172.21.15.98:0/3426922476 >> 172.21.15.98:0/3426922476 conn(0x7fe1f8067530 msgr2=0x7fe1f814d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 -- 172.21.15.98:0/3426922476 shutdown_connections 2024-05-27T00:18:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:29.072+0000 7fe1fea00640 1 -- 172.21.15.98:0/3426922476 wait complete. 2024-05-27T00:18:29.096 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:18:31.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:31.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.236+0000 7f8adc000640 1 Processor -- start 2024-05-27T00:18:31.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.236+0000 7f8adc000640 1 -- start start 2024-05-27T00:18:31.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.236+0000 7f8adc000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ad41529e0 0x7f8ad40c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:31.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.236+0000 7f8adc000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ad40c8fc0 0x7f8ad406fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:31.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.236+0000 7f8adc000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8ad40c8a00 0x7f8ad40c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:31.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.236+0000 7f8adc000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad406f170 con 0x7f8ad41529e0 2024-05-27T00:18:31.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.236+0000 7f8adc000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad40c36b0 con 0x7f8ad40c8a00 2024-05-27T00:18:31.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.236+0000 7f8adc000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad4070100 con 0x7f8ad40c8fc0 2024-05-27T00:18:31.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.236+0000 7f8ad9600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ad41529e0 0x7f8ad40c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:31.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ad9600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ad41529e0 0x7f8ad40c3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37624/0 (socket says 172.21.15.98:37624) 2024-05-27T00:18:31.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ad9600640 1 -- 172.21.15.98:0/2110881665 learned_addr learned my addr 172.21.15.98:0/2110881665 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:31.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ad8c00640 1 --2- 172.21.15.98:0/2110881665 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ad40c8fc0 0x7f8ad406fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:31.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ada000640 1 --2- 172.21.15.98:0/2110881665 >> v2:172.21.15.204:3300/0 conn(0x7f8ad40c8a00 0x7f8ad40c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:31.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ad9600640 1 -- 172.21.15.98:0/2110881665 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ad40c8fc0 msgr2=0x7f8ad406fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:31.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ad9600640 1 --2- 172.21.15.98:0/2110881665 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ad40c8fc0 0x7f8ad406fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ad9600640 1 -- 172.21.15.98:0/2110881665 >> v2:172.21.15.204:3300/0 conn(0x7f8ad40c8a00 msgr2=0x7f8ad40c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:31.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ad9600640 1 --2- 172.21.15.98:0/2110881665 >> v2:172.21.15.204:3300/0 conn(0x7f8ad40c8a00 0x7f8ad40c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ad9600640 1 -- 172.21.15.98:0/2110881665 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad40c4d70 con 0x7f8ad41529e0 2024-05-27T00:18:31.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ad9600640 1 --2- 172.21.15.98:0/2110881665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad41529e0 0x7f8ad40c3020 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4070af0 tx=0x7f8ac801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257cebd4c2ca92ad server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:31.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.240+0000 7f8ad3e00640 1 -- 172.21.15.98:0/2110881665 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80b1070 con 0x7f8ad41529e0 2024-05-27T00:18:31.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8ad3e00640 1 -- 172.21.15.98:0/2110881665 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac80ac630 con 0x7f8ad41529e0 2024-05-27T00:18:31.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8ad3e00640 1 -- 172.21.15.98:0/2110881665 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80a8050 con 0x7f8ad41529e0 2024-05-27T00:18:31.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 -- 172.21.15.98:0/2110881665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad41529e0 msgr2=0x7f8ad40c3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:31.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 --2- 172.21.15.98:0/2110881665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad41529e0 0x7f8ad40c3020 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4070af0 tx=0x7f8ac801c520 comp rx=0 tx=0).stop 2024-05-27T00:18:31.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 -- 172.21.15.98:0/2110881665 shutdown_connections 2024-05-27T00:18:31.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 --2- 172.21.15.98:0/2110881665 >> v2:172.21.15.204:3300/0 conn(0x7f8ad40c8a00 0x7f8ad40c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 --2- 172.21.15.98:0/2110881665 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ad40c8fc0 0x7f8ad406fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 --2- 172.21.15.98:0/2110881665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad41529e0 0x7f8ad40c3020 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 -- 172.21.15.98:0/2110881665 >> 172.21.15.98:0/2110881665 conn(0x7f8ad40ce000 msgr2=0x7f8ad40ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:31.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 -- 172.21.15.98:0/2110881665 shutdown_connections 2024-05-27T00:18:31.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 -- 172.21.15.98:0/2110881665 wait complete. 2024-05-27T00:18:31.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 Processor -- start 2024-05-27T00:18:31.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 -- start start 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad40c8a00 0x7f8ad41556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ad40c8fc0 0x7f8ad406b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ad41529e0 0x7f8ad4157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad415c6b0 con 0x7f8ad40c8a00 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad415c530 con 0x7f8ad41529e0 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8adc000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad415c830 con 0x7f8ad40c8fc0 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8ad9600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad40c8a00 0x7f8ad41556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8ad9600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad40c8a00 0x7f8ad41556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37630/0 (socket says 172.21.15.98:37630) 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8ad9600640 1 -- 172.21.15.98:0/614727181 learned_addr learned my addr 172.21.15.98:0/614727181 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8ada000640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ad41529e0 0x7f8ad4157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:31.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.244+0000 7f8ad8c00640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ad40c8fc0 0x7f8ad406b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:31.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad9600640 1 -- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ad40c8fc0 msgr2=0x7f8ad406b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:31.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad9600640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ad40c8fc0 0x7f8ad406b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad9600640 1 -- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ad41529e0 msgr2=0x7f8ad4157620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:31.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad9600640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ad41529e0 0x7f8ad4157620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad9600640 1 -- 172.21.15.98:0/614727181 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad4155e10 con 0x7f8ad40c8a00 2024-05-27T00:18:31.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad9600640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad40c8a00 0x7f8ad41556d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8ac80af040 tx=0x7f8ac80192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:31.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad1600640 1 -- 172.21.15.98:0/614727181 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80bc070 con 0x7f8ad40c8a00 2024-05-27T00:18:31.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad1600640 1 -- 172.21.15.98:0/614727181 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac80ab5e0 con 0x7f8ad40c8a00 2024-05-27T00:18:31.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad1600640 1 -- 172.21.15.98:0/614727181 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80b1070 con 0x7f8ad40c8a00 2024-05-27T00:18:31.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8adc000640 1 -- 172.21.15.98:0/614727181 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad414f7f0 con 0x7f8ad40c8a00 2024-05-27T00:18:31.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8adc000640 1 -- 172.21.15.98:0/614727181 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad414fdb0 con 0x7f8ad40c8a00 2024-05-27T00:18:31.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8adc000640 1 -- 172.21.15.98:0/614727181 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad4075cb0 con 0x7f8ad40c8a00 2024-05-27T00:18:31.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad1600640 1 -- 172.21.15.98:0/614727181 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8ac80ab9d0 con 0x7f8ad40c8a00 2024-05-27T00:18:31.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad1600640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ab40678f0 0x7f8ab4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:31.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.248+0000 7f8ad1600640 1 -- 172.21.15.98:0/614727181 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f8ac811f890 con 0x7f8ad40c8a00 2024-05-27T00:18:31.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.252+0000 7f8ad8c00640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ab40678f0 0x7f8ab4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:31.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.252+0000 7f8ad8c00640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ab40678f0 0x7f8ab4069db0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4151600 tx=0x7f8ac0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:31.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.256+0000 7f8ad1600640 1 -- 172.21.15.98:0/614727181 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8ac8016110 con 0x7f8ad40c8a00 2024-05-27T00:18:31.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.456+0000 7f8adc000640 1 -- 172.21.15.98:0/614727181 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ad40c7f90 con 0x7f8ad40c8a00 2024-05-27T00:18:31.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8ad1600640 1 -- 172.21.15.98:0/614727181 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6006 (secure 0 0 0) 0x7f8ac80efbc0 con 0x7f8ad40c8a00 2024-05-27T00:18:31.462 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:31.462 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:25.179272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:18:31.462 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 14 2024-05-27T00:18:31.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 -- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ab40678f0 msgr2=0x7f8ab4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:31.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ab40678f0 0x7f8ab4069db0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4151600 tx=0x7f8ac0008040 comp rx=0 tx=0).stop 2024-05-27T00:18:31.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 -- 172.21.15.98:0/614727181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad40c8a00 msgr2=0x7f8ad41556d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:31.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad40c8a00 0x7f8ad41556d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8ac80af040 tx=0x7f8ac80192c0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 -- 172.21.15.98:0/614727181 shutdown_connections 2024-05-27T00:18:31.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ab40678f0 0x7f8ab4069db0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ad41529e0 0x7f8ad4157620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ad40c8fc0 0x7f8ad406b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 --2- 172.21.15.98:0/614727181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad40c8a00 0x7f8ad41556d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:31.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 -- 172.21.15.98:0/614727181 >> 172.21.15.98:0/614727181 conn(0x7f8ad40ce000 msgr2=0x7f8ad4151db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:31.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 -- 172.21.15.98:0/614727181 shutdown_connections 2024-05-27T00:18:31.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:31.460+0000 7f8adc000640 1 -- 172.21.15.98:0/614727181 wait complete. 2024-05-27T00:18:31.484 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:18:33.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.620+0000 7f7de2e00640 1 Processor -- start 2024-05-27T00:18:33.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.620+0000 7f7de2e00640 1 -- start start 2024-05-27T00:18:33.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de2e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7ddc0735d0 0x7f7ddc151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:33.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de2e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ddc0c79f0 0x7f7ddc151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:33.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de2e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ddc0c17d0 con 0x7f7ddc0731f0 2024-05-27T00:18:33.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de2e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7ddc071990 con 0x7f7ddc0735d0 2024-05-27T00:18:33.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de2e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7ddc070e10 con 0x7f7ddc0c79f0 2024-05-27T00:18:33.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddbe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7ddc0731f0 0x7f7ddc0c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51578/0 (socket says 172.21.15.98:51578) 2024-05-27T00:18:33.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddbe00640 1 -- 172.21.15.98:0/2693650371 learned_addr learned my addr 172.21.15.98:0/2693650371 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:33.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7dd3400640 1 --2- 172.21.15.98:0/2693650371 >> v2:172.21.15.204:3300/0 conn(0x7f7ddc0735d0 0x7f7ddc151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:33.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de0e00640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ddc0c79f0 0x7f7ddc151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:33.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039698176 0 0) 0x7f7ddc0c17d0 con 0x7f7ddc0731f0 2024-05-27T00:18:33.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7dc0005740 con 0x7f7ddc0731f0 2024-05-27T00:18:33.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3709628969 0 0) 0x7f7dc0005740 con 0x7f7ddc0731f0 2024-05-27T00:18:33.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ddc0c79f0 msgr2=0x7f7ddc151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:33.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ddc0c79f0 0x7f7ddc151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 >> v2:172.21.15.204:3300/0 conn(0x7f7ddc0735d0 msgr2=0x7f7ddc151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 --2- 172.21.15.98:0/2693650371 >> v2:172.21.15.204:3300/0 conn(0x7f7ddc0735d0 0x7f7ddc151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ddc0710f0 con 0x7f7ddc0731f0 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f7dcc001680 con 0x7f7ddc0731f0 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 >> v1:172.21.15.98:6789/0 conn(0x7f7ddc0731f0 legacy=0x7f7ddc0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ddc0c79f0 0x7f7dc0005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dc0006150 0x7f7dc0006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc0006f60 0x7f7dc00073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ddc071990 con 0x7f7dc0006f60 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7ddc070e10 con 0x7f7dc0006150 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7dc000dc10 con 0x7f7ddc0c79f0 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de0e00640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc0006f60 0x7f7dc00073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:33.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7ddbe00640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ddc0c79f0 0x7f7dc0005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:33.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7dd3400640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dc0006150 0x7f7dc0006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:33.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de0e00640 1 -- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ddc0c79f0 msgr2=0x7f7dc0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:33.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de0e00640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ddc0c79f0 0x7f7dc0005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:33.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de0e00640 1 -- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dc0006150 msgr2=0x7f7dc0006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:33.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de0e00640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7dc0006150 0x7f7dc0006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:33.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.624+0000 7f7de0e00640 1 -- 172.21.15.98:0/2693650371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc0005740 con 0x7f7dc0006f60 2024-05-27T00:18:33.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7de0e00640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc0006f60 0x7f7dc00073e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009fc0 tx=0x7f7dd401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67240befc288b3c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:33.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40b2070 con 0x7f7dc0006f60 2024-05-27T00:18:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dd40a5c90 con 0x7f7dc0006f60 2024-05-27T00:18:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7ddb400640 1 -- 172.21.15.98:0/2693650371 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4009c10 con 0x7f7dc0006f60 2024-05-27T00:18:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7de2e00640 1 -- 172.21.15.98:0/2693650371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc0006f60 msgr2=0x7f7dc00073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7de2e00640 1 --2- 172.21.15.98:0/2693650371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7dc0006f60 0x7f7dc00073e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009fc0 tx=0x7f7dd401cc80 comp rx=0 tx=0).stop 2024-05-27T00:18:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7de0e00640 1 -- 172.21.15.98:0/2693650371 reap_dead start 2024-05-27T00:18:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7de2e00640 1 -- 172.21.15.98:0/2693650371 shutdown_connections 2024-05-27T00:18:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7de2e00640 1 -- 172.21.15.98:0/2693650371 >> 172.21.15.98:0/2693650371 conn(0x7f7ddc067530 msgr2=0x7f7ddc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7de2e00640 1 -- 172.21.15.98:0/2693650371 shutdown_connections 2024-05-27T00:18:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7de2e00640 1 -- 172.21.15.98:0/2693650371 wait complete. 2024-05-27T00:18:33.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7de2e00640 1 Processor -- start 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.628+0000 7f7de2e00640 1 -- start start 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de2e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7dc0006f60 0x7f7ddc146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de2e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ddc155210 0x7f7ddc1f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de2e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ddc155730 0x7f7ddc1f4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de2e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ddc0c8400 con 0x7f7ddc155730 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de2e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7ddc0716b0 con 0x7f7ddc155210 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de2e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7ddc151970 con 0x7f7dc0006f60 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de0e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ddc155730 0x7f7ddc1f4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de0e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ddc155730 0x7f7ddc1f4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40786/0 (socket says 172.21.15.98:40786) 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de0e00640 1 -- 172.21.15.98:0/4137328806 learned_addr learned my addr 172.21.15.98:0/4137328806 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7ddbe00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7dc0006f60 0x7f7ddc146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7dd3400640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ddc155210 0x7f7ddc1f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:33.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de0e00640 1 -- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7dc0006f60 msgr2=0x7f7ddc146120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de0e00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7dc0006f60 0x7f7ddc146120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de0e00640 1 -- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ddc155210 msgr2=0x7f7ddc1f4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de0e00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ddc155210 0x7f7ddc1f4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de0e00640 1 -- 172.21.15.98:0/4137328806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ddc1f0cb0 con 0x7f7ddc155730 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7ddbe00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7dc0006f60 0x7f7ddc146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de0e00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ddc155730 0x7f7ddc1f4980 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40b0040 tx=0x7f7dd4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7dd8c00640 1 -- 172.21.15.98:0/4137328806 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40bd070 con 0x7f7ddc155730 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7dd8c00640 1 -- 172.21.15.98:0/4137328806 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dd40a5a90 con 0x7f7ddc155730 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de2e00640 1 -- 172.21.15.98:0/4137328806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ddc1f0f90 con 0x7f7ddc155730 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7dd8c00640 1 -- 172.21.15.98:0/4137328806 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40b2040 con 0x7f7ddc155730 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de2e00640 1 -- 172.21.15.98:0/4137328806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ddc1f1550 con 0x7f7ddc155730 2024-05-27T00:18:33.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7de2e00640 1 -- 172.21.15.98:0/4137328806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ddc075cb0 con 0x7f7ddc155730 2024-05-27T00:18:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.632+0000 7f7dd8c00640 1 -- 172.21.15.98:0/4137328806 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7dd40c0070 con 0x7f7ddc155730 2024-05-27T00:18:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.636+0000 7f7dd8c00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7db80678c0 0x7f7db8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.636+0000 7f7dd8c00640 1 -- 172.21.15.98:0/4137328806 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f7dd411fe20 con 0x7f7ddc155730 2024-05-27T00:18:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.636+0000 7f7ddbe00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7db80678c0 0x7f7db8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.636+0000 7f7ddbe00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7db80678c0 0x7f7db8069d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc001060 tx=0x7f7dcc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.640+0000 7f7dd8c00640 1 -- 172.21.15.98:0/4137328806 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7dd4016990 con 0x7f7ddc155730 2024-05-27T00:18:33.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.828+0000 7f7de2e00640 1 -- 172.21.15.98:0/4137328806 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ddc1507c0 con 0x7f7ddc155730 2024-05-27T00:18:33.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.828+0000 7f7dd8c00640 1 -- 172.21.15.98:0/4137328806 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6010 (secure 0 0 0) 0x7f7dd40f00c0 con 0x7f7ddc155730 2024-05-27T00:18:33.832 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:33.833 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:32.662526+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:18:33.833 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 15 2024-05-27T00:18:33.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 -- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7db80678c0 msgr2=0x7f7db8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:33.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7db80678c0 0x7f7db8069d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc001060 tx=0x7f7dcc009040 comp rx=0 tx=0).stop 2024-05-27T00:18:33.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 -- 172.21.15.98:0/4137328806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ddc155730 msgr2=0x7f7ddc1f4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ddc155730 0x7f7ddc1f4980 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40b0040 tx=0x7f7dd4009750 comp rx=0 tx=0).stop 2024-05-27T00:18:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 -- 172.21.15.98:0/4137328806 shutdown_connections 2024-05-27T00:18:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ddc155730 0x7f7ddc1f4980 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7db80678c0 0x7f7db8069d80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ddc155210 0x7f7ddc1f4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 --2- 172.21.15.98:0/4137328806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7dc0006f60 0x7f7ddc146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 -- 172.21.15.98:0/4137328806 >> 172.21.15.98:0/4137328806 conn(0x7f7ddc067530 msgr2=0x7f7ddc0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 -- 172.21.15.98:0/4137328806 shutdown_connections 2024-05-27T00:18:33.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:33.832+0000 7f7de2e00640 1 -- 172.21.15.98:0/4137328806 wait complete. 2024-05-27T00:18:33.852 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:18:35.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:35.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdeea000640 1 Processor -- start 2024-05-27T00:18:35.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdeea000640 1 -- start start 2024-05-27T00:18:35.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdeea000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdee40731f0 0x7fdee40c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:35.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdeea000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee40c28f0 con 0x7fdee40c2220 2024-05-27T00:18:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdeea000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee40c2bd0 con 0x7fdee40735d0 2024-05-27T00:18:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdeea000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdee4071a90 con 0x7fdee40731f0 2024-05-27T00:18:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdee3400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdee40731f0 0x7fdee40c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdee3e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdee40c2220 0x7fdee4151310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51594/0 (socket says 172.21.15.98:51594) 2024-05-27T00:18:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdee3e00640 1 -- 172.21.15.98:0/842366193 learned_addr learned my addr 172.21.15.98:0/842366193 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdee3400640 1 -- 172.21.15.98:0/842366193 >> v1:172.21.15.204:6789/0 conn(0x7fdee40735d0 legacy=0x7fdee415aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdee3400640 1 -- 172.21.15.98:0/842366193 >> v1:172.21.15.98:6789/0 conn(0x7fdee40c2220 legacy=0x7fdee4151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:35.996+0000 7fdee3400640 1 -- 172.21.15.98:0/842366193 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee4070fb0 con 0x7fdee40731f0 2024-05-27T00:18:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdee3400640 1 --2- 172.21.15.98:0/842366193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 0x7fdee40c79f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fded40097c0 tx=0x7fded401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce719be9cfb941b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdee2000640 1 -- 172.21.15.98:0/842366193 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded40b0070 con 0x7fdee40731f0 2024-05-27T00:18:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdee2000640 1 -- 172.21.15.98:0/842366193 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded4004230 con 0x7fdee40731f0 2024-05-27T00:18:36.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdee2000640 1 -- 172.21.15.98:0/842366193 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4003620 con 0x7fdee40731f0 2024-05-27T00:18:36.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdeea000640 1 -- 172.21.15.98:0/842366193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 msgr2=0x7fdee40c79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:36.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdeea000640 1 --2- 172.21.15.98:0/842366193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 0x7fdee40c79f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fded40097c0 tx=0x7fded401c460 comp rx=0 tx=0).stop 2024-05-27T00:18:36.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdeea000640 1 -- 172.21.15.98:0/842366193 shutdown_connections 2024-05-27T00:18:36.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdeea000640 1 --2- 172.21.15.98:0/842366193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 0x7fdee40c79f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:36.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdeea000640 1 -- 172.21.15.98:0/842366193 >> 172.21.15.98:0/842366193 conn(0x7fdee4067530 msgr2=0x7fdee406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:36.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdeea000640 1 -- 172.21.15.98:0/842366193 shutdown_connections 2024-05-27T00:18:36.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdeea000640 1 -- 172.21.15.98:0/842366193 wait complete. 2024-05-27T00:18:36.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.000+0000 7fdeea000640 1 Processor -- start 2024-05-27T00:18:36.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdeea000640 1 -- start start 2024-05-27T00:18:36.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdeea000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 0x7fdee4154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:36.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdeea000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdee40735d0 0x7fdee40c1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:36.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdeea000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdee40c2220 0x7fdee4146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:36.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdeea000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdee415b430 con 0x7fdee40731f0 2024-05-27T00:18:36.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdeea000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdee415b2b0 con 0x7fdee40735d0 2024-05-27T00:18:36.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdeea000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdee4165760 con 0x7fdee40c2220 2024-05-27T00:18:36.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 0x7fdee4154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:36.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 0x7fdee4154580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40808/0 (socket says 172.21.15.98:40808) 2024-05-27T00:18:36.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3400640 1 -- 172.21.15.98:0/1862986410 learned_addr learned my addr 172.21.15.98:0/1862986410 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3e00640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdee40c2220 0x7fdee4146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee2a00640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdee40735d0 0x7fdee40c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3400640 1 -- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdee40c2220 msgr2=0x7fdee4146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3400640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdee40c2220 0x7fdee4146fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3400640 1 -- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdee40735d0 msgr2=0x7fdee40c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3400640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdee40735d0 0x7fdee40c1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3400640 1 -- 172.21.15.98:0/1862986410 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee4154c30 con 0x7fdee40731f0 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3e00640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdee40c2220 0x7fdee4146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee2a00640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdee40735d0 0x7fdee40c1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdee3400640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 0x7fdee4154580 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fded40ae040 tx=0x7fded4019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdedb400640 1 -- 172.21.15.98:0/1862986410 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded40b0040 con 0x7fdee40731f0 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdedb400640 1 -- 172.21.15.98:0/1862986410 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded40bc070 con 0x7fdee40731f0 2024-05-27T00:18:36.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdeea000640 1 -- 172.21.15.98:0/1862986410 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee41f7010 con 0x7fdee40731f0 2024-05-27T00:18:36.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdedb400640 1 -- 172.21.15.98:0/1862986410 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded40b5480 con 0x7fdee40731f0 2024-05-27T00:18:36.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdeea000640 1 -- 172.21.15.98:0/1862986410 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee41f75d0 con 0x7fdee40731f0 2024-05-27T00:18:36.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.004+0000 7fdeea000640 1 -- 172.21.15.98:0/1862986410 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdee4075cb0 con 0x7fdee40731f0 2024-05-27T00:18:36.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.008+0000 7fdedb400640 1 -- 172.21.15.98:0/1862986410 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fded40b3070 con 0x7fdee40731f0 2024-05-27T00:18:36.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.008+0000 7fdedb400640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdec40678c0 0x7fdec4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:36.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.008+0000 7fdedb400640 1 -- 172.21.15.98:0/1862986410 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7fded411fde0 con 0x7fdee40731f0 2024-05-27T00:18:36.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.008+0000 7fdee2a00640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdec40678c0 0x7fdec4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:36.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.008+0000 7fdee2a00640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdec40678c0 0x7fdec4069d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdee414da10 tx=0x7fded0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:36.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.012+0000 7fdedb400640 1 -- 172.21.15.98:0/1862986410 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fded4016050 con 0x7fdee40731f0 2024-05-27T00:18:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.200+0000 7fdeea000640 1 -- 172.21.15.98:0/1862986410 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdee40c1fe0 con 0x7fdee40731f0 2024-05-27T00:18:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.200+0000 7fdedb400640 1 -- 172.21.15.98:0/1862986410 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6010 (secure 0 0 0) 0x7fded40aa3a0 con 0x7fdee40731f0 2024-05-27T00:18:36.205 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:36.205 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:32.662526+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:18:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 15 2024-05-27T00:18:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 -- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdec40678c0 msgr2=0x7fdec4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdec40678c0 0x7fdec4069d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdee414da10 tx=0x7fded0009040 comp rx=0 tx=0).stop 2024-05-27T00:18:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 -- 172.21.15.98:0/1862986410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 msgr2=0x7fdee4154580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 0x7fdee4154580 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fded40ae040 tx=0x7fded4019200 comp rx=0 tx=0).stop 2024-05-27T00:18:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 -- 172.21.15.98:0/1862986410 shutdown_connections 2024-05-27T00:18:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdee40c2220 0x7fdee4146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdec40678c0 0x7fdec4069d80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdee40735d0 0x7fdee40c1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:36.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 --2- 172.21.15.98:0/1862986410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdee40731f0 0x7fdee4154580 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 -- 172.21.15.98:0/1862986410 >> 172.21.15.98:0/1862986410 conn(0x7fdee4067530 msgr2=0x7fdee40c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 -- 172.21.15.98:0/1862986410 shutdown_connections 2024-05-27T00:18:36.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:36.204+0000 7fdeea000640 1 -- 172.21.15.98:0/1862986410 wait complete. 2024-05-27T00:18:36.225 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:18:38.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:38.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.368+0000 7f1ac0c00640 1 Processor -- start 2024-05-27T00:18:38.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.368+0000 7f1ac0c00640 1 -- start start 2024-05-27T00:18:38.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1ac0c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1abc0735d0 0x7f1abc151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1ac0c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1abc0c79f0 0x7f1abc151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1ac0c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1abc0c17d0 con 0x7f1abc0731f0 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1ac0c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1abc071990 con 0x7f1abc0c79f0 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1ac0c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1abc070e10 con 0x7f1abc0735d0 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1aba800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1abc0c79f0 0x7f1abc151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1aba800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1abc0c79f0 0x7f1abc151970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40822/0 (socket says 172.21.15.98:40822) 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1aba800640 1 -- 172.21.15.98:0/4052627487 learned_addr learned my addr 172.21.15.98:0/4052627487 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1ab1400640 1 --2- 172.21.15.98:0/4052627487 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1abc0735d0 0x7f1abc151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1aba800640 1 -- 172.21.15.98:0/4052627487 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1abc0735d0 msgr2=0x7f1abc151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1aba800640 1 --2- 172.21.15.98:0/4052627487 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1abc0735d0 0x7f1abc151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1aba800640 1 -- 172.21.15.98:0/4052627487 >> v1:172.21.15.204:6789/0 conn(0x7f1abc0731f0 legacy=0x7f1abc0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:38.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1aba800640 1 -- 172.21.15.98:0/4052627487 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1abc0710f0 con 0x7f1abc0c79f0 2024-05-27T00:18:38.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.372+0000 7f1aba800640 1 --2- 172.21.15.98:0/4052627487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0c79f0 0x7f1abc151970 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009e50 tx=0x7f1ab401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61d51dc25610c2e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:38.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ab9400640 1 -- 172.21.15.98:0/4052627487 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40b4070 con 0x7f1abc0c79f0 2024-05-27T00:18:38.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ab9400640 1 -- 172.21.15.98:0/4052627487 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab40af400 con 0x7f1abc0c79f0 2024-05-27T00:18:38.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ab9400640 1 -- 172.21.15.98:0/4052627487 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40af7d0 con 0x7f1abc0c79f0 2024-05-27T00:18:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 -- 172.21.15.98:0/4052627487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0c79f0 msgr2=0x7f1abc151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 --2- 172.21.15.98:0/4052627487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0c79f0 0x7f1abc151970 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009e50 tx=0x7f1ab401f210 comp rx=0 tx=0).stop 2024-05-27T00:18:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 -- 172.21.15.98:0/4052627487 shutdown_connections 2024-05-27T00:18:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 --2- 172.21.15.98:0/4052627487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0c79f0 0x7f1abc151970 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 --2- 172.21.15.98:0/4052627487 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1abc0735d0 0x7f1abc151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 -- 172.21.15.98:0/4052627487 >> 172.21.15.98:0/4052627487 conn(0x7f1abc067530 msgr2=0x7f1abc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 -- 172.21.15.98:0/4052627487 shutdown_connections 2024-05-27T00:18:38.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 -- 172.21.15.98:0/4052627487 wait complete. 2024-05-27T00:18:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 Processor -- start 2024-05-27T00:18:38.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 -- start start 2024-05-27T00:18:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0731f0 0x7f1abc15b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1abc0735d0 0x7f1abc1f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1abc0c79f0 0x7f1abc1f1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc0c8400 con 0x7f1abc0731f0 2024-05-27T00:18:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc0716b0 con 0x7f1abc0c79f0 2024-05-27T00:18:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ac0c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1abc157970 con 0x7f1abc0735d0 2024-05-27T00:18:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ab9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0731f0 0x7f1abc15b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ab9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0731f0 0x7f1abc15b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40824/0 (socket says 172.21.15.98:40824) 2024-05-27T00:18:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ab9e00640 1 -- 172.21.15.98:0/1684200277 learned_addr learned my addr 172.21.15.98:0/1684200277 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1ab1400640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1abc0735d0 0x7f1abc1f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.376+0000 7f1aba800640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1abc0c79f0 0x7f1abc1f1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab9e00640 1 -- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1abc0735d0 msgr2=0x7f1abc1f0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab9e00640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1abc0735d0 0x7f1abc1f0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab9e00640 1 -- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1abc0c79f0 msgr2=0x7f1abc1f1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab9e00640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1abc0c79f0 0x7f1abc1f1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:38.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab9e00640 1 -- 172.21.15.98:0/1684200277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1abc158b10 con 0x7f1abc0731f0 2024-05-27T00:18:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1aba800640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1abc0c79f0 0x7f1abc1f1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:18:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab9e00640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0731f0 0x7f1abc15b0c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00c450 tx=0x7f1aac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab2a00640 1 -- 172.21.15.98:0/1684200277 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac016020 con 0x7f1abc0731f0 2024-05-27T00:18:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab2a00640 1 -- 172.21.15.98:0/1684200277 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aac0052d0 con 0x7f1abc0731f0 2024-05-27T00:18:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ac0c00640 1 -- 172.21.15.98:0/1684200277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1abc1f06d0 con 0x7f1abc0731f0 2024-05-27T00:18:38.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab2a00640 1 -- 172.21.15.98:0/1684200277 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac007260 con 0x7f1abc0731f0 2024-05-27T00:18:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ac0c00640 1 -- 172.21.15.98:0/1684200277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1abc158dd0 con 0x7f1abc0731f0 2024-05-27T00:18:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ac0c00640 1 -- 172.21.15.98:0/1684200277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1abc075cb0 con 0x7f1abc0731f0 2024-05-27T00:18:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab2a00640 1 -- 172.21.15.98:0/1684200277 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1aac007400 con 0x7f1abc0731f0 2024-05-27T00:18:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab2a00640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a980678c0 0x7f1a98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab2a00640 1 -- 172.21.15.98:0/1684200277 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f1aac085810 con 0x7f1abc0731f0 2024-05-27T00:18:38.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.380+0000 7f1ab1400640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a980678c0 0x7f1a98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:38.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.384+0000 7f1ab1400640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a980678c0 0x7f1a98069d80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1abc072670 tx=0x7f1aa4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:38.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.384+0000 7f1ab2a00640 1 -- 172.21.15.98:0/1684200277 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1aac020020 con 0x7f1abc0731f0 2024-05-27T00:18:38.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.580+0000 7f1ac0c00640 1 -- 172.21.15.98:0/1684200277 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1abc158510 con 0x7f1abc0731f0 2024-05-27T00:18:38.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.580+0000 7f1ab2a00640 1 -- 172.21.15.98:0/1684200277 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6007 (secure 0 0 0) 0x7f1aac005480 con 0x7f1abc0731f0 2024-05-27T00:18:38.585 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:38.585 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:38.440600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:18:38.586 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 16 2024-05-27T00:18:38.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 -- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a980678c0 msgr2=0x7f1a98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:38.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a980678c0 0x7f1a98069d80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1abc072670 tx=0x7f1aa4008040 comp rx=0 tx=0).stop 2024-05-27T00:18:38.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 -- 172.21.15.98:0/1684200277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0731f0 msgr2=0x7f1abc15b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0731f0 0x7f1abc15b0c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00c450 tx=0x7f1aac00c910 comp rx=0 tx=0).stop 2024-05-27T00:18:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 -- 172.21.15.98:0/1684200277 shutdown_connections 2024-05-27T00:18:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a980678c0 0x7f1a98069d80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1abc0c79f0 0x7f1abc1f1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1abc0735d0 0x7f1abc1f0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 --2- 172.21.15.98:0/1684200277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1abc0731f0 0x7f1abc15b0c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 -- 172.21.15.98:0/1684200277 >> 172.21.15.98:0/1684200277 conn(0x7f1abc067530 msgr2=0x7f1abc0c6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 -- 172.21.15.98:0/1684200277 shutdown_connections 2024-05-27T00:18:38.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:38.584+0000 7f1ac0c00640 1 -- 172.21.15.98:0/1684200277 wait complete. 2024-05-27T00:18:38.607 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 16, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 16, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4894": { "addr": "172.21.15.98:6837/1781395818", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 1781395818, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 1781395818, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4894, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:18:38.440600+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4894 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:18:40.608 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3f1e00640 1 Processor -- start 2024-05-27T00:18:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3f1e00640 1 -- start start 2024-05-27T00:18:40.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3f1e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 0x7fe3ec0b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:40.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3f1e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe3ec0d0070 0x7fe3ec0d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3f1e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3ec0b3f20 con 0x7fe3ec154a00 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3f1e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe3ec0b4200 con 0x7fe3ec0d0070 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3f1e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe3ec0b44e0 con 0x7fe3ec154de0 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3eaa00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe3ec154a00 0x7fe3ec15b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51606/0 (socket says 172.21.15.98:51606) 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3eaa00640 1 -- 172.21.15.98:0/3210705883 learned_addr learned my addr 172.21.15.98:0/3210705883 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3eb400640 1 --2- 172.21.15.98:0/3210705883 >> v2:172.21.15.98:3300/0 conn(0x7fe3ec0d0070 0x7fe3ec0d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3e2000640 1 --2- 172.21.15.98:0/3210705883 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 0x7fe3ec0b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3ea000640 1 -- 172.21.15.98:0/3210705883 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2224255350 0 0) 0x7fe3ec0b3f20 con 0x7fe3ec154a00 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3ea000640 1 -- 172.21.15.98:0/3210705883 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3d0003530 con 0x7fe3ec154a00 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3eb400640 1 -- 172.21.15.98:0/3210705883 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 msgr2=0x7fe3ec0b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3eb400640 1 --2- 172.21.15.98:0/3210705883 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 0x7fe3ec0b2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3eb400640 1 -- 172.21.15.98:0/3210705883 >> v1:172.21.15.98:6789/0 conn(0x7fe3ec154a00 legacy=0x7fe3ec15b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3eb400640 1 -- 172.21.15.98:0/3210705883 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec0b47c0 con 0x7fe3ec0d0070 2024-05-27T00:18:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.752+0000 7fe3e2000640 1 --2- 172.21.15.98:0/3210705883 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 0x7fe3ec0b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:18:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3eb400640 1 --2- 172.21.15.98:0/3210705883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 0x7fe3ec0d04c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009d90 tx=0x7fe3e401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928577dda6ed605d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3ea000640 1 -- 172.21.15.98:0/3210705883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40b4070 con 0x7fe3ec0d0070 2024-05-27T00:18:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3ea000640 1 -- 172.21.15.98:0/3210705883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e40af600 con 0x7fe3ec0d0070 2024-05-27T00:18:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3ea000640 1 -- 172.21.15.98:0/3210705883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40af9d0 con 0x7fe3ec0d0070 2024-05-27T00:18:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3f1e00640 1 -- 172.21.15.98:0/3210705883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 msgr2=0x7fe3ec0d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3f1e00640 1 --2- 172.21.15.98:0/3210705883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 0x7fe3ec0d04c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009d90 tx=0x7fe3e401f150 comp rx=0 tx=0).stop 2024-05-27T00:18:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3f1e00640 1 -- 172.21.15.98:0/3210705883 shutdown_connections 2024-05-27T00:18:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3f1e00640 1 --2- 172.21.15.98:0/3210705883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 0x7fe3ec0d04c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3f1e00640 1 --2- 172.21.15.98:0/3210705883 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 0x7fe3ec0b2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3f1e00640 1 -- 172.21.15.98:0/3210705883 >> 172.21.15.98:0/3210705883 conn(0x7fe3ec067530 msgr2=0x7fe3ec06c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3f1e00640 1 -- 172.21.15.98:0/3210705883 shutdown_connections 2024-05-27T00:18:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.756+0000 7fe3f1e00640 1 -- 172.21.15.98:0/3210705883 wait complete. 2024-05-27T00:18:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 Processor -- start 2024-05-27T00:18:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 -- start start 2024-05-27T00:18:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 0x7fe3ec1f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3ec154a00 0x7fe3ec158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 0x7fe3ec15d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec0b37a0 con 0x7fe3ec0d0070 2024-05-27T00:18:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec0b3620 con 0x7fe3ec154a00 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe3ec0cc7e0 con 0x7fe3ec154de0 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3eaa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 0x7fe3ec1f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3eaa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 0x7fe3ec1f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40834/0 (socket says 172.21.15.98:40834) 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3eaa00640 1 -- 172.21.15.98:0/2523899678 learned_addr learned my addr 172.21.15.98:0/2523899678 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3e2000640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3ec154a00 0x7fe3ec158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3eb400640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 0x7fe3ec15d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3eaa00640 1 -- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 msgr2=0x7fe3ec15d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3eaa00640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 0x7fe3ec15d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3eaa00640 1 -- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3ec154a00 msgr2=0x7fe3ec158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3eaa00640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3ec154a00 0x7fe3ec158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3eaa00640 1 -- 172.21.15.98:0/2523899678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec15cd80 con 0x7fe3ec0d0070 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3eaa00640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 0x7fe3ec1f43b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00c480 tx=0x7fe3dc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3e3e00640 1 -- 172.21.15.98:0/2523899678 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc017070 con 0x7fe3ec0d0070 2024-05-27T00:18:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3e3e00640 1 -- 172.21.15.98:0/2523899678 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3dc005cf0 con 0x7fe3ec0d0070 2024-05-27T00:18:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3e3e00640 1 -- 172.21.15.98:0/2523899678 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc007290 con 0x7fe3ec0d0070 2024-05-27T00:18:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 -- 172.21.15.98:0/2523899678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3ec1f0430 con 0x7fe3ec0d0070 2024-05-27T00:18:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 -- 172.21.15.98:0/2523899678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3ec1f09d0 con 0x7fe3ec0d0070 2024-05-27T00:18:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.760+0000 7fe3f1e00640 1 -- 172.21.15.98:0/2523899678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3ec075cb0 con 0x7fe3ec0d0070 2024-05-27T00:18:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.764+0000 7fe3e3e00640 1 -- 172.21.15.98:0/2523899678 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe3dc01a070 con 0x7fe3ec0d0070 2024-05-27T00:18:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.764+0000 7fe3e3e00640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe3c80678f0 0x7fe3c8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:40.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.764+0000 7fe3e3e00640 1 -- 172.21.15.98:0/2523899678 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7fe3dc085210 con 0x7fe3ec0d0070 2024-05-27T00:18:40.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.764+0000 7fe3e2000640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe3c80678f0 0x7fe3c8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.764+0000 7fe3e2000640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe3c80678f0 0x7fe3c8069db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec15ad80 tx=0x7fe3d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:40.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.768+0000 7fe3e3e00640 1 -- 172.21.15.98:0/2523899678 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fe3dc055c20 con 0x7fe3ec0d0070 2024-05-27T00:18:40.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.956+0000 7fe3f1e00640 1 -- 172.21.15.98:0/2523899678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3ec0d0730 con 0x7fe3ec0d0070 2024-05-27T00:18:40.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.956+0000 7fe3e3e00640 1 -- 172.21.15.98:0/2523899678 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6007 (secure 0 0 0) 0x7fe3dc055520 con 0x7fe3ec0d0070 2024-05-27T00:18:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:40.961 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:38.440600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:18:40.961 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 16 2024-05-27T00:18:40.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 -- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe3c80678f0 msgr2=0x7fe3c8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:40.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe3c80678f0 0x7fe3c8069db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec15ad80 tx=0x7fe3d4008040 comp rx=0 tx=0).stop 2024-05-27T00:18:40.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 -- 172.21.15.98:0/2523899678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 msgr2=0x7fe3ec1f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:40.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 0x7fe3ec1f43b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc00c480 tx=0x7fe3dc00c940 comp rx=0 tx=0).stop 2024-05-27T00:18:40.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 -- 172.21.15.98:0/2523899678 shutdown_connections 2024-05-27T00:18:40.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe3ec154de0 0x7fe3ec15d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:40.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe3c80678f0 0x7fe3c8069db0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:40.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3ec154a00 0x7fe3ec158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:40.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 --2- 172.21.15.98:0/2523899678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3ec0d0070 0x7fe3ec1f43b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:40.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 -- 172.21.15.98:0/2523899678 >> 172.21.15.98:0/2523899678 conn(0x7fe3ec067530 msgr2=0x7fe3ec15b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:40.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 -- 172.21.15.98:0/2523899678 shutdown_connections 2024-05-27T00:18:40.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:40.960+0000 7fe3f1e00640 1 -- 172.21.15.98:0/2523899678 wait complete. 2024-05-27T00:18:40.982 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:18:42.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:43.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.124+0000 7fb31aa00640 1 Processor -- start 2024-05-27T00:18:43.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.124+0000 7fb31aa00640 1 -- start start 2024-05-27T00:18:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb31aa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3140731f0 0x7fb3140c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb31aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3140735d0 0x7fb314151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb31aa00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3140c17d0 con 0x7fb3140c79f0 2024-05-27T00:18:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb31aa00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb314071990 con 0x7fb3140731f0 2024-05-27T00:18:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb31aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb314070e10 con 0x7fb3140735d0 2024-05-27T00:18:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb313e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3140731f0 0x7fb3140c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:43.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb313e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3140731f0 0x7fb3140c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40840/0 (socket says 172.21.15.98:40840) 2024-05-27T00:18:43.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb313e00640 1 -- 172.21.15.98:0/462169534 learned_addr learned my addr 172.21.15.98:0/462169534 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:43.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb318a00640 1 --1- 172.21.15.98:0/462169534 >> v1:172.21.15.98:6789/0 conn(0x7fb3140c79f0 0x7fb314157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51618/0 (socket says 172.21.15.98:51618) 2024-05-27T00:18:43.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb30b400640 1 --2- 172.21.15.98:0/462169534 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3140735d0 0x7fb314151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:43.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb313e00640 1 -- 172.21.15.98:0/462169534 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3140735d0 msgr2=0x7fb314151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:43.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb313e00640 1 --2- 172.21.15.98:0/462169534 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3140735d0 0x7fb314151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:43.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb313e00640 1 -- 172.21.15.98:0/462169534 >> v1:172.21.15.98:6789/0 conn(0x7fb3140c79f0 legacy=0x7fb314157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:43.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb313e00640 1 -- 172.21.15.98:0/462169534 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3140710f0 con 0x7fb3140731f0 2024-05-27T00:18:43.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.128+0000 7fb313400640 1 -- 172.21.15.98:0/462169534 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2989717008 0 0) 0x7fb3140c17d0 con 0x7fb3140c79f0 2024-05-27T00:18:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb313e00640 1 --2- 172.21.15.98:0/462169534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140731f0 0x7fb3140c0eb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb304009870 tx=0x7fb30401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa65bc0a50a434dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb313400640 1 -- 172.21.15.98:0/462169534 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3040b5070 con 0x7fb3140731f0 2024-05-27T00:18:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb313400640 1 -- 172.21.15.98:0/462169534 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3040a8a50 con 0x7fb3140731f0 2024-05-27T00:18:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb313400640 1 -- 172.21.15.98:0/462169534 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3040a8e20 con 0x7fb3140731f0 2024-05-27T00:18:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb31aa00640 1 -- 172.21.15.98:0/462169534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140731f0 msgr2=0x7fb3140c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb31aa00640 1 --2- 172.21.15.98:0/462169534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140731f0 0x7fb3140c0eb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb304009870 tx=0x7fb30401ed50 comp rx=0 tx=0).stop 2024-05-27T00:18:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb31aa00640 1 -- 172.21.15.98:0/462169534 shutdown_connections 2024-05-27T00:18:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb31aa00640 1 --2- 172.21.15.98:0/462169534 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3140735d0 0x7fb314151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb31aa00640 1 --2- 172.21.15.98:0/462169534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140731f0 0x7fb3140c0eb0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb31aa00640 1 -- 172.21.15.98:0/462169534 >> 172.21.15.98:0/462169534 conn(0x7fb314067530 msgr2=0x7fb31406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb31aa00640 1 -- 172.21.15.98:0/462169534 shutdown_connections 2024-05-27T00:18:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb31aa00640 1 -- 172.21.15.98:0/462169534 wait complete. 2024-05-27T00:18:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb31aa00640 1 Processor -- start 2024-05-27T00:18:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.132+0000 7fb31aa00640 1 -- start start 2024-05-27T00:18:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb31aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3140731f0 0x7fb314158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb31aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3140735d0 0x7fb31414dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb31aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140c79f0 0x7fb3141f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb31aa00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3140c8400 con 0x7fb3140c79f0 2024-05-27T00:18:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb31aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb3140716b0 con 0x7fb3140731f0 2024-05-27T00:18:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb31aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb3140c13f0 con 0x7fb3140735d0 2024-05-27T00:18:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb318a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140c79f0 0x7fb3141f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb318a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140c79f0 0x7fb3141f0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40854/0 (socket says 172.21.15.98:40854) 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb318a00640 1 -- 172.21.15.98:0/2818534445 learned_addr learned my addr 172.21.15.98:0/2818534445 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb313e00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3140731f0 0x7fb314158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb30b400640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3140735d0 0x7fb31414dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb318a00640 1 -- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3140735d0 msgr2=0x7fb31414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb318a00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3140735d0 0x7fb31414dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb318a00640 1 -- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3140731f0 msgr2=0x7fb314158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb318a00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3140731f0 0x7fb314158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb318a00640 1 -- 172.21.15.98:0/2818534445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3141f0c00 con 0x7fb3140c79f0 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb318a00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140c79f0 0x7fb3141f0300 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c700 tx=0x7fb30c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb310c00640 1 -- 172.21.15.98:0/2818534445 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c018070 con 0x7fb3140c79f0 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb310c00640 1 -- 172.21.15.98:0/2818534445 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb30c00ce10 con 0x7fb3140c79f0 2024-05-27T00:18:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb310c00640 1 -- 172.21.15.98:0/2818534445 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c00a880 con 0x7fb3140c79f0 2024-05-27T00:18:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb31aa00640 1 -- 172.21.15.98:0/2818534445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3141f0ee0 con 0x7fb3140c79f0 2024-05-27T00:18:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb31aa00640 1 -- 172.21.15.98:0/2818534445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb31414d150 con 0x7fb3140c79f0 2024-05-27T00:18:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb31aa00640 1 -- 172.21.15.98:0/2818534445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb314075cb0 con 0x7fb3140c79f0 2024-05-27T00:18:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.136+0000 7fb310c00640 1 -- 172.21.15.98:0/2818534445 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb30c00aa20 con 0x7fb3140c79f0 2024-05-27T00:18:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.140+0000 7fb310c00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb2f00678c0 0x7fb2f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.140+0000 7fb310c00640 1 -- 172.21.15.98:0/2818534445 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7fb30c0852b0 con 0x7fb3140c79f0 2024-05-27T00:18:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.140+0000 7fb313e00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb2f00678c0 0x7fb2f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.140+0000 7fb313e00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb2f00678c0 0x7fb2f0069d80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb3040b2000 tx=0x7fb304008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.144+0000 7fb310c00640 1 -- 172.21.15.98:0/2818534445 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb30c055670 con 0x7fb3140c79f0 2024-05-27T00:18:43.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 -- 172.21.15.98:0/2818534445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb314158510 con 0x7fb3140c79f0 2024-05-27T00:18:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb310c00640 1 -- 172.21.15.98:0/2818534445 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6007 (secure 0 0 0) 0x7fb30c059970 con 0x7fb3140c79f0 2024-05-27T00:18:43.335 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:43.335 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:38.440600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:18:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 16 2024-05-27T00:18:43.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 -- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb2f00678c0 msgr2=0x7fb2f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:43.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb2f00678c0 0x7fb2f0069d80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb3040b2000 tx=0x7fb304008040 comp rx=0 tx=0).stop 2024-05-27T00:18:43.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 -- 172.21.15.98:0/2818534445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140c79f0 msgr2=0x7fb3141f0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:43.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140c79f0 0x7fb3141f0300 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c700 tx=0x7fb30c00cbc0 comp rx=0 tx=0).stop 2024-05-27T00:18:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 -- 172.21.15.98:0/2818534445 shutdown_connections 2024-05-27T00:18:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3140c79f0 0x7fb3141f0300 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3140735d0 0x7fb31414dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb2f00678c0 0x7fb2f0069d80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 --2- 172.21.15.98:0/2818534445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3140731f0 0x7fb314158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 -- 172.21.15.98:0/2818534445 >> 172.21.15.98:0/2818534445 conn(0x7fb314067530 msgr2=0x7fb31414f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 -- 172.21.15.98:0/2818534445 shutdown_connections 2024-05-27T00:18:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:43.332+0000 7fb31aa00640 1 -- 172.21.15.98:0/2818534445 wait complete. 2024-05-27T00:18:43.355 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:18:44.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:18:44.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:18:45.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:45.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.500+0000 7f1e1a200640 1 Processor -- start 2024-05-27T00:18:45.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.500+0000 7f1e1a200640 1 -- start start 2024-05-27T00:18:45.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.500+0000 7f1e1a200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1e14070660 0x7f1e140c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:45.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.500+0000 7f1e1a200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e14071010 con 0x7f1e140b3260 2024-05-27T00:18:45.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.500+0000 7f1e1a200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e140c88c0 con 0x7f1e140c36c0 2024-05-27T00:18:45.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.500+0000 7f1e1a200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1e140c8b80 con 0x7f1e14070660 2024-05-27T00:18:45.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.500+0000 7f1e13e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1e14070660 0x7f1e140c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:45.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.500+0000 7f1e18e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1e140b3260 0x7f1e140c7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39272/0 (socket says 172.21.15.98:39272) 2024-05-27T00:18:45.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.500+0000 7f1e18e00640 1 -- 172.21.15.98:0/3133517074 learned_addr learned my addr 172.21.15.98:0/3133517074 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:45.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.504+0000 7f1e13e00640 1 -- 172.21.15.98:0/3133517074 >> v1:172.21.15.204:6789/0 conn(0x7f1e140c36c0 legacy=0x7f1e140c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:45.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.504+0000 7f1e13e00640 1 -- 172.21.15.98:0/3133517074 >> v1:172.21.15.98:6789/0 conn(0x7f1e140b3260 legacy=0x7f1e140c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:45.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.504+0000 7f1e13e00640 1 -- 172.21.15.98:0/3133517074 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e140c8e60 con 0x7f1e14070660 2024-05-27T00:18:45.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.504+0000 7f1e13e00640 1 --2- 172.21.15.98:0/3133517074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e14070660 0x7f1e140c3180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1e000097c0 tx=0x7f1e0001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5bcdaad8cac0c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:45.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.504+0000 7f1e13400640 1 -- 172.21.15.98:0/3133517074 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e000b2070 con 0x7f1e14070660 2024-05-27T00:18:45.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.504+0000 7f1e13400640 1 -- 172.21.15.98:0/3133517074 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e000a9070 con 0x7f1e14070660 2024-05-27T00:18:45.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.504+0000 7f1e13400640 1 -- 172.21.15.98:0/3133517074 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e000ad880 con 0x7f1e14070660 2024-05-27T00:18:45.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.504+0000 7f1e1a200640 1 -- 172.21.15.98:0/3133517074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e14070660 msgr2=0x7f1e140c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:45.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.504+0000 7f1e1a200640 1 --2- 172.21.15.98:0/3133517074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e14070660 0x7f1e140c3180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1e000097c0 tx=0x7f1e0001c3c0 comp rx=0 tx=0).stop 2024-05-27T00:18:45.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 -- 172.21.15.98:0/3133517074 shutdown_connections 2024-05-27T00:18:45.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 --2- 172.21.15.98:0/3133517074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e14070660 0x7f1e140c3180 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:45.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 -- 172.21.15.98:0/3133517074 >> 172.21.15.98:0/3133517074 conn(0x7f1e140cdeb0 msgr2=0x7f1e140ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:45.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 -- 172.21.15.98:0/3133517074 shutdown_connections 2024-05-27T00:18:45.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 -- 172.21.15.98:0/3133517074 wait complete. 2024-05-27T00:18:45.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 Processor -- start 2024-05-27T00:18:45.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 -- start start 2024-05-27T00:18:45.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1e14070660 0x7f1e141de8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:45.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140c36c0 0x7f1e141e3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:45.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e141deec0 0x7f1e140cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:45.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e140c9940 con 0x7f1e140c36c0 2024-05-27T00:18:45.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1e140c9420 con 0x7f1e141deec0 2024-05-27T00:18:45.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e1a200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1e140c9ac0 con 0x7f1e14070660 2024-05-27T00:18:45.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e13e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140c36c0 0x7f1e141e3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:45.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e13e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140c36c0 0x7f1e141e3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39164/0 (socket says 172.21.15.98:39164) 2024-05-27T00:18:45.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e13e00640 1 -- 172.21.15.98:0/2081615907 learned_addr learned my addr 172.21.15.98:0/2081615907 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:45.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e18e00640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1e14070660 0x7f1e141de8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:45.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e19800640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e141deec0 0x7f1e140cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:45.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e13e00640 1 -- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1e14070660 msgr2=0x7f1e141de8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:45.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e13e00640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1e14070660 0x7f1e141de8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:45.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e13e00640 1 -- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e141deec0 msgr2=0x7f1e140cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:45.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e13e00640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e141deec0 0x7f1e140cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:45.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e13e00640 1 -- 172.21.15.98:0/2081615907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e141f0a30 con 0x7f1e140c36c0 2024-05-27T00:18:45.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.508+0000 7f1e13e00640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140c36c0 0x7f1e141e3ff0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1e000097c0 tx=0x7f1e00019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:45.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.512+0000 7f1e10c00640 1 -- 172.21.15.98:0/2081615907 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e000b2040 con 0x7f1e140c36c0 2024-05-27T00:18:45.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.512+0000 7f1e10c00640 1 -- 172.21.15.98:0/2081615907 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e000bc070 con 0x7f1e140c36c0 2024-05-27T00:18:45.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.512+0000 7f1e10c00640 1 -- 172.21.15.98:0/2081615907 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e000b6870 con 0x7f1e140c36c0 2024-05-27T00:18:45.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.512+0000 7f1e1a200640 1 -- 172.21.15.98:0/2081615907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e141f0d10 con 0x7f1e140c36c0 2024-05-27T00:18:45.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.512+0000 7f1e1a200640 1 -- 172.21.15.98:0/2081615907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e141dddc0 con 0x7f1e140c36c0 2024-05-27T00:18:45.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.512+0000 7f1e1a200640 1 -- 172.21.15.98:0/2081615907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e14075c40 con 0x7f1e140c36c0 2024-05-27T00:18:45.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.512+0000 7f1e10c00640 1 -- 172.21.15.98:0/2081615907 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1e000ab070 con 0x7f1e140c36c0 2024-05-27T00:18:45.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.512+0000 7f1e10c00640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1df40678f0 0x7f1df4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:45.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.512+0000 7f1e10c00640 1 -- 172.21.15.98:0/2081615907 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f1e0011fed0 con 0x7f1e140c36c0 2024-05-27T00:18:45.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.512+0000 7f1e18e00640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1df40678f0 0x7f1df4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:45.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.516+0000 7f1e18e00640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1df40678f0 0x7f1df4069db0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1e04004880 tx=0x7f1e0400acf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:45.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.516+0000 7f1e10c00640 1 -- 172.21.15.98:0/2081615907 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1e00015fb0 con 0x7f1e140c36c0 2024-05-27T00:18:45.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 -- 172.21.15.98:0/2081615907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e140cc970 con 0x7f1e140c36c0 2024-05-27T00:18:45.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e10c00640 1 -- 172.21.15.98:0/2081615907 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6014 (secure 0 0 0) 0x7f1e000f08f0 con 0x7f1e140c36c0 2024-05-27T00:18:45.716 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:45.716 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:44.479020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4894,"qdb_cluster":[4894]},"id":1}]} 2024-05-27T00:18:45.716 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2024-05-27T00:18:45.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 -- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1df40678f0 msgr2=0x7f1df4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:45.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1df40678f0 0x7f1df4069db0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1e04004880 tx=0x7f1e0400acf0 comp rx=0 tx=0).stop 2024-05-27T00:18:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 -- 172.21.15.98:0/2081615907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140c36c0 msgr2=0x7f1e141e3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140c36c0 0x7f1e141e3ff0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1e000097c0 tx=0x7f1e00019160 comp rx=0 tx=0).stop 2024-05-27T00:18:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 -- 172.21.15.98:0/2081615907 shutdown_connections 2024-05-27T00:18:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1df40678f0 0x7f1df4069db0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e141deec0 0x7f1e140cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e140c36c0 0x7f1e141e3ff0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 --2- 172.21.15.98:0/2081615907 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1e14070660 0x7f1e141de8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 -- 172.21.15.98:0/2081615907 >> 172.21.15.98:0/2081615907 conn(0x7f1e140cdeb0 msgr2=0x7f1e141dc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 -- 172.21.15.98:0/2081615907 shutdown_connections 2024-05-27T00:18:45.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:45.712+0000 7f1e1a200640 1 -- 172.21.15.98:0/2081615907 wait complete. 2024-05-27T00:18:45.737 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=4281 2024-05-27T00:18:45.737 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.a 2024-05-27T00:18:52.504 INFO:tasks.ceph.mds.c.smithi098.stderr:2024-05-27T00:18:52.500+0000 7f5fb8400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:18:54.738 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-05-27T00:18:54.739 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-27T00:18:54.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-27T00:18:54.771 INFO:tasks.ceph.mds.a:Started 2024-05-27T00:18:54.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:55.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5247200640 1 Processor -- start 2024-05-27T00:18:55.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5247200640 1 -- start start 2024-05-27T00:18:55.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5247200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f524808cbd0 0x7f524816ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5247200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5248164d30 0x7f524816b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5247200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52480c6a10 con 0x7f524806e7a0 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5247200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f52480c6cf0 con 0x7f524808cbd0 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5247200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f52480c6fd0 con 0x7f5248164d30 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f523fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f524808cbd0 0x7f524816ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5245e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f524806e7a0 0x7f5248164620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57882/0 (socket says 172.21.15.98:57882) 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f523fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f524808cbd0 0x7f524816ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55904/0 (socket says 172.21.15.98:55904) 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5245e00640 1 -- 172.21.15.98:0/2124526993 learned_addr learned my addr 172.21.15.98:0/2124526993 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5245400640 1 -- 172.21.15.98:0/2124526993 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4053794341 0 0) 0x7f52480c6a10 con 0x7f524806e7a0 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:54.960+0000 7f5246800640 1 --2- 172.21.15.98:0/2124526993 >> v2:172.21.15.204:3300/0 conn(0x7f5248164d30 0x7f524816b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.116+0000 7f5245400640 1 -- 172.21.15.98:0/2124526993 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f522c003510 con 0x7f524806e7a0 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.116+0000 7f523fe00640 1 -- 172.21.15.98:0/2124526993 >> v2:172.21.15.204:3300/0 conn(0x7f5248164d30 msgr2=0x7f524816b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:18:55.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.116+0000 7f523fe00640 1 --2- 172.21.15.98:0/2124526993 >> v2:172.21.15.204:3300/0 conn(0x7f5248164d30 0x7f524816b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.116+0000 7f5245400640 1 -- 172.21.15.98:0/2124526993 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 973684910 0 0) 0x7f522c003510 con 0x7f524806e7a0 2024-05-27T00:18:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.116+0000 7f523fe00640 1 -- 172.21.15.98:0/2124526993 >> v1:172.21.15.98:6789/0 conn(0x7f524806e7a0 legacy=0x7f5248164620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:55.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.116+0000 7f523fe00640 1 -- 172.21.15.98:0/2124526993 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52480c72b0 con 0x7f524808cbd0 2024-05-27T00:18:55.121 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2024-05-27T00:18:55.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f523fe00640 1 --2- 172.21.15.98:0/2124526993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524808cbd0 0x7f524816ac20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f523000b8c0 tx=0x7f523001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc0dc380a71547a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:55.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5245400640 1 -- 172.21.15.98:0/2124526993 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52300ab070 con 0x7f524808cbd0 2024-05-27T00:18:55.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5245400640 1 -- 172.21.15.98:0/2124526993 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52300a5720 con 0x7f524808cbd0 2024-05-27T00:18:55.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5245400640 1 -- 172.21.15.98:0/2124526993 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5238001680 con 0x7f524808cbd0 2024-05-27T00:18:55.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5247200640 1 -- 172.21.15.98:0/2124526993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524808cbd0 msgr2=0x7f524816ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:55.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5247200640 1 --2- 172.21.15.98:0/2124526993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524808cbd0 0x7f524816ac20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f523000b8c0 tx=0x7f523001c990 comp rx=0 tx=0).stop 2024-05-27T00:18:55.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5247200640 1 -- 172.21.15.98:0/2124526993 shutdown_connections 2024-05-27T00:18:55.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5247200640 1 --2- 172.21.15.98:0/2124526993 >> v2:172.21.15.204:3300/0 conn(0x7f5248164d30 0x7f524816b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:55.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5247200640 1 --2- 172.21.15.98:0/2124526993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524808cbd0 0x7f524816ac20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:55.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5247200640 1 -- 172.21.15.98:0/2124526993 >> 172.21.15.98:0/2124526993 conn(0x7f52480cbe00 msgr2=0x7f52480cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:55.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5247200640 1 -- 172.21.15.98:0/2124526993 shutdown_connections 2024-05-27T00:18:55.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.120+0000 7f5247200640 1 -- 172.21.15.98:0/2124526993 wait complete. 2024-05-27T00:18:55.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5247200640 1 Processor -- start 2024-05-27T00:18:55.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5247200640 1 -- start start 2024-05-27T00:18:55.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5247200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524806e7a0 0x7f524815a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:55.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5247200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f524808cbd0 0x7f52480cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:55.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5247200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5248164d30 0x7f5248158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:55.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5247200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f524806f390 con 0x7f524806e7a0 2024-05-27T00:18:55.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5247200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f52481744d0 con 0x7f524808cbd0 2024-05-27T00:18:55.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5247200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f524815ad40 con 0x7f5248164d30 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5245e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524806e7a0 0x7f524815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5245e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524806e7a0 0x7f524815a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55914/0 (socket says 172.21.15.98:55914) 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5245e00640 1 -- 172.21.15.98:0/2445602307 learned_addr learned my addr 172.21.15.98:0/2445602307 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f523fe00640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f524808cbd0 0x7f52480cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5246800640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5248164d30 0x7f5248158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5245e00640 1 -- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5248164d30 msgr2=0x7f5248158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.124+0000 7f5245e00640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5248164d30 0x7f5248158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f5245e00640 1 -- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f524808cbd0 msgr2=0x7f52480cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f5245e00640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f524808cbd0 0x7f52480cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f5245e00640 1 -- 172.21.15.98:0/2445602307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5248200860 con 0x7f524806e7a0 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f5245e00640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524806e7a0 0x7f524815a690 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f523800c750 tx=0x7f523800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f523e000640 1 -- 172.21.15.98:0/2445602307 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5238005ae0 con 0x7f524806e7a0 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f523e000640 1 -- 172.21.15.98:0/2445602307 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5238005ca0 con 0x7f524806e7a0 2024-05-27T00:18:55.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f523e000640 1 -- 172.21.15.98:0/2445602307 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52380072b0 con 0x7f524806e7a0 2024-05-27T00:18:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f5247200640 1 -- 172.21.15.98:0/2445602307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5248200b40 con 0x7f524806e7a0 2024-05-27T00:18:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f5247200640 1 -- 172.21.15.98:0/2445602307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5248201100 con 0x7f524806e7a0 2024-05-27T00:18:55.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f5247200640 1 -- 172.21.15.98:0/2445602307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5248075c40 con 0x7f524806e7a0 2024-05-27T00:18:55.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f523e000640 1 -- 172.21.15.98:0/2445602307 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5238007450 con 0x7f524806e7a0 2024-05-27T00:18:55.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f523e000640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52240678f0 0x7f5224069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:55.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.128+0000 7f523e000640 1 -- 172.21.15.98:0/2445602307 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f5238085d10 con 0x7f524806e7a0 2024-05-27T00:18:55.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.132+0000 7f523fe00640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52240678f0 0x7f5224069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:55.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.132+0000 7f523fe00640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52240678f0 0x7f5224069db0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f52300b0000 tx=0x7f52300a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:55.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.132+0000 7f523e000640 1 -- 172.21.15.98:0/2445602307 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f523808b020 con 0x7f524806e7a0 2024-05-27T00:18:55.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.384+0000 7f5247200640 1 -- 172.21.15.98:0/2445602307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52480c2ac0 con 0x7f524806e7a0 2024-05-27T00:18:55.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.384+0000 7f523e000640 1 -- 172.21.15.98:0/2445602307 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6014 (secure 0 0 0) 0x7f5238056690 con 0x7f524806e7a0 2024-05-27T00:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:44.479020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4894,"qdb_cluster":[4894]},"id":1}]} 2024-05-27T00:18:55.389 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2024-05-27T00:18:55.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 -- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52240678f0 msgr2=0x7f5224069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:55.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52240678f0 0x7f5224069db0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f52300b0000 tx=0x7f52300a7070 comp rx=0 tx=0).stop 2024-05-27T00:18:55.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 -- 172.21.15.98:0/2445602307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524806e7a0 msgr2=0x7f524815a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524806e7a0 0x7f524815a690 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f523800c750 tx=0x7f523800cc10 comp rx=0 tx=0).stop 2024-05-27T00:18:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 -- 172.21.15.98:0/2445602307 shutdown_connections 2024-05-27T00:18:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5248164d30 0x7f5248158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f52240678f0 0x7f5224069db0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f524808cbd0 0x7f52480cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 --2- 172.21.15.98:0/2445602307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f524806e7a0 0x7f524815a690 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 -- 172.21.15.98:0/2445602307 >> 172.21.15.98:0/2445602307 conn(0x7f52480cbe00 msgr2=0x7f52481638b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 -- 172.21.15.98:0/2445602307 shutdown_connections 2024-05-27T00:18:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:55.388+0000 7f5247200640 1 -- 172.21.15.98:0/2445602307 wait complete. 2024-05-27T00:18:55.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-05-27T00:18:57.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:18:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.547+0000 7f75b7000640 1 Processor -- start 2024-05-27T00:18:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75b7000640 1 -- start start 2024-05-27T00:18:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75b7000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f75b00cdf90 0x7f75b006f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75b7000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b006cfb0 con 0x7f75b00c4fd0 2024-05-27T00:18:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75b7000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b00ca840 con 0x7f75b00ce370 2024-05-27T00:18:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75b7000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f75b006d500 con 0x7f75b00cdf90 2024-05-27T00:18:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75af400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f75b00c4fd0 0x7f75b00c8fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57890/0 (socket says 172.21.15.98:57890) 2024-05-27T00:18:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75af400640 1 -- 172.21.15.98:0/2160410184 learned_addr learned my addr 172.21.15.98:0/2160410184 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:18:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75afe00640 1 --2- 172.21.15.98:0/2160410184 >> v2:172.21.15.204:3300/0 conn(0x7f75b00cdf90 0x7f75b006f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75aea00640 1 -- 172.21.15.98:0/2160410184 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 84825729 0 0) 0x7f75b006cfb0 con 0x7f75b00c4fd0 2024-05-27T00:18:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75aea00640 1 -- 172.21.15.98:0/2160410184 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7594003530 con 0x7f75b00c4fd0 2024-05-27T00:18:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75afe00640 1 -- 172.21.15.98:0/2160410184 >> v1:172.21.15.204:6789/0 conn(0x7f75b00ce370 legacy=0x7f75b0165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75afe00640 1 -- 172.21.15.98:0/2160410184 >> v1:172.21.15.98:6789/0 conn(0x7f75b00c4fd0 legacy=0x7f75b00c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.551+0000 7f75afe00640 1 -- 172.21.15.98:0/2160410184 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b006b610 con 0x7f75b00cdf90 2024-05-27T00:18:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75afe00640 1 --2- 172.21.15.98:0/2160410184 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b00cdf90 0x7f75b006f760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f75a00097c0 tx=0x7f75a001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d64db13a342996b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75aea00640 1 -- 172.21.15.98:0/2160410184 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00b5070 con 0x7f75b00cdf90 2024-05-27T00:18:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75aea00640 1 -- 172.21.15.98:0/2160410184 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a00a8990 con 0x7f75b00cdf90 2024-05-27T00:18:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75aea00640 1 -- 172.21.15.98:0/2160410184 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00a8d40 con 0x7f75b00cdf90 2024-05-27T00:18:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75b7000640 1 -- 172.21.15.98:0/2160410184 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b00cdf90 msgr2=0x7f75b006f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75b7000640 1 --2- 172.21.15.98:0/2160410184 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b00cdf90 0x7f75b006f760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f75a00097c0 tx=0x7f75a001ec90 comp rx=0 tx=0).stop 2024-05-27T00:18:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75b7000640 1 -- 172.21.15.98:0/2160410184 shutdown_connections 2024-05-27T00:18:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75b7000640 1 --2- 172.21.15.98:0/2160410184 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b00cdf90 0x7f75b006f760 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75b7000640 1 -- 172.21.15.98:0/2160410184 >> 172.21.15.98:0/2160410184 conn(0x7f75b006d680 msgr2=0x7f75b006da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75b7000640 1 -- 172.21.15.98:0/2160410184 shutdown_connections 2024-05-27T00:18:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.555+0000 7f75b7000640 1 -- 172.21.15.98:0/2160410184 wait complete. 2024-05-27T00:18:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 Processor -- start 2024-05-27T00:18:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 -- start start 2024-05-27T00:18:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75b00c4fd0 0x7f75b00c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b00cdf90 0x7f75b014ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75b00ce370 0x7f75b01527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75b00ce800 con 0x7f75b00ce370 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f75b006ef50 con 0x7f75b00cdf90 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f75b006d220 con 0x7f75b00c4fd0 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b5000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75b00ce370 0x7f75b01527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b5000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75b00ce370 0x7f75b01527f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55922/0 (socket says 172.21.15.98:55922) 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b5000640 1 -- 172.21.15.98:0/4006160302 learned_addr learned my addr 172.21.15.98:0/4006160302 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75afe00640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75b00c4fd0 0x7f75b00c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75af400640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b00cdf90 0x7f75b014ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b5000640 1 -- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75b00c4fd0 msgr2=0x7f75b00c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b5000640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75b00c4fd0 0x7f75b00c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b5000640 1 -- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b00cdf90 msgr2=0x7f75b014ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b5000640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b00cdf90 0x7f75b014ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b5000640 1 -- 172.21.15.98:0/4006160302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b01536f0 con 0x7f75b00ce370 2024-05-27T00:18:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75af400640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b00cdf90 0x7f75b014ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:18:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b5000640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75b00ce370 0x7f75b01527f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f75a800c770 tx=0x7f75a800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75a7e00640 1 -- 172.21.15.98:0/4006160302 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8003b80 con 0x7f75b00ce370 2024-05-27T00:18:57.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75a7e00640 1 -- 172.21.15.98:0/4006160302 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a8003d20 con 0x7f75b00ce370 2024-05-27T00:18:57.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75a7e00640 1 -- 172.21.15.98:0/4006160302 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a800aa90 con 0x7f75b00ce370 2024-05-27T00:18:57.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 -- 172.21.15.98:0/4006160302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b01539b0 con 0x7f75b00ce370 2024-05-27T00:18:57.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 -- 172.21.15.98:0/4006160302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b0153f70 con 0x7f75b00ce370 2024-05-27T00:18:57.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.559+0000 7f75b7000640 1 -- 172.21.15.98:0/4006160302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b0075cb0 con 0x7f75b00ce370 2024-05-27T00:18:57.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.563+0000 7f75a7e00640 1 -- 172.21.15.98:0/4006160302 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f75a800ac30 con 0x7f75b00ce370 2024-05-27T00:18:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.563+0000 7f75a7e00640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f75900678c0 0x7f7590069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:18:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.563+0000 7f75a7e00640 1 -- 172.21.15.98:0/4006160302 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f75a8085ce0 con 0x7f75b00ce370 2024-05-27T00:18:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.563+0000 7f75afe00640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f75900678c0 0x7f7590069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:18:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.563+0000 7f75afe00640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f75900678c0 0x7f7590069d80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75a00b2000 tx=0x7f75a00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:18:57.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.567+0000 7f75a7e00640 1 -- 172.21.15.98:0/4006160302 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f75a8056660 con 0x7f75b00ce370 2024-05-27T00:18:57.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.811+0000 7f75b7000640 1 -- 172.21.15.98:0/4006160302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75b006f540 con 0x7f75b00ce370 2024-05-27T00:18:57.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.811+0000 7f75a7e00640 1 -- 172.21.15.98:0/4006160302 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6839 (secure 0 0 0) 0x7f75a8055f60 con 0x7f75b00ce370 2024-05-27T00:18:57.817 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:18:57.817 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5073,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:44.479020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4894,"qdb_cluster":[4894]},"id":1}]} 2024-05-27T00:18:57.817 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 18 2024-05-27T00:18:57.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 -- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f75900678c0 msgr2=0x7f7590069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f75900678c0 0x7f7590069d80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75a00b2000 tx=0x7f75a00aa070 comp rx=0 tx=0).stop 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 -- 172.21.15.98:0/4006160302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75b00ce370 msgr2=0x7f75b01527f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75b00ce370 0x7f75b01527f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f75a800c770 tx=0x7f75a800cc30 comp rx=0 tx=0).stop 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 -- 172.21.15.98:0/4006160302 shutdown_connections 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75b00ce370 0x7f75b01527f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f75900678c0 0x7f7590069d80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b00cdf90 0x7f75b014ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 --2- 172.21.15.98:0/4006160302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75b00c4fd0 0x7f75b00c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 -- 172.21.15.98:0/4006160302 >> 172.21.15.98:0/4006160302 conn(0x7f75b006d680 msgr2=0x7f75b00c3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 -- 172.21.15.98:0/4006160302 shutdown_connections 2024-05-27T00:18:57.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:18:57.815+0000 7f75b7000640 1 -- 172.21.15.98:0/4006160302 wait complete. 2024-05-27T00:18:57.839 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-05-27T00:18:57.839 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 21 secs before thrashing 2024-05-27T00:19:14.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:19:14.908 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:19:18.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742d800640 1 Processor -- start 2024-05-27T00:19:19.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742d800640 1 -- start start 2024-05-27T00:19:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742d800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7428070660 0x7f74280c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742d800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74280c36c0 0x7f74280c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742d800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7428071010 con 0x7f74280b3260 2024-05-27T00:19:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742d800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f74280c88c0 con 0x7f74280c36c0 2024-05-27T00:19:19.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742d800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f74280c8b80 con 0x7f7428070660 2024-05-27T00:19:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742ce00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74280c36c0 0x7f74280c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742ce00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74280c36c0 0x7f74280c9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34770/0 (socket says 172.21.15.98:34770) 2024-05-27T00:19:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742ce00640 1 -- 172.21.15.98:0/865260925 learned_addr learned my addr 172.21.15.98:0/865260925 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f7427400640 1 --2- 172.21.15.98:0/865260925 >> v2:172.21.15.204:3300/0 conn(0x7f7428070660 0x7f74280c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742ce00640 1 -- 172.21.15.98:0/865260925 >> v2:172.21.15.204:3300/0 conn(0x7f7428070660 msgr2=0x7f74280c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742ce00640 1 --2- 172.21.15.98:0/865260925 >> v2:172.21.15.204:3300/0 conn(0x7f7428070660 0x7f74280c3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742ce00640 1 -- 172.21.15.98:0/865260925 >> v1:172.21.15.204:6789/0 conn(0x7f74280b3260 legacy=0x7f74280c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f7426a00640 1 -- 172.21.15.98:0/865260925 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 580809676 0 0) 0x7f7418001120 con 0x7f74280b3260 2024-05-27T00:19:19.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.067+0000 7f742ce00640 1 -- 172.21.15.98:0/865260925 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74280c8e60 con 0x7f74280c36c0 2024-05-27T00:19:19.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f742ce00640 1 --2- 172.21.15.98:0/865260925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74280c36c0 0x7f74280c9ef0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f74280c67a0 tx=0x7f742001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfa01af8f224eb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:19.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f7426a00640 1 -- 172.21.15.98:0/865260925 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74200b4070 con 0x7f74280c36c0 2024-05-27T00:19:19.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f7426a00640 1 -- 172.21.15.98:0/865260925 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74200af400 con 0x7f74280c36c0 2024-05-27T00:19:19.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f7426a00640 1 -- 172.21.15.98:0/865260925 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74200af7d0 con 0x7f74280c36c0 2024-05-27T00:19:19.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f742d800640 1 -- 172.21.15.98:0/865260925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74280c36c0 msgr2=0x7f74280c9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:19.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f742d800640 1 --2- 172.21.15.98:0/865260925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74280c36c0 0x7f74280c9ef0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f74280c67a0 tx=0x7f742001f1e0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f742d800640 1 -- 172.21.15.98:0/865260925 shutdown_connections 2024-05-27T00:19:19.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f742d800640 1 --2- 172.21.15.98:0/865260925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74280c36c0 0x7f74280c9ef0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f742d800640 1 --2- 172.21.15.98:0/865260925 >> v2:172.21.15.204:3300/0 conn(0x7f7428070660 0x7f74280c3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f742d800640 1 -- 172.21.15.98:0/865260925 >> 172.21.15.98:0/865260925 conn(0x7f74280cdeb0 msgr2=0x7f74280ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:19.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f742d800640 1 -- 172.21.15.98:0/865260925 shutdown_connections 2024-05-27T00:19:19.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.075+0000 7f742d800640 1 -- 172.21.15.98:0/865260925 wait complete. 2024-05-27T00:19:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742d800640 1 Processor -- start 2024-05-27T00:19:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742d800640 1 -- start start 2024-05-27T00:19:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742d800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7428070660 0x7f74281e5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74280c36c0 0x7f74280af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742d800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f742806e690 0x7f74280af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742d800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f74280c9940 con 0x7f742806e690 2024-05-27T00:19:19.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742d800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f74280c9420 con 0x7f74280c36c0 2024-05-27T00:19:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742d800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f74280c9ac0 con 0x7f7428070660 2024-05-27T00:19:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742ce00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f742806e690 0x7f74280af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f7427400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74280c36c0 0x7f74280af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f7427400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74280c36c0 0x7f74280af3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:43856/0 (socket says 172.21.15.98:43856) 2024-05-27T00:19:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f7427400640 1 -- 172.21.15.98:0/942579907 learned_addr learned my addr 172.21.15.98:0/942579907 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742ce00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f742806e690 0x7f74280af8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34782/0 (socket says 172.21.15.98:34782) 2024-05-27T00:19:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f7427e00640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7428070660 0x7f74281e5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f7427400640 1 -- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7428070660 msgr2=0x7f74281e5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f7427400640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7428070660 0x7f74281e5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f7427400640 1 -- 172.21.15.98:0/942579907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f742806e690 msgr2=0x7f74280af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f7427400640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f742806e690 0x7f74280af8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f7427400640 1 -- 172.21.15.98:0/942579907 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742806ecb0 con 0x7f74280c36c0 2024-05-27T00:19:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f742ce00640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f742806e690 0x7f74280af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:19:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.079+0000 7f7427400640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74280c36c0 0x7f74280af3a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f741400e870 tx=0x7f741400ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.083+0000 7f741fe00640 1 -- 172.21.15.98:0/942579907 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414017070 con 0x7f74280c36c0 2024-05-27T00:19:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.083+0000 7f741fe00640 1 -- 172.21.15.98:0/942579907 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f741400c940 con 0x7f74280c36c0 2024-05-27T00:19:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.083+0000 7f742d800640 1 -- 172.21.15.98:0/942579907 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74280cb710 con 0x7f74280c36c0 2024-05-27T00:19:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.083+0000 7f741fe00640 1 -- 172.21.15.98:0/942579907 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741400ccf0 con 0x7f74280c36c0 2024-05-27T00:19:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.083+0000 7f742d800640 1 -- 172.21.15.98:0/942579907 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74280c19b0 con 0x7f74280c36c0 2024-05-27T00:19:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.083+0000 7f742d800640 1 -- 172.21.15.98:0/942579907 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7428075c40 con 0x7f74280c36c0 2024-05-27T00:19:19.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.087+0000 7f741fe00640 1 -- 172.21.15.98:0/942579907 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f741401a070 con 0x7f74280c36c0 2024-05-27T00:19:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.087+0000 7f741fe00640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74080678f0 0x7f7408069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.087+0000 7f741fe00640 1 -- 172.21.15.98:0/942579907 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f7414086280 con 0x7f74280c36c0 2024-05-27T00:19:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.087+0000 7f7427e00640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74080678f0 0x7f7408069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.087+0000 7f7427e00640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74080678f0 0x7f7408069db0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7418007fd0 tx=0x7f7418009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.099+0000 7f741fe00640 1 -- 172.21.15.98:0/942579907 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7414056730 con 0x7f74280c36c0 2024-05-27T00:19:19.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.495+0000 7f742d800640 1 -- 172.21.15.98:0/942579907 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74280c5480 con 0x7f74280c36c0 2024-05-27T00:19:19.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f741fe00640 1 -- 172.21.15.98:0/942579907 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6839 (secure 0 0 0) 0x7f741405aa30 con 0x7f74280c36c0 2024-05-27T00:19:19.503 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:19.503 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5073,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:44.479020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4894,"qdb_cluster":[4894]},"id":1}]} 2024-05-27T00:19:19.503 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 18 2024-05-27T00:19:19.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 -- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74080678f0 msgr2=0x7f7408069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:19.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74080678f0 0x7f7408069db0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7418007fd0 tx=0x7f7418009040 comp rx=0 tx=0).stop 2024-05-27T00:19:19.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 -- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74280c36c0 msgr2=0x7f74280af3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:19.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74280c36c0 0x7f74280af3a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f741400e870 tx=0x7f741400ec70 comp rx=0 tx=0).stop 2024-05-27T00:19:19.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 -- 172.21.15.98:0/942579907 shutdown_connections 2024-05-27T00:19:19.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f742806e690 0x7f74280af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74080678f0 0x7f7408069db0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74280c36c0 0x7f74280af3a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 --2- 172.21.15.98:0/942579907 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7428070660 0x7f74281e5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 -- 172.21.15.98:0/942579907 >> 172.21.15.98:0/942579907 conn(0x7f74280cdeb0 msgr2=0x7f74280c41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:19.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.499+0000 7f742d800640 1 -- 172.21.15.98:0/942579907 shutdown_connections 2024-05-27T00:19:19.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.503+0000 7f742d800640 1 -- 172.21.15.98:0/942579907 wait complete. 2024-05-27T00:19:19.535 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-05-27T00:19:19.536 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-27T00:19:19.536 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:19:19.571 INFO:tasks.ceph.mds.c:Stopped 2024-05-27T00:19:19.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.835+0000 7fcd0e600640 1 Processor -- start 2024-05-27T00:19:19.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.835+0000 7fcd0e600640 1 -- start start 2024-05-27T00:19:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0e600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd080b3260 0x7fcd080c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0e600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fcd08070aa0 0x7fcd080c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0e600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd08071010 con 0x7fcd08070660 2024-05-27T00:19:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0e600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fcd080c3890 con 0x7fcd08070aa0 2024-05-27T00:19:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0e600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcd080c88c0 con 0x7fcd080b3260 2024-05-27T00:19:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd07e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcd08070660 0x7fcd080c3180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48906/0 (socket says 172.21.15.98:48906) 2024-05-27T00:19:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd07e00640 1 -- 172.21.15.98:0/1277907444 learned_addr learned my addr 172.21.15.98:0/1277907444 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:19:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0d200640 1 --2- 172.21.15.98:0/1277907444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd080b3260 0x7fcd080c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0dc00640 1 --2- 172.21.15.98:0/1277907444 >> v2:172.21.15.204:3300/0 conn(0x7fcd08070aa0 0x7fcd080c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0dc00640 1 -- 172.21.15.98:0/1277907444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd080b3260 msgr2=0x7fcd080c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:19.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd07400640 1 -- 172.21.15.98:0/1277907444 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1336356001 0 0) 0x7fccf4001120 con 0x7fcd08070660 2024-05-27T00:19:19.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0dc00640 1 --2- 172.21.15.98:0/1277907444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd080b3260 0x7fcd080c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0dc00640 1 -- 172.21.15.98:0/1277907444 >> v1:172.21.15.98:6789/0 conn(0x7fcd08070660 legacy=0x7fcd080c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:19.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0dc00640 1 -- 172.21.15.98:0/1277907444 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd080c8b80 con 0x7fcd08070aa0 2024-05-27T00:19:19.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.839+0000 7fcd0d200640 1 --2- 172.21.15.98:0/1277907444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd080b3260 0x7fcd080c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:19:19.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd0dc00640 1 --2- 172.21.15.98:0/1277907444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd08070aa0 0x7fcd080c9ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009d90 tx=0x7fcd0001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ac6a179820c88cc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:19.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd07400640 1 -- 172.21.15.98:0/1277907444 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd000b4070 con 0x7fcd08070aa0 2024-05-27T00:19:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd07400640 1 -- 172.21.15.98:0/1277907444 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd000af600 con 0x7fcd08070aa0 2024-05-27T00:19:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd07400640 1 -- 172.21.15.98:0/1277907444 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd000af9d0 con 0x7fcd08070aa0 2024-05-27T00:19:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd0e600640 1 -- 172.21.15.98:0/1277907444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd08070aa0 msgr2=0x7fcd080c9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd0e600640 1 --2- 172.21.15.98:0/1277907444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd08070aa0 0x7fcd080c9ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009d90 tx=0x7fcd0001f150 comp rx=0 tx=0).stop 2024-05-27T00:19:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd0e600640 1 -- 172.21.15.98:0/1277907444 shutdown_connections 2024-05-27T00:19:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd0e600640 1 --2- 172.21.15.98:0/1277907444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd08070aa0 0x7fcd080c9ef0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd0e600640 1 --2- 172.21.15.98:0/1277907444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd080b3260 0x7fcd080c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd0e600640 1 -- 172.21.15.98:0/1277907444 >> 172.21.15.98:0/1277907444 conn(0x7fcd080cdeb0 msgr2=0x7fcd080ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd0e600640 1 -- 172.21.15.98:0/1277907444 shutdown_connections 2024-05-27T00:19:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.843+0000 7fcd0e600640 1 -- 172.21.15.98:0/1277907444 wait complete. 2024-05-27T00:19:19.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.847+0000 7fcd0e600640 1 Processor -- start 2024-05-27T00:19:19.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.847+0000 7fcd0e600640 1 -- start start 2024-05-27T00:19:19.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.847+0000 7fcd0e600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd08070660 0x7fcd08156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.847+0000 7fcd0e600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd08070aa0 0x7fcd081598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.847+0000 7fcd0e600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd0815a1a0 0x7fcd081f3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.847+0000 7fcd0e600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd080c9420 con 0x7fcd08070aa0 2024-05-27T00:19:19.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.847+0000 7fcd0e600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcd080c9140 con 0x7fcd0815a1a0 2024-05-27T00:19:19.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.847+0000 7fcd0e600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcd080c8200 con 0x7fcd08070660 2024-05-27T00:19:19.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0d200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd08070660 0x7fcd08156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd07e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd08070aa0 0x7fcd081598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0d200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd08070660 0x7fcd08156fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:60218/0 (socket says 172.21.15.98:60218) 2024-05-27T00:19:19.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0d200640 1 -- 172.21.15.98:0/1390265539 learned_addr learned my addr 172.21.15.98:0/1390265539 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:19.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0dc00640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd0815a1a0 0x7fcd081f3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0d200640 1 -- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd0815a1a0 msgr2=0x7fcd081f3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:19.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0d200640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd0815a1a0 0x7fcd081f3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0d200640 1 -- 172.21.15.98:0/1390265539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd08070aa0 msgr2=0x7fcd081598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:19.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0d200640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd08070aa0 0x7fcd081598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:19.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0d200640 1 -- 172.21.15.98:0/1390265539 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd081550b0 con 0x7fcd08070660 2024-05-27T00:19:19.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd07e00640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd08070aa0 0x7fcd081598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:19:19.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0d200640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd08070660 0x7fcd08156fe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00e980 tx=0x7fccfc00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:19.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd04c00640 1 -- 172.21.15.98:0/1390265539 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc017070 con 0x7fcd08070660 2024-05-27T00:19:19.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd04c00640 1 -- 172.21.15.98:0/1390265539 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccfc00c9a0 con 0x7fcd08070660 2024-05-27T00:19:19.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd04c00640 1 -- 172.21.15.98:0/1390265539 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc00cd70 con 0x7fcd08070660 2024-05-27T00:19:19.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0e600640 1 -- 172.21.15.98:0/1390265539 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd081580b0 con 0x7fcd08070660 2024-05-27T00:19:19.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0e600640 1 -- 172.21.15.98:0/1390265539 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd081568b0 con 0x7fcd08070660 2024-05-27T00:19:19.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.851+0000 7fcd0e600640 1 -- 172.21.15.98:0/1390265539 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd08075c40 con 0x7fcd08070660 2024-05-27T00:19:19.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.855+0000 7fcd04c00640 1 -- 172.21.15.98:0/1390265539 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fccfc01a070 con 0x7fcd08070660 2024-05-27T00:19:19.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.855+0000 7fcd04c00640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcce80678f0 0x7fcce8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:19.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.855+0000 7fcd04c00640 1 -- 172.21.15.98:0/1390265539 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7fccfc086200 con 0x7fcd08070660 2024-05-27T00:19:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.855+0000 7fcd07e00640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcce80678f0 0x7fcce8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:19.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.859+0000 7fcd07e00640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcce80678f0 0x7fcce8069db0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcd080b2c80 tx=0x7fccf4007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:19.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:19.867+0000 7fcd04c00640 1 -- 172.21.15.98:0/1390265539 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fccfc0566b0 con 0x7fcd08070660 2024-05-27T00:19:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.259+0000 7fcd0e600640 1 -- 172.21.15.98:0/1390265539 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd080c4dc0 con 0x7fcd08070660 2024-05-27T00:19:20.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.259+0000 7fcd04c00640 1 -- 172.21.15.98:0/1390265539 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6839 (secure 0 0 0) 0x7fccfc05a9b0 con 0x7fcd08070660 2024-05-27T00:19:20.265 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:20.265 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5073,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:44.479020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4894,"qdb_cluster":[4894]},"id":1}]} 2024-05-27T00:19:20.266 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 18 2024-05-27T00:19:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 -- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcce80678f0 msgr2=0x7fcce8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcce80678f0 0x7fcce8069db0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcd080b2c80 tx=0x7fccf4007b60 comp rx=0 tx=0).stop 2024-05-27T00:19:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 -- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd08070660 msgr2=0x7fcd08156fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd08070660 0x7fcd08156fe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00e980 tx=0x7fccfc00ee40 comp rx=0 tx=0).stop 2024-05-27T00:19:20.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 -- 172.21.15.98:0/1390265539 shutdown_connections 2024-05-27T00:19:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcce80678f0 0x7fcce8069db0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd0815a1a0 0x7fcd081f3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd08070aa0 0x7fcd081598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 --2- 172.21.15.98:0/1390265539 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcd08070660 0x7fcd08156fe0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 -- 172.21.15.98:0/1390265539 >> 172.21.15.98:0/1390265539 conn(0x7fcd080cdeb0 msgr2=0x7fcd080cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 -- 172.21.15.98:0/1390265539 shutdown_connections 2024-05-27T00:19:20.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:20.263+0000 7fcd0e600640 1 -- 172.21.15.98:0/1390265539 wait complete. 2024-05-27T00:19:20.299 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:19:22.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:22.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.623+0000 7fb9c6600640 1 Processor -- start 2024-05-27T00:19:22.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.623+0000 7fb9c6600640 1 -- start start 2024-05-27T00:19:22.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.623+0000 7fb9c6600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb9c00c6900 0x7fb9c00c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:22.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.623+0000 7fb9bf400640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fb9c0154aa0 0x7fb9c015b110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:46020/0 (socket says 172.21.15.98:46020) 2024-05-27T00:19:22.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.627+0000 7fb9bea00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb9c015df20 0x7fb9c00cbbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48910/0 (socket says 172.21.15.98:48910) 2024-05-27T00:19:22.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.631+0000 7fb9bf400640 1 -- 172.21.15.98:0/2070856273 learned_addr learned my addr 172.21.15.98:0/2070856273 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:19:22.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.631+0000 7fb9c6600640 1 -- 172.21.15.98:0/2070856273 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9c00c17a0 con 0x7fb9c015df20 2024-05-27T00:19:22.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.631+0000 7fb9c6600640 1 -- 172.21.15.98:0/2070856273 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9c015a930 con 0x7fb9c0154aa0 2024-05-27T00:19:22.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.631+0000 7fb9c6600640 1 -- 172.21.15.98:0/2070856273 --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb9c015e410 con 0x7fb9c00c6900 2024-05-27T00:19:22.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.631+0000 7fb9bfe00640 1 --2- 172.21.15.98:0/2070856273 >> v2:172.21.15.204:3300/0 conn(0x7fb9c00c6900 0x7fb9c00c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.631+0000 7fb9be000640 1 -- 172.21.15.98:0/2070856273 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4012560761 0 0) 0x7fb9c00c17a0 con 0x7fb9c015df20 2024-05-27T00:19:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.631+0000 7fb9be000640 1 -- 172.21.15.98:0/2070856273 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9a4003530 con 0x7fb9c015df20 2024-05-27T00:19:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.631+0000 7fb9bfe00640 1 -- 172.21.15.98:0/2070856273 >> v1:172.21.15.204:6789/0 conn(0x7fb9c0154aa0 legacy=0x7fb9c015b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.631+0000 7fb9bfe00640 1 -- 172.21.15.98:0/2070856273 >> v1:172.21.15.98:6789/0 conn(0x7fb9c015df20 legacy=0x7fb9c00cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.631+0000 7fb9bfe00640 1 -- 172.21.15.98:0/2070856273 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c015e6f0 con 0x7fb9c00c6900 2024-05-27T00:19:22.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.635+0000 7fb9bfe00640 1 --2- 172.21.15.98:0/2070856273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9c00c6900 0x7fb9c00c6d70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac0097c0 tx=0x7fb9ac01c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20f7981dba69b624 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:22.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.635+0000 7fb9be000640 1 -- 172.21.15.98:0/2070856273 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0b1070 con 0x7fb9c00c6900 2024-05-27T00:19:22.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.635+0000 7fb9be000640 1 -- 172.21.15.98:0/2070856273 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac0ac5f0 con 0x7fb9c00c6900 2024-05-27T00:19:22.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.639+0000 7fb9c6600640 1 -- 172.21.15.98:0/2070856273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9c00c6900 msgr2=0x7fb9c00c6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:22.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.639+0000 7fb9c6600640 1 --2- 172.21.15.98:0/2070856273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9c00c6900 0x7fb9c00c6d70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac0097c0 tx=0x7fb9ac01c4e0 comp rx=0 tx=0).stop 2024-05-27T00:19:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.639+0000 7fb9c6600640 1 -- 172.21.15.98:0/2070856273 shutdown_connections 2024-05-27T00:19:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.639+0000 7fb9c6600640 1 --2- 172.21.15.98:0/2070856273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9c00c6900 0x7fb9c00c6d70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.639+0000 7fb9c6600640 1 -- 172.21.15.98:0/2070856273 >> 172.21.15.98:0/2070856273 conn(0x7fb9c0067530 msgr2=0x7fb9c006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.639+0000 7fb9c6600640 1 -- 172.21.15.98:0/2070856273 shutdown_connections 2024-05-27T00:19:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.639+0000 7fb9c6600640 1 -- 172.21.15.98:0/2070856273 wait complete. 2024-05-27T00:19:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.643+0000 7fb9c6600640 1 Processor -- start 2024-05-27T00:19:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.643+0000 7fb9c6600640 1 -- start start 2024-05-27T00:19:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9c6600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9c00c6900 0x7fb9c0155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9c6600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0154aa0 0x7fb9c00c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9c6600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9c015df20 0x7fb9c00c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9c6600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c00cc2c0 con 0x7fb9c0154aa0 2024-05-27T00:19:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9c6600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c015b820 con 0x7fb9c015df20 2024-05-27T00:19:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9c6600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb9c00cc440 con 0x7fb9c00c6900 2024-05-27T00:19:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bf400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9c00c6900 0x7fb9c0155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0154aa0 0x7fb9c00c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bf400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9c00c6900 0x7fb9c0155670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:60222/0 (socket says 172.21.15.98:60222) 2024-05-27T00:19:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bf400640 1 -- 172.21.15.98:0/1817069145 learned_addr learned my addr 172.21.15.98:0/1817069145 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:22.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bfe00640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9c015df20 0x7fb9c00c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:22.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bfe00640 1 -- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9c00c6900 msgr2=0x7fb9c0155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:22.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bfe00640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9c00c6900 0x7fb9c0155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:22.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bfe00640 1 -- 172.21.15.98:0/1817069145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0154aa0 msgr2=0x7fb9c00c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:22.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bfe00640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0154aa0 0x7fb9c00c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:22.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bfe00640 1 -- 172.21.15.98:0/1817069145 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c0156cc0 con 0x7fb9c015df20 2024-05-27T00:19:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9bfe00640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9c015df20 0x7fb9c00c4740 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac0ac000 tx=0x7fb9ac019280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9b7400640 1 -- 172.21.15.98:0/1817069145 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0b1040 con 0x7fb9c015df20 2024-05-27T00:19:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9c6600640 1 -- 172.21.15.98:0/1817069145 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c0158980 con 0x7fb9c015df20 2024-05-27T00:19:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9b7400640 1 -- 172.21.15.98:0/1817069145 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ac0ac7a0 con 0x7fb9c015df20 2024-05-27T00:19:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.647+0000 7fb9b7400640 1 -- 172.21.15.98:0/1817069145 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0acb70 con 0x7fb9c015df20 2024-05-27T00:19:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.651+0000 7fb9c6600640 1 -- 172.21.15.98:0/1817069145 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c0157240 con 0x7fb9c015df20 2024-05-27T00:19:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.655+0000 7fb9b7400640 1 -- 172.21.15.98:0/1817069145 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb9ac0be020 con 0x7fb9c015df20 2024-05-27T00:19:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.659+0000 7fb9b7400640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9a00678f0 0x7fb9a0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.659+0000 7fb9b4c00640 1 -- 172.21.15.98:0/1817069145 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9940052a0 con 0x7fb9c015df20 2024-05-27T00:19:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.659+0000 7fb9b7400640 1 -- 172.21.15.98:0/1817069145 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7fb9ac11f340 con 0x7fb9c015df20 2024-05-27T00:19:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.659+0000 7fb9bf400640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9a00678f0 0x7fb9a0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.659+0000 7fb9bf400640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9a00678f0 0x7fb9a0069db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0005890 tx=0x7fb9b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:22.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:22.683+0000 7fb9b7400640 1 -- 172.21.15.98:0/1817069145 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb9ac0c5020 con 0x7fb9c015df20 2024-05-27T00:19:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.119+0000 7fb9b4c00640 1 -- 172.21.15.98:0/1817069145 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb994005590 con 0x7fb9c015df20 2024-05-27T00:19:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.119+0000 7fb9b7400640 1 -- 172.21.15.98:0/1817069145 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6839 (secure 0 0 0) 0x7fb9ac0a9070 con 0x7fb9c015df20 2024-05-27T00:19:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:23.125 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5073,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:44.479020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4894,"qdb_cluster":[4894]},"id":1}]} 2024-05-27T00:19:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 18 2024-05-27T00:19:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 -- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9a00678f0 msgr2=0x7fb9a0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9a00678f0 0x7fb9a0069db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0005890 tx=0x7fb9b0009040 comp rx=0 tx=0).stop 2024-05-27T00:19:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 -- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9c015df20 msgr2=0x7fb9c00c4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:23.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9c015df20 0x7fb9c00c4740 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac0ac000 tx=0x7fb9ac019280 comp rx=0 tx=0).stop 2024-05-27T00:19:23.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 -- 172.21.15.98:0/1817069145 shutdown_connections 2024-05-27T00:19:23.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9a00678f0 0x7fb9a0069db0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:23.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9c015df20 0x7fb9c00c4740 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:23.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9c0154aa0 0x7fb9c00c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:23.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 --2- 172.21.15.98:0/1817069145 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9c00c6900 0x7fb9c0155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:23.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 -- 172.21.15.98:0/1817069145 >> 172.21.15.98:0/1817069145 conn(0x7fb9c0067530 msgr2=0x7fb9c0152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:23.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 -- 172.21.15.98:0/1817069145 shutdown_connections 2024-05-27T00:19:23.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:23.135+0000 7fb9b4c00640 1 -- 172.21.15.98:0/1817069145 wait complete. 2024-05-27T00:19:23.180 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:19:25.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:25.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.403+0000 7f12c6000640 1 Processor -- start 2024-05-27T00:19:25.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.403+0000 7f12c6000640 1 -- start start 2024-05-27T00:19:25.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12c6000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f12c0070660 0x7f12c00c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12c6000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f12c00c36c0 0x7f12c00c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12c6000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12c0071010 con 0x7f12c00b3260 2024-05-27T00:19:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12c6000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f12c00c88c0 con 0x7f12c0070660 2024-05-27T00:19:25.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12c6000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f12c00c8b80 con 0x7f12c00c36c0 2024-05-27T00:19:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12bfe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f12c0070660 0x7f12c00c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12bfe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f12c0070660 0x7f12c00c3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47810/0 (socket says 172.21.15.98:47810) 2024-05-27T00:19:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12bfe00640 1 -- 172.21.15.98:0/2154593268 learned_addr learned my addr 172.21.15.98:0/2154593268 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12c5600640 1 --2- 172.21.15.98:0/2154593268 >> v2:172.21.15.204:3300/0 conn(0x7f12c00c36c0 0x7f12c00c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12bfe00640 1 -- 172.21.15.98:0/2154593268 >> v2:172.21.15.204:3300/0 conn(0x7f12c00c36c0 msgr2=0x7f12c00c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12bfe00640 1 --2- 172.21.15.98:0/2154593268 >> v2:172.21.15.204:3300/0 conn(0x7f12c00c36c0 0x7f12c00c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12bfe00640 1 -- 172.21.15.98:0/2154593268 >> v1:172.21.15.98:6789/0 conn(0x7f12c00b3260 legacy=0x7f12c00c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:25.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.407+0000 7f12bfe00640 1 -- 172.21.15.98:0/2154593268 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c00c8e60 con 0x7f12c0070660 2024-05-27T00:19:25.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.411+0000 7f12bfe00640 1 --2- 172.21.15.98:0/2154593268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c0070660 0x7f12c00c3180 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f12c00cee80 tx=0x7f12ac01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fe22417afdc5b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:25.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12bf400640 1 -- 172.21.15.98:0/2154593268 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac0b5070 con 0x7f12c0070660 2024-05-27T00:19:25.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12bf400640 1 -- 172.21.15.98:0/2154593268 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12ac0a8a10 con 0x7f12c0070660 2024-05-27T00:19:25.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12bf400640 1 -- 172.21.15.98:0/2154593268 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac0a8de0 con 0x7f12c0070660 2024-05-27T00:19:25.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12c6000640 1 -- 172.21.15.98:0/2154593268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c0070660 msgr2=0x7f12c00c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:25.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12c6000640 1 --2- 172.21.15.98:0/2154593268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c0070660 0x7f12c00c3180 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f12c00cee80 tx=0x7f12ac01ed10 comp rx=0 tx=0).stop 2024-05-27T00:19:25.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12c6000640 1 -- 172.21.15.98:0/2154593268 shutdown_connections 2024-05-27T00:19:25.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12c6000640 1 --2- 172.21.15.98:0/2154593268 >> v2:172.21.15.204:3300/0 conn(0x7f12c00c36c0 0x7f12c00c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:25.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12c6000640 1 --2- 172.21.15.98:0/2154593268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c0070660 0x7f12c00c3180 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:25.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12c6000640 1 -- 172.21.15.98:0/2154593268 >> 172.21.15.98:0/2154593268 conn(0x7f12c00cdeb0 msgr2=0x7f12c00ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:25.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12c6000640 1 -- 172.21.15.98:0/2154593268 shutdown_connections 2024-05-27T00:19:25.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.415+0000 7f12c6000640 1 -- 172.21.15.98:0/2154593268 wait complete. 2024-05-27T00:19:25.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12c6000640 1 Processor -- start 2024-05-27T00:19:25.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12c6000640 1 -- start start 2024-05-27T00:19:25.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12c6000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12c0070660 0x7f12c01df490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:25.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12c6000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c00c36c0 0x7f12c01e4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:25.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12c6000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12c00b1b30 0x7f12c00b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:25.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12c6000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12c00c9940 con 0x7f12c00c36c0 2024-05-27T00:19:25.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12c6000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f12c00c9420 con 0x7f12c0070660 2024-05-27T00:19:25.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12c6000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f12c00c9ac0 con 0x7f12c00b1b30 2024-05-27T00:19:25.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12bfe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c00c36c0 0x7f12c01e4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:25.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12bfe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c00c36c0 0x7f12c01e4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47812/0 (socket says 172.21.15.98:47812) 2024-05-27T00:19:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12bfe00640 1 -- 172.21.15.98:0/428581073 learned_addr learned my addr 172.21.15.98:0/428581073 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12c4c00640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12c0070660 0x7f12c01df490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12c5600640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12c00b1b30 0x7f12c00b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12bfe00640 1 -- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12c00b1b30 msgr2=0x7f12c00b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12bfe00640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12c00b1b30 0x7f12c00b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12bfe00640 1 -- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12c0070660 msgr2=0x7f12c01df490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12bfe00640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12c0070660 0x7f12c01df490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12bfe00640 1 -- 172.21.15.98:0/428581073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c01f3970 con 0x7f12c00c36c0 2024-05-27T00:19:25.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.419+0000 7f12bfe00640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c00c36c0 0x7f12c01e4e70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f12ac01b920 tx=0x7f12ac01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:25.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.423+0000 7f12bcc00640 1 -- 172.21.15.98:0/428581073 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac0b5070 con 0x7f12c00c36c0 2024-05-27T00:19:25.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.423+0000 7f12c6000640 1 -- 172.21.15.98:0/428581073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c01f3c50 con 0x7f12c00c36c0 2024-05-27T00:19:25.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.423+0000 7f12bcc00640 1 -- 172.21.15.98:0/428581073 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12ac0b0ab0 con 0x7f12c00c36c0 2024-05-27T00:19:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.423+0000 7f12bcc00640 1 -- 172.21.15.98:0/428581073 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac0bb410 con 0x7f12c00c36c0 2024-05-27T00:19:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.423+0000 7f12c6000640 1 -- 172.21.15.98:0/428581073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c01f4210 con 0x7f12c00c36c0 2024-05-27T00:19:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.423+0000 7f12c6000640 1 -- 172.21.15.98:0/428581073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c006d470 con 0x7f12c00c36c0 2024-05-27T00:19:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.423+0000 7f12bcc00640 1 -- 172.21.15.98:0/428581073 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f12ac0c2060 con 0x7f12c00c36c0 2024-05-27T00:19:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.427+0000 7f12bcc00640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12a00678c0 0x7f12a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.427+0000 7f12bcc00640 1 -- 172.21.15.98:0/428581073 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f12ac122dc0 con 0x7f12c00c36c0 2024-05-27T00:19:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.427+0000 7f12c4c00640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12a00678c0 0x7f12a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:25.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.427+0000 7f12c4c00640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12a00678c0 0x7f12a0069d80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f12b4001750 tx=0x7f12b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:25.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.435+0000 7f12bcc00640 1 -- 172.21.15.98:0/428581073 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f12c006d470 con 0x7f12c00c36c0 2024-05-27T00:19:25.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.831+0000 7f12c6000640 1 -- 172.21.15.98:0/428581073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12c0075c40 con 0x7f12c00c36c0 2024-05-27T00:19:25.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.831+0000 7f12bcc00640 1 -- 172.21.15.98:0/428581073 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6839 (secure 0 0 0) 0x7f12ac018890 con 0x7f12c00c36c0 2024-05-27T00:19:25.836 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:25.836 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5073,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:44.479020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4894,"qdb_cluster":[4894]},"id":1}]} 2024-05-27T00:19:25.837 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 18 2024-05-27T00:19:25.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 -- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12a00678c0 msgr2=0x7f12a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:25.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12a00678c0 0x7f12a0069d80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f12b4001750 tx=0x7f12b4009040 comp rx=0 tx=0).stop 2024-05-27T00:19:25.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 -- 172.21.15.98:0/428581073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c00c36c0 msgr2=0x7f12c01e4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:25.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c00c36c0 0x7f12c01e4e70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f12ac01b920 tx=0x7f12ac01bab0 comp rx=0 tx=0).stop 2024-05-27T00:19:25.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 -- 172.21.15.98:0/428581073 shutdown_connections 2024-05-27T00:19:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12c00b1b30 0x7f12c00b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c00c36c0 0x7f12c01e4e70 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f12ac01b920 tx=0x7f12ac01bab0 comp rx=0 tx=0).stop 2024-05-27T00:19:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12a00678c0 0x7f12a0069d80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 --2- 172.21.15.98:0/428581073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12c0070660 0x7f12c01df490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 -- 172.21.15.98:0/428581073 >> 172.21.15.98:0/428581073 conn(0x7f12c00cdeb0 msgr2=0x7f12c01e56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 -- 172.21.15.98:0/428581073 shutdown_connections 2024-05-27T00:19:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:25.835+0000 7f12c6000640 1 -- 172.21.15.98:0/428581073 wait complete. 2024-05-27T00:19:25.870 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:19:27.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:28.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48ae400640 1 Processor -- start 2024-05-27T00:19:28.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48ae400640 1 -- start start 2024-05-27T00:19:28.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48ae400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f48a80cc4f0 0x7f48a80c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:28.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48ae400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f48a80c67c0 0x7f48a80cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:28.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48ae400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a80c70b0 con 0x7f48a80629d0 2024-05-27T00:19:28.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48ae400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f48a80c6330 con 0x7f48a80c67c0 2024-05-27T00:19:28.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48ae400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f48a80c6e60 con 0x7f48a80cc4f0 2024-05-27T00:19:28.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48a7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f48a80c67c0 0x7f48a80cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:28.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48a7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f48a80c67c0 0x7f48a80cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47826/0 (socket says 172.21.15.98:47826) 2024-05-27T00:19:28.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48a7e00640 1 -- 172.21.15.98:0/3594609864 learned_addr learned my addr 172.21.15.98:0/3594609864 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:28.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.095+0000 7f48ad000640 1 --2- 172.21.15.98:0/3594609864 >> v2:172.21.15.204:3300/0 conn(0x7f48a80cc4f0 0x7f48a80c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:28.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.099+0000 7f48a7e00640 1 -- 172.21.15.98:0/3594609864 >> v2:172.21.15.204:3300/0 conn(0x7f48a80cc4f0 msgr2=0x7f48a80c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:28.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.099+0000 7f48a7e00640 1 --2- 172.21.15.98:0/3594609864 >> v2:172.21.15.204:3300/0 conn(0x7f48a80cc4f0 0x7f48a80c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:28.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.099+0000 7f48a7e00640 1 -- 172.21.15.98:0/3594609864 >> v1:172.21.15.204:6789/0 conn(0x7f48a80629d0 legacy=0x7f48a815bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:28.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.099+0000 7f48a7e00640 1 -- 172.21.15.98:0/3594609864 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a80c8930 con 0x7f48a80c67c0 2024-05-27T00:19:28.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48a7e00640 1 --2- 172.21.15.98:0/3594609864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80c67c0 0x7f48a80cc8d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f48a80c3930 tx=0x7f489401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6a6b74d7c1b4a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:28.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48a7400640 1 -- 172.21.15.98:0/3594609864 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48940b0070 con 0x7f48a80c67c0 2024-05-27T00:19:28.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48a7400640 1 -- 172.21.15.98:0/3594609864 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48940ab400 con 0x7f48a80c67c0 2024-05-27T00:19:28.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48a7400640 1 -- 172.21.15.98:0/3594609864 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48940ab7d0 con 0x7f48a80c67c0 2024-05-27T00:19:28.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48ae400640 1 -- 172.21.15.98:0/3594609864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80c67c0 msgr2=0x7f48a80cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:28.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48ae400640 1 --2- 172.21.15.98:0/3594609864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80c67c0 0x7f48a80cc8d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f48a80c3930 tx=0x7f489401c520 comp rx=0 tx=0).stop 2024-05-27T00:19:28.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48ae400640 1 -- 172.21.15.98:0/3594609864 shutdown_connections 2024-05-27T00:19:28.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48ae400640 1 --2- 172.21.15.98:0/3594609864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80c67c0 0x7f48a80cc8d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:28.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48ae400640 1 --2- 172.21.15.98:0/3594609864 >> v2:172.21.15.204:3300/0 conn(0x7f48a80cc4f0 0x7f48a80c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:28.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48ae400640 1 -- 172.21.15.98:0/3594609864 >> 172.21.15.98:0/3594609864 conn(0x7f48a806dd00 msgr2=0x7f48a806b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:28.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48ae400640 1 -- 172.21.15.98:0/3594609864 shutdown_connections 2024-05-27T00:19:28.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.103+0000 7f48ae400640 1 -- 172.21.15.98:0/3594609864 wait complete. 2024-05-27T00:19:28.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ae400640 1 Processor -- start 2024-05-27T00:19:28.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ae400640 1 -- start start 2024-05-27T00:19:28.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ae400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80629d0 0x7f48a815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:28.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ae400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80c67c0 0x7f48a8143190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:28.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ae400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48a80cc4f0 0x7f48a815a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:28.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ae400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48a8163a90 con 0x7f48a80629d0 2024-05-27T00:19:28.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ae400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f48a8163910 con 0x7f48a80cc4f0 2024-05-27T00:19:28.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ae400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f48a8163c10 con 0x7f48a80c67c0 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ad000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80629d0 0x7f48a815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ad000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80629d0 0x7f48a815b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47830/0 (socket says 172.21.15.98:47830) 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ad000640 1 -- 172.21.15.98:0/303881883 learned_addr learned my addr 172.21.15.98:0/303881883 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ada00640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48a80cc4f0 0x7f48a815a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48a7e00640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80c67c0 0x7f48a8143190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ad000640 1 -- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80c67c0 msgr2=0x7f48a8143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ad000640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80c67c0 0x7f48a8143190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ad000640 1 -- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48a80cc4f0 msgr2=0x7f48a815a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.107+0000 7f48ad000640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48a80cc4f0 0x7f48a815a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.111+0000 7f48ad000640 1 -- 172.21.15.98:0/303881883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a81436d0 con 0x7f48a80629d0 2024-05-27T00:19:28.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.111+0000 7f48ad000640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80629d0 0x7f48a815b970 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f489c012ad0 tx=0x7f489c012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:28.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.111+0000 7f48a4c00640 1 -- 172.21.15.98:0/303881883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c010a80 con 0x7f48a80629d0 2024-05-27T00:19:28.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.111+0000 7f48a4c00640 1 -- 172.21.15.98:0/303881883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f489c01d070 con 0x7f48a80629d0 2024-05-27T00:19:28.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.111+0000 7f48ae400640 1 -- 172.21.15.98:0/303881883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a81e9650 con 0x7f48a80629d0 2024-05-27T00:19:28.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.111+0000 7f48a4c00640 1 -- 172.21.15.98:0/303881883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c010e20 con 0x7f48a80629d0 2024-05-27T00:19:28.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.111+0000 7f48ae400640 1 -- 172.21.15.98:0/303881883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a81e98f0 con 0x7f48a80629d0 2024-05-27T00:19:28.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.111+0000 7f48ae400640 1 -- 172.21.15.98:0/303881883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a8075c40 con 0x7f48a80629d0 2024-05-27T00:19:28.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.115+0000 7f48a4c00640 1 -- 172.21.15.98:0/303881883 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f489c016050 con 0x7f48a80629d0 2024-05-27T00:19:28.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.115+0000 7f48a4c00640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4888067910 0x7f4888069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:28.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.115+0000 7f48a4c00640 1 -- 172.21.15.98:0/303881883 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f489c089a10 con 0x7f48a80629d0 2024-05-27T00:19:28.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.115+0000 7f48a7e00640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4888067910 0x7f4888069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:28.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.115+0000 7f48a7e00640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4888067910 0x7f4888069dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f48940ad000 tx=0x7f4894008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:28.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.123+0000 7f48a4c00640 1 -- 172.21.15.98:0/303881883 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f489c026020 con 0x7f48a80629d0 2024-05-27T00:19:28.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.519+0000 7f48ae400640 1 -- 172.21.15.98:0/303881883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48a80c7500 con 0x7f48a80629d0 2024-05-27T00:19:28.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.519+0000 7f48a4c00640 1 -- 172.21.15.98:0/303881883 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6839 (secure 0 0 0) 0x7f489c014040 con 0x7f48a80629d0 2024-05-27T00:19:28.523 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:28.523 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5073,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:44.479020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4894,"qdb_cluster":[4894]},"id":1}]} 2024-05-27T00:19:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 18 2024-05-27T00:19:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.519+0000 7f48ae400640 1 -- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4888067910 msgr2=0x7f4888069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.519+0000 7f48ae400640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4888067910 0x7f4888069dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f48940ad000 tx=0x7f4894008040 comp rx=0 tx=0).stop 2024-05-27T00:19:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.519+0000 7f48ae400640 1 -- 172.21.15.98:0/303881883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80629d0 msgr2=0x7f48a815b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.519+0000 7f48ae400640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80629d0 0x7f48a815b970 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f489c012ad0 tx=0x7f489c012f90 comp rx=0 tx=0).stop 2024-05-27T00:19:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.523+0000 7f48ae400640 1 -- 172.21.15.98:0/303881883 shutdown_connections 2024-05-27T00:19:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.523+0000 7f48ae400640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4888067910 0x7f4888069dd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.523+0000 7f48ae400640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48a80cc4f0 0x7f48a815a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.523+0000 7f48ae400640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80c67c0 0x7f48a8143190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.523+0000 7f48ae400640 1 --2- 172.21.15.98:0/303881883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48a80629d0 0x7f48a815b970 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.523+0000 7f48ae400640 1 -- 172.21.15.98:0/303881883 >> 172.21.15.98:0/303881883 conn(0x7f48a806dd00 msgr2=0x7f48a80cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.523+0000 7f48ae400640 1 -- 172.21.15.98:0/303881883 shutdown_connections 2024-05-27T00:19:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:28.523+0000 7f48ae400640 1 -- 172.21.15.98:0/303881883 wait complete. 2024-05-27T00:19:28.556 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:19:30.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:30.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.699+0000 7fb933c00640 1 Processor -- start 2024-05-27T00:19:30.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.699+0000 7fb933c00640 1 -- start start 2024-05-27T00:19:30.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb933c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb92c0731f0 0x7fb92c0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:30.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb933c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb92c0c79f0 0x7fb92c157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:30.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb933c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb92c0c17d0 con 0x7fb92c0735d0 2024-05-27T00:19:30.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb933c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb92c071990 con 0x7fb92c0c79f0 2024-05-27T00:19:30.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb933c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb92c070e10 con 0x7fb92c0731f0 2024-05-27T00:19:30.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb931c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb92c0c79f0 0x7fb92c157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:30.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb931200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb92c0731f0 0x7fb92c0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:30.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb931c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb92c0c79f0 0x7fb92c157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:53112/0 (socket says 172.21.15.98:53112) 2024-05-27T00:19:30.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb931c00640 1 -- 172.21.15.98:0/1500785512 learned_addr learned my addr 172.21.15.98:0/1500785512 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:30.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb931c00640 1 -- 172.21.15.98:0/1500785512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb92c0731f0 msgr2=0x7fb92c0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:30.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb931c00640 1 --2- 172.21.15.98:0/1500785512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb92c0731f0 0x7fb92c0c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:30.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb931c00640 1 -- 172.21.15.98:0/1500785512 >> v1:172.21.15.204:6789/0 conn(0x7fb92c0735d0 legacy=0x7fb92c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:30.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.703+0000 7fb931c00640 1 -- 172.21.15.98:0/1500785512 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c0710f0 con 0x7fb92c0c79f0 2024-05-27T00:19:30.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb931c00640 1 --2- 172.21.15.98:0/1500785512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c0c79f0 0x7fb92c157970 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb924009e50 tx=0x7fb92401f210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21cadd4813e074c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:30.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb92be00640 1 -- 172.21.15.98:0/1500785512 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240b4070 con 0x7fb92c0c79f0 2024-05-27T00:19:30.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb92be00640 1 -- 172.21.15.98:0/1500785512 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9240af400 con 0x7fb92c0c79f0 2024-05-27T00:19:30.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb92be00640 1 -- 172.21.15.98:0/1500785512 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9240af7d0 con 0x7fb92c0c79f0 2024-05-27T00:19:30.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 -- 172.21.15.98:0/1500785512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c0c79f0 msgr2=0x7fb92c157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:30.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 --2- 172.21.15.98:0/1500785512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c0c79f0 0x7fb92c157970 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb924009e50 tx=0x7fb92401f210 comp rx=0 tx=0).stop 2024-05-27T00:19:30.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 -- 172.21.15.98:0/1500785512 shutdown_connections 2024-05-27T00:19:30.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 --2- 172.21.15.98:0/1500785512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c0c79f0 0x7fb92c157970 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:30.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 --2- 172.21.15.98:0/1500785512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb92c0731f0 0x7fb92c0c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:30.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 -- 172.21.15.98:0/1500785512 >> 172.21.15.98:0/1500785512 conn(0x7fb92c067530 msgr2=0x7fb92c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:30.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 -- 172.21.15.98:0/1500785512 shutdown_connections 2024-05-27T00:19:30.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 -- 172.21.15.98:0/1500785512 wait complete. 2024-05-27T00:19:30.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 Processor -- start 2024-05-27T00:19:30.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 -- start start 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c0731f0 0x7fb92c14cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c0735d0 0x7fb92c1f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb92c0c79f0 0x7fb92c1f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c0c8400 con 0x7fb92c0731f0 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c0716b0 con 0x7fb92c0735d0 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.707+0000 7fb933c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb92c0c13f0 con 0x7fb92c0c79f0 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb931200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c0731f0 0x7fb92c14cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb931200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c0731f0 0x7fb92c14cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47832/0 (socket says 172.21.15.98:47832) 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb931200640 1 -- 172.21.15.98:0/2209200958 learned_addr learned my addr 172.21.15.98:0/2209200958 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb923e00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c0735d0 0x7fb92c1f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:30.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb931c00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb92c0c79f0 0x7fb92c1f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb931200640 1 -- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb92c0c79f0 msgr2=0x7fb92c1f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb931200640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb92c0c79f0 0x7fb92c1f0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb931200640 1 -- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c0735d0 msgr2=0x7fb92c1f0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb931200640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c0735d0 0x7fb92c1f0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb931200640 1 -- 172.21.15.98:0/2209200958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c14d830 con 0x7fb92c0731f0 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb923e00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c0735d0 0x7fb92c1f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb931200640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c0731f0 0x7fb92c14cbf0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00e8a0 tx=0x7fb91c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb929600640 1 -- 172.21.15.98:0/2209200958 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c017070 con 0x7fb92c0731f0 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb929600640 1 -- 172.21.15.98:0/2209200958 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb91c00c940 con 0x7fb92c0731f0 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb929600640 1 -- 172.21.15.98:0/2209200958 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c00ccf0 con 0x7fb92c0731f0 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb933c00640 1 -- 172.21.15.98:0/2209200958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92c15b890 con 0x7fb92c0731f0 2024-05-27T00:19:30.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb933c00640 1 -- 172.21.15.98:0/2209200958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb92c1efcb0 con 0x7fb92c0731f0 2024-05-27T00:19:30.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb933c00640 1 -- 172.21.15.98:0/2209200958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb92c075cb0 con 0x7fb92c0731f0 2024-05-27T00:19:30.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.711+0000 7fb929600640 1 -- 172.21.15.98:0/2209200958 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb91c01a070 con 0x7fb92c0731f0 2024-05-27T00:19:30.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.715+0000 7fb929600640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9080678f0 0x7fb908069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:30.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.715+0000 7fb929600640 1 -- 172.21.15.98:0/2209200958 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7fb91c085ce0 con 0x7fb92c0731f0 2024-05-27T00:19:30.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.715+0000 7fb923e00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9080678f0 0x7fb908069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:30.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.715+0000 7fb923e00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9080678f0 0x7fb908069db0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb92c14f860 tx=0x7fb914009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:30.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.719+0000 7fb929600640 1 -- 172.21.15.98:0/2209200958 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb91c056660 con 0x7fb92c0731f0 2024-05-27T00:19:30.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 -- 172.21.15.98:0/2209200958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb92c158510 con 0x7fb92c0731f0 2024-05-27T00:19:30.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb929600640 1 -- 172.21.15.98:0/2209200958 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6839 (secure 0 0 0) 0x7fb91c055f60 con 0x7fb92c0731f0 2024-05-27T00:19:30.911 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:30.911 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5073,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:44.479020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4894,"qdb_cluster":[4894]},"id":1}]} 2024-05-27T00:19:30.911 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 18 2024-05-27T00:19:30.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 -- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9080678f0 msgr2=0x7fb908069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:30.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9080678f0 0x7fb908069db0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb92c14f860 tx=0x7fb914009040 comp rx=0 tx=0).stop 2024-05-27T00:19:30.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 -- 172.21.15.98:0/2209200958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c0731f0 msgr2=0x7fb92c14cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:30.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c0731f0 0x7fb92c14cbf0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00e8a0 tx=0x7fb91c00eca0 comp rx=0 tx=0).stop 2024-05-27T00:19:30.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 -- 172.21.15.98:0/2209200958 shutdown_connections 2024-05-27T00:19:30.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb92c0c79f0 0x7fb92c1f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:30.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb9080678f0 0x7fb908069db0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:30.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c0735d0 0x7fb92c1f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:30.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 --2- 172.21.15.98:0/2209200958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb92c0731f0 0x7fb92c14cbf0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:30.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 -- 172.21.15.98:0/2209200958 >> 172.21.15.98:0/2209200958 conn(0x7fb92c067530 msgr2=0x7fb92c14feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:30.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 -- 172.21.15.98:0/2209200958 shutdown_connections 2024-05-27T00:19:30.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:30.907+0000 7fb933c00640 1 -- 172.21.15.98:0/2209200958 wait complete. 2024-05-27T00:19:30.931 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:19:32.932 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:33.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.075+0000 7f59f1e00640 1 Processor -- start 2024-05-27T00:19:33.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.075+0000 7f59f1e00640 1 -- start start 2024-05-27T00:19:33.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59ec157cd0 0x7f59ec0c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:33.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f59ec0b3cb0 0x7f59ec0c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:33.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ec0c59d0 con 0x7f59ec1580b0 2024-05-27T00:19:33.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f59ec0c51e0 con 0x7f59ec0b3cb0 2024-05-27T00:19:33.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f59ec0cabb0 con 0x7f59ec157cd0 2024-05-27T00:19:33.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59e3e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f59ec1580b0 0x7f59ec15e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46412/0 (socket says 172.21.15.98:46412) 2024-05-27T00:19:33.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59e3e00640 1 -- 172.21.15.98:0/2964595285 learned_addr learned my addr 172.21.15.98:0/2964595285 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:19:33.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1400640 1 --2- 172.21.15.98:0/2964595285 >> v2:172.21.15.204:3300/0 conn(0x7f59ec0b3cb0 0x7f59ec0c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:33.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f0a00640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59ec157cd0 0x7f59ec0c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101761563 0 0) 0x7f59ec0c59d0 con 0x7f59ec1580b0 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d0003530 con 0x7f59ec1580b0 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 99433887 0 0) 0x7f59d0003530 con 0x7f59ec1580b0 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59ec157cd0 msgr2=0x7f59ec0c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59ec157cd0 0x7f59ec0c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 >> v2:172.21.15.204:3300/0 conn(0x7f59ec0b3cb0 msgr2=0x7f59ec0c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 --2- 172.21.15.98:0/2964595285 >> v2:172.21.15.204:3300/0 conn(0x7f59ec0b3cb0 0x7f59ec0c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ec0cae50 con 0x7f59ec1580b0 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f59d4001680 con 0x7f59ec1580b0 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 >> v1:172.21.15.98:6789/0 conn(0x7f59ec1580b0 legacy=0x7f59ec15e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59d00039c0 0x7f59d0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59ec157cd0 0x7f59d0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d0004d50 0x7f59d00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:33.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec0c51e0 con 0x7f59d0004d50 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec0cabb0 con 0x7f59d00039c0 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f59d000b9d0 con 0x7f59ec157cd0 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1400640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d0004d50 0x7f59d00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59e3e00640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59ec157cd0 0x7f59d0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f0a00640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59d00039c0 0x7f59d0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1400640 1 -- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59ec157cd0 msgr2=0x7f59d0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1400640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59ec157cd0 0x7f59d0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1400640 1 -- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59d00039c0 msgr2=0x7f59d0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1400640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59d00039c0 0x7f59d0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f1400640 1 -- 172.21.15.98:0/2964595285 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d0003530 con 0x7f59d0004d50 2024-05-27T00:19:33.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.079+0000 7f59f0a00640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59d00039c0 0x7f59d0003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:19:33.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59f1400640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d0004d50 0x7f59d00051d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009fc0 tx=0x7f59e401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce7ed55c981726a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:33.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e40b2070 con 0x7f59d0004d50 2024-05-27T00:19:33.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e40a5cd0 con 0x7f59d0004d50 2024-05-27T00:19:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59ebe00640 1 -- 172.21.15.98:0/2964595285 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4009c10 con 0x7f59d0004d50 2024-05-27T00:19:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59f1e00640 1 -- 172.21.15.98:0/2964595285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d0004d50 msgr2=0x7f59d00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59f1e00640 1 --2- 172.21.15.98:0/2964595285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d0004d50 0x7f59d00051d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f59e4009fc0 tx=0x7f59e401ccc0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59f1400640 1 -- 172.21.15.98:0/2964595285 reap_dead start 2024-05-27T00:19:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59f1e00640 1 -- 172.21.15.98:0/2964595285 shutdown_connections 2024-05-27T00:19:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59f1e00640 1 -- 172.21.15.98:0/2964595285 >> 172.21.15.98:0/2964595285 conn(0x7f59ec070b30 msgr2=0x7f59ec070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59f1e00640 1 -- 172.21.15.98:0/2964595285 shutdown_connections 2024-05-27T00:19:33.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59f1e00640 1 -- 172.21.15.98:0/2964595285 wait complete. 2024-05-27T00:19:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59f1e00640 1 Processor -- start 2024-05-27T00:19:33.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.083+0000 7f59f1e00640 1 -- start start 2024-05-27T00:19:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59d0004d50 0x7f59ec15a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59ec148ca0 0x7f59ec149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59ec149aa0 0x7f59ec0c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec15e120 con 0x7f59ec149aa0 2024-05-27T00:19:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec15de40 con 0x7f59ec148ca0 2024-05-27T00:19:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f59ec0cb410 con 0x7f59d0004d50 2024-05-27T00:19:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59ec149aa0 0x7f59ec0c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59ec149aa0 0x7f59ec0c93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47848/0 (socket says 172.21.15.98:47848) 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1400640 1 -- 172.21.15.98:0/373990432 learned_addr learned my addr 172.21.15.98:0/373990432 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f0a00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59d0004d50 0x7f59ec15a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59e3e00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59ec148ca0 0x7f59ec149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1400640 1 -- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59d0004d50 msgr2=0x7f59ec15a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1400640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59d0004d50 0x7f59ec15a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1400640 1 -- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59ec148ca0 msgr2=0x7f59ec149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1400640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59ec148ca0 0x7f59ec149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1400640 1 -- 172.21.15.98:0/373990432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ec149f20 con 0x7f59ec149aa0 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59e3e00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59ec148ca0 0x7f59ec149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1400640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59ec149aa0 0x7f59ec0c93e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f59e401c730 tx=0x7f59e4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59e9600640 1 -- 172.21.15.98:0/373990432 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e40bd070 con 0x7f59ec149aa0 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59e9600640 1 -- 172.21.15.98:0/373990432 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e40a5ad0 con 0x7f59ec149aa0 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59e9600640 1 -- 172.21.15.98:0/373990432 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e40b2040 con 0x7f59ec149aa0 2024-05-27T00:19:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1e00640 1 -- 172.21.15.98:0/373990432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59ec0c8320 con 0x7f59ec149aa0 2024-05-27T00:19:33.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1e00640 1 -- 172.21.15.98:0/373990432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59ec0c88e0 con 0x7f59ec149aa0 2024-05-27T00:19:33.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59f1e00640 1 -- 172.21.15.98:0/373990432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59ec075c40 con 0x7f59ec149aa0 2024-05-27T00:19:33.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.087+0000 7f59e9600640 1 -- 172.21.15.98:0/373990432 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f59e40039a0 con 0x7f59ec149aa0 2024-05-27T00:19:33.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.091+0000 7f59e9600640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59c80678f0 0x7f59c8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:33.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.091+0000 7f59e9600640 1 -- 172.21.15.98:0/373990432 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5997+0+0 (secure 0 0 0) 0x7f59e411fc00 con 0x7f59ec149aa0 2024-05-27T00:19:33.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.091+0000 7f59f0a00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59c80678f0 0x7f59c8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:33.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.091+0000 7f59f0a00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59c80678f0 0x7f59c8069db0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009b70 tx=0x7f59dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:33.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.095+0000 7f59e9600640 1 -- 172.21.15.98:0/373990432 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f59ec149f20 con 0x7f59ec149aa0 2024-05-27T00:19:33.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.287+0000 7f59f1e00640 1 -- 172.21.15.98:0/373990432 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59ec0ca570 con 0x7f59ec149aa0 2024-05-27T00:19:33.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.287+0000 7f59e9600640 1 -- 172.21.15.98:0/373990432 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6839 (secure 0 0 0) 0x7f59e40169d0 con 0x7f59ec149aa0 2024-05-27T00:19:33.292 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:33.292 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5073,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:18:44.479020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4894},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4894":{"gid":4894,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1781395818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1781395818},{"type":"v1","addr":"172.21.15.98:6837","nonce":1781395818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4894,"qdb_cluster":[4894]},"id":1}]} 2024-05-27T00:19:33.292 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 18 2024-05-27T00:19:33.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 -- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59c80678f0 msgr2=0x7f59c8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:33.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59c80678f0 0x7f59c8069db0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009b70 tx=0x7f59dc008040 comp rx=0 tx=0).stop 2024-05-27T00:19:33.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 -- 172.21.15.98:0/373990432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59ec149aa0 msgr2=0x7f59ec0c93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:33.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59ec149aa0 0x7f59ec0c93e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f59e401c730 tx=0x7f59e4009750 comp rx=0 tx=0).stop 2024-05-27T00:19:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 -- 172.21.15.98:0/373990432 shutdown_connections 2024-05-27T00:19:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59ec149aa0 0x7f59ec0c93e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59c80678f0 0x7f59c8069db0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59ec148ca0 0x7f59ec149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 --2- 172.21.15.98:0/373990432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59d0004d50 0x7f59ec15a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 -- 172.21.15.98:0/373990432 >> 172.21.15.98:0/373990432 conn(0x7f59ec070b30 msgr2=0x7f59ec0c7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 -- 172.21.15.98:0/373990432 shutdown_connections 2024-05-27T00:19:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:33.291+0000 7f59f1e00640 1 -- 172.21.15.98:0/373990432 wait complete. 2024-05-27T00:19:33.313 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:19:35.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:35.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd42800640 1 Processor -- start 2024-05-27T00:19:35.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd42800640 1 -- start start 2024-05-27T00:19:35.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd42800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c073630 0x7efd3c071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd42800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd3c0c32f0 con 0x7efd3c151b60 2024-05-27T00:19:35.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd42800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd3c0c35d0 con 0x7efd3c0c85a0 2024-05-27T00:19:35.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd42800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efd3c0c38b0 con 0x7efd3c073630 2024-05-27T00:19:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3b400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efd3c151b60 0x7efd3c0c7e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59402/0 (socket says 172.21.15.98:59402) 2024-05-27T00:19:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3b400640 1 -- 172.21.15.98:0/4221389637 learned_addr learned my addr 172.21.15.98:0/4221389637 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:19:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3be00640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c073630 0x7efd3c071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3215844044 0 0) 0x7efd3c0c32f0 con 0x7efd3c151b60 2024-05-27T00:19:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd20005740 con 0x7efd3c151b60 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3803161060 0 0) 0x7efd20005740 con 0x7efd3c151b60 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c073630 msgr2=0x7efd3c071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c073630 0x7efd3c071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 >> v1:172.21.15.204:6789/0 conn(0x7efd3c0c85a0 legacy=0x7efd3c157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3c0c3b90 con 0x7efd3c151b60 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7efd2c001a00 con 0x7efd3c151b60 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 >> v1:172.21.15.98:6789/0 conn(0x7efd3c151b60 legacy=0x7efd3c0c7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c073630 0x7efd20005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd20006180 0x7efd20006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd20006f60 0x7efd200073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c0c38b0 con 0x7efd20006180 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd2000dcc0 con 0x7efd20006f60 2024-05-27T00:19:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efd2000dec0 con 0x7efd3c073630 2024-05-27T00:19:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3aa00640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd20006180 0x7efd20006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3b400640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c073630 0x7efd20005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3be00640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd20006f60 0x7efd200073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3aa00640 1 -- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c073630 msgr2=0x7efd20005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3aa00640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c073630 0x7efd20005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3aa00640 1 -- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd20006f60 msgr2=0x7efd200073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3aa00640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd20006f60 0x7efd200073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.459+0000 7efd3aa00640 1 -- 172.21.15.98:0/4221389637 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd20005740 con 0x7efd20006180 2024-05-27T00:19:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd3aa00640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd20006180 0x7efd20006a20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efd28001780 tx=0x7efd2801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a87ec39192dcd1e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd280013a0 con 0x7efd20006180 2024-05-27T00:19:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd280052f0 con 0x7efd20006180 2024-05-27T00:19:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd3a000640 1 -- 172.21.15.98:0/4221389637 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd28003840 con 0x7efd20006180 2024-05-27T00:19:35.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd42800640 1 -- 172.21.15.98:0/4221389637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd20006180 msgr2=0x7efd20006a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd42800640 1 --2- 172.21.15.98:0/4221389637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd20006180 0x7efd20006a20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efd28001780 tx=0x7efd2801c420 comp rx=0 tx=0).stop 2024-05-27T00:19:35.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd3be00640 1 -- 172.21.15.98:0/4221389637 reap_dead start 2024-05-27T00:19:35.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd42800640 1 -- 172.21.15.98:0/4221389637 shutdown_connections 2024-05-27T00:19:35.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd42800640 1 -- 172.21.15.98:0/4221389637 >> 172.21.15.98:0/4221389637 conn(0x7efd3c0c9af0 msgr2=0x7efd3c0c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:35.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd42800640 1 -- 172.21.15.98:0/4221389637 shutdown_connections 2024-05-27T00:19:35.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.463+0000 7efd42800640 1 -- 172.21.15.98:0/4221389637 wait complete. 2024-05-27T00:19:35.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd42800640 1 Processor -- start 2024-05-27T00:19:35.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd42800640 1 -- start start 2024-05-27T00:19:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd42800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd20006180 0x7efd3c1e1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd42800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c14c300 0x7efd3c1f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd42800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c14c850 0x7efd3c1f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd42800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c072080 con 0x7efd3c14c850 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd42800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c071f00 con 0x7efd20006180 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd42800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efd3c072200 con 0x7efd3c14c300 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c14c850 0x7efd3c1f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c14c850 0x7efd3c1f40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57778/0 (socket says 172.21.15.98:57778) 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3be00640 1 -- 172.21.15.98:0/1522341483 learned_addr learned my addr 172.21.15.98:0/1522341483 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3b400640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd20006180 0x7efd3c1e1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3aa00640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c14c300 0x7efd3c1f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3be00640 1 -- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c14c300 msgr2=0x7efd3c1f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3be00640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c14c300 0x7efd3c1f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3be00640 1 -- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd20006180 msgr2=0x7efd3c1e1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3be00640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd20006180 0x7efd3c1e1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3be00640 1 -- 172.21.15.98:0/1522341483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3c14d1c0 con 0x7efd3c14c850 2024-05-27T00:19:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3aa00640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c14c300 0x7efd3c1f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:19:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd3be00640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c14c850 0x7efd3c1f40d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efd3c073b80 tx=0x7efd34011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd33400640 1 -- 172.21.15.98:0/1522341483 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd34019020 con 0x7efd3c14c850 2024-05-27T00:19:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd33400640 1 -- 172.21.15.98:0/1522341483 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd34004240 con 0x7efd3c14c850 2024-05-27T00:19:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd33400640 1 -- 172.21.15.98:0/1522341483 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3400f7a0 con 0x7efd3c14c850 2024-05-27T00:19:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd42800640 1 -- 172.21.15.98:0/1522341483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3c154180 con 0x7efd3c14c850 2024-05-27T00:19:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.467+0000 7efd42800640 1 -- 172.21.15.98:0/1522341483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd3c154740 con 0x7efd3c14c850 2024-05-27T00:19:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.471+0000 7efd42800640 1 -- 172.21.15.98:0/1522341483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd3c075cb0 con 0x7efd3c14c850 2024-05-27T00:19:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.471+0000 7efd33400640 1 -- 172.21.15.98:0/1522341483 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efd3401c020 con 0x7efd3c14c850 2024-05-27T00:19:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.471+0000 7efd33400640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd1c0678f0 0x7efd1c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.471+0000 7efd33400640 1 -- 172.21.15.98:0/1522341483 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7efd340860b0 con 0x7efd3c14c850 2024-05-27T00:19:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.471+0000 7efd3b400640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd1c0678f0 0x7efd1c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.471+0000 7efd3b400640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd1c0678f0 0x7efd1c069db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efd2c002060 tx=0x7efd2c00a410 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:35.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.475+0000 7efd33400640 1 -- 172.21.15.98:0/1522341483 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7efd34023020 con 0x7efd3c14c850 2024-05-27T00:19:35.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.663+0000 7efd42800640 1 -- 172.21.15.98:0/1522341483 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd3c1582b0 con 0x7efd3c14c850 2024-05-27T00:19:35.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.663+0000 7efd33400640 1 -- 172.21.15.98:0/1522341483 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6006 (secure 0 0 0) 0x7efd34056a60 con 0x7efd3c14c850 2024-05-27T00:19:35.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:35.669 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:35.190051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:19:35.669 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 19 2024-05-27T00:19:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 -- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd1c0678f0 msgr2=0x7efd1c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd1c0678f0 0x7efd1c069db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efd2c002060 tx=0x7efd2c00a410 comp rx=0 tx=0).stop 2024-05-27T00:19:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 -- 172.21.15.98:0/1522341483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c14c850 msgr2=0x7efd3c1f40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c14c850 0x7efd3c1f40d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efd3c073b80 tx=0x7efd34011cf0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 -- 172.21.15.98:0/1522341483 shutdown_connections 2024-05-27T00:19:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c14c850 0x7efd3c1f40d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c14c300 0x7efd3c1f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd1c0678f0 0x7efd1c069db0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 --2- 172.21.15.98:0/1522341483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd20006180 0x7efd3c1e1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 -- 172.21.15.98:0/1522341483 >> 172.21.15.98:0/1522341483 conn(0x7efd3c0c9af0 msgr2=0x7efd3c14fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 -- 172.21.15.98:0/1522341483 shutdown_connections 2024-05-27T00:19:35.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.667+0000 7efd42800640 1 -- 172.21.15.98:0/1522341483 wait complete. 2024-05-27T00:19:35.691 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-05-27T00:19:35.691 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:19:35.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87e00640 1 Processor -- start 2024-05-27T00:19:35.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87e00640 1 -- start start 2024-05-27T00:19:35.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4a88157d80 0x7f4a880c3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4a880c5490 0x7f4a880c3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a8806e7e0 con 0x7f4a88070110 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8815b3e0 con 0x7f4a880c5490 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8806ea80 con 0x7f4a88157d80 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4a880c5490 0x7f4a880c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4a880c5490 0x7f4a880c3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57790/0 (socket says 172.21.15.98:57790) 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87400640 1 -- 172.21.15.98:0/3554069666 learned_addr learned my addr 172.21.15.98:0/3554069666 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a86a00640 1 --2- 172.21.15.98:0/3554069666 >> v2:172.21.15.204:3300/0 conn(0x7f4a88157d80 0x7f4a880c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87400640 1 -- 172.21.15.98:0/3554069666 >> v2:172.21.15.204:3300/0 conn(0x7f4a88157d80 msgr2=0x7f4a880c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87400640 1 --2- 172.21.15.98:0/3554069666 >> v2:172.21.15.204:3300/0 conn(0x7f4a88157d80 0x7f4a880c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87400640 1 -- 172.21.15.98:0/3554069666 >> v1:172.21.15.98:6789/0 conn(0x7f4a88070110 legacy=0x7f4a880c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a87400640 1 -- 172.21.15.98:0/3554069666 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a880c67f0 con 0x7f4a880c5490 2024-05-27T00:19:35.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.823+0000 7f4a85600640 1 -- 172.21.15.98:0/3554069666 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 665610697 0 0) 0x7f4a8806e7e0 con 0x7f4a88070110 2024-05-27T00:19:35.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87400640 1 --2- 172.21.15.98:0/3554069666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a880c5490 0x7f4a880c3960 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4a880706b0 tx=0x7f4a8001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a66d87c02a044a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:35.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a85600640 1 -- 172.21.15.98:0/3554069666 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a800a57b0 con 0x7f4a880c5490 2024-05-27T00:19:35.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a85600640 1 -- 172.21.15.98:0/3554069666 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a800a5950 con 0x7f4a880c5490 2024-05-27T00:19:35.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a85600640 1 -- 172.21.15.98:0/3554069666 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a800a5d20 con 0x7f4a880c5490 2024-05-27T00:19:35.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87e00640 1 -- 172.21.15.98:0/3554069666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a880c5490 msgr2=0x7f4a880c3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87e00640 1 --2- 172.21.15.98:0/3554069666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a880c5490 0x7f4a880c3960 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4a880706b0 tx=0x7f4a8001c8d0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87e00640 1 -- 172.21.15.98:0/3554069666 shutdown_connections 2024-05-27T00:19:35.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87e00640 1 --2- 172.21.15.98:0/3554069666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a880c5490 0x7f4a880c3960 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87e00640 1 --2- 172.21.15.98:0/3554069666 >> v2:172.21.15.204:3300/0 conn(0x7f4a88157d80 0x7f4a880c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87e00640 1 -- 172.21.15.98:0/3554069666 >> 172.21.15.98:0/3554069666 conn(0x7f4a88070a10 msgr2=0x7f4a88070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:35.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87e00640 1 -- 172.21.15.98:0/3554069666 shutdown_connections 2024-05-27T00:19:35.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87e00640 1 -- 172.21.15.98:0/3554069666 wait complete. 2024-05-27T00:19:35.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87e00640 1 Processor -- start 2024-05-27T00:19:35.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.827+0000 7f4a87e00640 1 -- start start 2024-05-27T00:19:35.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a87e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a88070110 0x7f4a881594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a87e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a880c5490 0x7f4a88146140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a87e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a88157d80 0x7f4a88143960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a87e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a88167a00 con 0x7f4a88070110 2024-05-27T00:19:35.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a87e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4a88167880 con 0x7f4a88157d80 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a87e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4a88167b80 con 0x7f4a880c5490 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a88070110 0x7f4a881594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a88070110 0x7f4a881594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57796/0 (socket says 172.21.15.98:57796) 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86a00640 1 -- 172.21.15.98:0/2129957987 learned_addr learned my addr 172.21.15.98:0/2129957987 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a87400640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a88157d80 0x7f4a88143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86000640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a880c5490 0x7f4a88146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86a00640 1 -- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a880c5490 msgr2=0x7f4a88146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86a00640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a880c5490 0x7f4a88146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86a00640 1 -- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a88157d80 msgr2=0x7f4a88143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86a00640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a88157d80 0x7f4a88143960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86a00640 1 -- 172.21.15.98:0/2129957987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a88159c20 con 0x7f4a88070110 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86000640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a880c5490 0x7f4a88146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a86a00640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a88070110 0x7f4a881594a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800d800 tx=0x7f4a7800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:35.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a7ea00640 1 -- 172.21.15.98:0/2129957987 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7800b660 con 0x7f4a88070110 2024-05-27T00:19:35.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a7ea00640 1 -- 172.21.15.98:0/2129957987 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a78019070 con 0x7f4a88070110 2024-05-27T00:19:35.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a7ea00640 1 -- 172.21.15.98:0/2129957987 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7800ba00 con 0x7f4a88070110 2024-05-27T00:19:35.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a87e00640 1 -- 172.21.15.98:0/2129957987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a880cc3d0 con 0x7f4a88070110 2024-05-27T00:19:35.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a87e00640 1 -- 172.21.15.98:0/2129957987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a880cc990 con 0x7f4a88070110 2024-05-27T00:19:35.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a87e00640 1 -- 172.21.15.98:0/2129957987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a88075c40 con 0x7f4a88070110 2024-05-27T00:19:35.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.831+0000 7f4a7ea00640 1 -- 172.21.15.98:0/2129957987 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4a7800bba0 con 0x7f4a88070110 2024-05-27T00:19:35.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.835+0000 7f4a7ea00640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4a680678c0 0x7f4a68069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:35.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.835+0000 7f4a7ea00640 1 -- 172.21.15.98:0/2129957987 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7f4a78086380 con 0x7f4a88070110 2024-05-27T00:19:35.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.835+0000 7f4a86000640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4a680678c0 0x7f4a68069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:35.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.835+0000 7f4a86000640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4a680678c0 0x7f4a68069d80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4a881434c0 tx=0x7f4a74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:35.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:35.839+0000 7f4a7ea00640 1 -- 172.21.15.98:0/2129957987 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4a78056c10 con 0x7f4a88070110 2024-05-27T00:19:36.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.023+0000 7f4a87e00640 1 -- 172.21.15.98:0/2129957987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a880c4630 con 0x7f4a88070110 2024-05-27T00:19:36.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.023+0000 7f4a7ea00640 1 -- 172.21.15.98:0/2129957987 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6006 (secure 0 0 0) 0x7f4a78056510 con 0x7f4a88070110 2024-05-27T00:19:36.031 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:36.031 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:35.190051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:19:36.032 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 19 2024-05-27T00:19:36.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 -- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4a680678c0 msgr2=0x7f4a68069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:36.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4a680678c0 0x7f4a68069d80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4a881434c0 tx=0x7f4a74009040 comp rx=0 tx=0).stop 2024-05-27T00:19:36.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 -- 172.21.15.98:0/2129957987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a88070110 msgr2=0x7f4a881594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:36.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a88070110 0x7f4a881594a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800d800 tx=0x7f4a7800dcc0 comp rx=0 tx=0).stop 2024-05-27T00:19:36.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 -- 172.21.15.98:0/2129957987 shutdown_connections 2024-05-27T00:19:36.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4a680678c0 0x7f4a68069d80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:36.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a88157d80 0x7f4a88143960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:36.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a880c5490 0x7f4a88146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:36.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 --2- 172.21.15.98:0/2129957987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a88070110 0x7f4a881594a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:36.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 -- 172.21.15.98:0/2129957987 >> 172.21.15.98:0/2129957987 conn(0x7f4a88070a10 msgr2=0x7f4a88157050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:36.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 -- 172.21.15.98:0/2129957987 shutdown_connections 2024-05-27T00:19:36.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:36.027+0000 7f4a87e00640 1 -- 172.21.15.98:0/2129957987 wait complete. 2024-05-27T00:19:36.050 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 19, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5073": { "addr": "172.21.15.98:6834/3247908011", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 3247908011, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 3247908011, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5073, "incarnation": 19, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:19:35.190051+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5073 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:19:38.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:38.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e5a600640 1 Processor -- start 2024-05-27T00:19:38.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e5a600640 1 -- start start 2024-05-27T00:19:38.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e5a600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e540731f0 0x7f2e540c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:38.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e5a600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2e540735d0 0x7f2e54151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:38.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e5a600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e540c17d0 con 0x7f2e540c79f0 2024-05-27T00:19:38.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e5a600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2e54071990 con 0x7f2e540735d0 2024-05-27T00:19:38.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e5a600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2e54070e10 con 0x7f2e540731f0 2024-05-27T00:19:38.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e4aa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2e540735d0 0x7f2e54151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:38.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e4aa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2e540735d0 0x7f2e54151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57798/0 (socket says 172.21.15.98:57798) 2024-05-27T00:19:38.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e4aa00640 1 -- 172.21.15.98:0/4116010843 learned_addr learned my addr 172.21.15.98:0/4116010843 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:38.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e53e00640 1 --1- 172.21.15.98:0/4116010843 >> v1:172.21.15.98:6789/0 conn(0x7f2e540c79f0 0x7f2e54157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59408/0 (socket says 172.21.15.98:59408) 2024-05-27T00:19:38.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e53400640 1 --2- 172.21.15.98:0/4116010843 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e540731f0 0x7f2e540c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:38.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.195+0000 7f2e4aa00640 1 -- 172.21.15.98:0/4116010843 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e540731f0 msgr2=0x7f2e540c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:38.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e4aa00640 1 --2- 172.21.15.98:0/4116010843 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e540731f0 0x7f2e540c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:38.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e4aa00640 1 -- 172.21.15.98:0/4116010843 >> v1:172.21.15.98:6789/0 conn(0x7f2e540c79f0 legacy=0x7f2e54157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:38.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e4aa00640 1 -- 172.21.15.98:0/4116010843 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e540710f0 con 0x7f2e540735d0 2024-05-27T00:19:38.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e4aa00640 1 --2- 172.21.15.98:0/4116010843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540735d0 0x7f2e54151430 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009870 tx=0x7f2e3c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc288d0ca3ea418c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:38.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e52a00640 1 -- 172.21.15.98:0/4116010843 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c0b5070 con 0x7f2e540735d0 2024-05-27T00:19:38.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e52a00640 1 -- 172.21.15.98:0/4116010843 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e3c0a8a50 con 0x7f2e540735d0 2024-05-27T00:19:38.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e52a00640 1 -- 172.21.15.98:0/4116010843 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c0a8e20 con 0x7f2e540735d0 2024-05-27T00:19:38.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e5a600640 1 -- 172.21.15.98:0/4116010843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540735d0 msgr2=0x7f2e54151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:38.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e5a600640 1 --2- 172.21.15.98:0/4116010843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540735d0 0x7f2e54151430 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009870 tx=0x7f2e3c01ed50 comp rx=0 tx=0).stop 2024-05-27T00:19:38.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e5a600640 1 -- 172.21.15.98:0/4116010843 shutdown_connections 2024-05-27T00:19:38.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e5a600640 1 --2- 172.21.15.98:0/4116010843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540735d0 0x7f2e54151430 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:38.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e5a600640 1 --2- 172.21.15.98:0/4116010843 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e540731f0 0x7f2e540c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:38.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e5a600640 1 -- 172.21.15.98:0/4116010843 >> 172.21.15.98:0/4116010843 conn(0x7f2e54067530 msgr2=0x7f2e5406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:38.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e5a600640 1 -- 172.21.15.98:0/4116010843 shutdown_connections 2024-05-27T00:19:38.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.199+0000 7f2e5a600640 1 -- 172.21.15.98:0/4116010843 wait complete. 2024-05-27T00:19:38.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e5a600640 1 Processor -- start 2024-05-27T00:19:38.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e5a600640 1 -- start start 2024-05-27T00:19:38.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e5a600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e540731f0 0x7f2e540c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e5a600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e540735d0 0x7f2e541f0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e5a600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540c79f0 0x7f2e541442e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e5a600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e540c8400 con 0x7f2e540c79f0 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e5a600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2e540716b0 con 0x7f2e540731f0 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e5a600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2e540c13f0 con 0x7f2e540735d0 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540c79f0 0x7f2e541442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540c79f0 0x7f2e541442e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57804/0 (socket says 172.21.15.98:57804) 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53e00640 1 -- 172.21.15.98:0/2503282412 learned_addr learned my addr 172.21.15.98:0/2503282412 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e4aa00640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e540735d0 0x7f2e541f0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53400640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e540731f0 0x7f2e540c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53e00640 1 -- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e540735d0 msgr2=0x7f2e541f0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53e00640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e540735d0 0x7f2e541f0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53e00640 1 -- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e540731f0 msgr2=0x7f2e540c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:38.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53e00640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e540731f0 0x7f2e540c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53e00640 1 -- 172.21.15.98:0/2503282412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e541f0370 con 0x7f2e540c79f0 2024-05-27T00:19:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53400640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e540731f0 0x7f2e540c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:19:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e53e00640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540c79f0 0x7f2e541442e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00c3c0 tx=0x7f2e4c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:38.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e4be00640 1 -- 172.21.15.98:0/2503282412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c0074e0 con 0x7f2e540c79f0 2024-05-27T00:19:38.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e4be00640 1 -- 172.21.15.98:0/2503282412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4c007680 con 0x7f2e540c79f0 2024-05-27T00:19:38.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e4be00640 1 -- 172.21.15.98:0/2503282412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c004a10 con 0x7f2e540c79f0 2024-05-27T00:19:38.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e5a600640 1 -- 172.21.15.98:0/2503282412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5414eba0 con 0x7f2e540c79f0 2024-05-27T00:19:38.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.203+0000 7f2e5a600640 1 -- 172.21.15.98:0/2503282412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e541f4140 con 0x7f2e540c79f0 2024-05-27T00:19:38.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.207+0000 7f2e5a600640 1 -- 172.21.15.98:0/2503282412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e54075cb0 con 0x7f2e540c79f0 2024-05-27T00:19:38.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.207+0000 7f2e4be00640 1 -- 172.21.15.98:0/2503282412 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e4c01a020 con 0x7f2e540c79f0 2024-05-27T00:19:38.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.207+0000 7f2e4be00640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2e300678c0 0x7f2e30069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:38.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.207+0000 7f2e4be00640 1 -- 172.21.15.98:0/2503282412 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7f2e4c085610 con 0x7f2e540c79f0 2024-05-27T00:19:38.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.207+0000 7f2e53400640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2e300678c0 0x7f2e30069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:38.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.207+0000 7f2e53400640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2e300678c0 0x7f2e30069d80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2e44007330 tx=0x7f2e4400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:38.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.211+0000 7f2e4be00640 1 -- 172.21.15.98:0/2503282412 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f2e541f0370 con 0x7f2e540c79f0 2024-05-27T00:19:38.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 -- 172.21.15.98:0/2503282412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e54158510 con 0x7f2e540c79f0 2024-05-27T00:19:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e4be00640 1 -- 172.21.15.98:0/2503282412 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6006 (secure 0 0 0) 0x7f2e4c055f30 con 0x7f2e540c79f0 2024-05-27T00:19:38.407 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:38.407 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:35.190051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:19:38.407 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 19 2024-05-27T00:19:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 -- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2e300678c0 msgr2=0x7f2e30069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2e300678c0 0x7f2e30069d80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2e44007330 tx=0x7f2e4400b040 comp rx=0 tx=0).stop 2024-05-27T00:19:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 -- 172.21.15.98:0/2503282412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540c79f0 msgr2=0x7f2e541442e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540c79f0 0x7f2e541442e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00c3c0 tx=0x7f2e4c00c880 comp rx=0 tx=0).stop 2024-05-27T00:19:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 -- 172.21.15.98:0/2503282412 shutdown_connections 2024-05-27T00:19:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e540c79f0 0x7f2e541442e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e540735d0 0x7f2e541f0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2e300678c0 0x7f2e30069d80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:38.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 --2- 172.21.15.98:0/2503282412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e540731f0 0x7f2e540c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 -- 172.21.15.98:0/2503282412 >> 172.21.15.98:0/2503282412 conn(0x7f2e54067530 msgr2=0x7f2e54156240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 -- 172.21.15.98:0/2503282412 shutdown_connections 2024-05-27T00:19:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:38.403+0000 7f2e5a600640 1 -- 172.21.15.98:0/2503282412 wait complete. 2024-05-27T00:19:38.429 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:19:40.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:40.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.567+0000 7fcf1be00640 1 Processor -- start 2024-05-27T00:19:40.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.567+0000 7fcf1be00640 1 -- start start 2024-05-27T00:19:40.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.567+0000 7fcf1be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c0b3260 0x7fcf1c0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:40.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.567+0000 7fcf1be00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c071010 con 0x7fcf1c070660 2024-05-27T00:19:40.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.567+0000 7fcf1be00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c0c3890 con 0x7fcf1c070aa0 2024-05-27T00:19:40.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.567+0000 7fcf1be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcf1c0c88c0 con 0x7fcf1c0b3260 2024-05-27T00:19:40.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.567+0000 7fcf1a000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcf1c070660 0x7fcf1c0c3180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59418/0 (socket says 172.21.15.98:59418) 2024-05-27T00:19:40.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.567+0000 7fcf1a000640 1 -- 172.21.15.98:0/867751618 learned_addr learned my addr 172.21.15.98:0/867751618 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:19:40.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.571+0000 7fcf1aa00640 1 --2- 172.21.15.98:0/867751618 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c0b3260 0x7fcf1c0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:40.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.571+0000 7fcf19600640 1 -- 172.21.15.98:0/867751618 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1493015989 0 0) 0x7fcf08001120 con 0x7fcf1c070660 2024-05-27T00:19:40.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.571+0000 7fcf19600640 1 -- 172.21.15.98:0/867751618 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf00003530 con 0x7fcf1c070660 2024-05-27T00:19:40.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.571+0000 7fcf19600640 1 -- 172.21.15.98:0/867751618 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1144257175 0 0) 0x7fcf1c0c3890 con 0x7fcf1c070aa0 2024-05-27T00:19:40.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.571+0000 7fcf19600640 1 -- 172.21.15.98:0/867751618 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf08001120 con 0x7fcf1c070aa0 2024-05-27T00:19:40.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.571+0000 7fcf19600640 1 -- 172.21.15.98:0/867751618 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3984660670 0 0) 0x7fcf00003530 con 0x7fcf1c070660 2024-05-27T00:19:40.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.571+0000 7fcf1aa00640 1 -- 172.21.15.98:0/867751618 >> v1:172.21.15.204:6789/0 conn(0x7fcf1c070aa0 legacy=0x7fcf1c0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:40.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.571+0000 7fcf1aa00640 1 -- 172.21.15.98:0/867751618 >> v1:172.21.15.98:6789/0 conn(0x7fcf1c070660 legacy=0x7fcf1c0c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:40.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.571+0000 7fcf1aa00640 1 -- 172.21.15.98:0/867751618 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c0c8b80 con 0x7fcf1c0b3260 2024-05-27T00:19:40.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf1aa00640 1 --2- 172.21.15.98:0/867751618 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c0b3260 0x7fcf1c0c7cc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0097f0 tx=0x7fcf0c01ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a6cc7dc4db8fe97 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:40.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf19600640 1 -- 172.21.15.98:0/867751618 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0b5070 con 0x7fcf1c0b3260 2024-05-27T00:19:40.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf19600640 1 -- 172.21.15.98:0/867751618 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf0c0a8670 con 0x7fcf1c0b3260 2024-05-27T00:19:40.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf19600640 1 -- 172.21.15.98:0/867751618 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08001850 con 0x7fcf1c0b3260 2024-05-27T00:19:40.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf1be00640 1 -- 172.21.15.98:0/867751618 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c0b3260 msgr2=0x7fcf1c0c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:40.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf1be00640 1 --2- 172.21.15.98:0/867751618 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c0b3260 0x7fcf1c0c7cc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0097f0 tx=0x7fcf0c01ebf0 comp rx=0 tx=0).stop 2024-05-27T00:19:40.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf1be00640 1 -- 172.21.15.98:0/867751618 shutdown_connections 2024-05-27T00:19:40.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf1be00640 1 --2- 172.21.15.98:0/867751618 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c0b3260 0x7fcf1c0c7cc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:40.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf1be00640 1 -- 172.21.15.98:0/867751618 >> 172.21.15.98:0/867751618 conn(0x7fcf1c0cdeb0 msgr2=0x7fcf1c0ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:40.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf1be00640 1 -- 172.21.15.98:0/867751618 shutdown_connections 2024-05-27T00:19:40.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.575+0000 7fcf1be00640 1 -- 172.21.15.98:0/867751618 wait complete. 2024-05-27T00:19:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1be00640 1 Processor -- start 2024-05-27T00:19:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1be00640 1 -- start start 2024-05-27T00:19:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf1c070660 0x7fcf1c0cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf1c070aa0 0x7fcf1c1e6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c1e4cd0 0x7fcf1c06b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1be00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c0c9420 con 0x7fcf1c070660 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c0c9140 con 0x7fcf1c070aa0 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcf1c0c8200 con 0x7fcf1c1e4cd0 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf1c070660 0x7fcf1c0cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf1c070660 0x7fcf1c0cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57808/0 (socket says 172.21.15.98:57808) 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1aa00640 1 -- 172.21.15.98:0/2466483914 learned_addr learned my addr 172.21.15.98:0/2466483914 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1a000640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf1c070aa0 0x7fcf1c1e6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1b400640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c1e4cd0 0x7fcf1c06b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1aa00640 1 -- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c1e4cd0 msgr2=0x7fcf1c06b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1aa00640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c1e4cd0 0x7fcf1c06b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1aa00640 1 -- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf1c070aa0 msgr2=0x7fcf1c1e6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1aa00640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf1c070aa0 0x7fcf1c1e6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.579+0000 7fcf1aa00640 1 -- 172.21.15.98:0/2466483914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c1e5e00 con 0x7fcf1c070660 2024-05-27T00:19:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.583+0000 7fcf1aa00640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf1c070660 0x7fcf1c0cd070 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0b3040 tx=0x7fcf0c01b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.583+0000 7fcf12a00640 1 -- 172.21.15.98:0/2466483914 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0bf070 con 0x7fcf1c070660 2024-05-27T00:19:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.583+0000 7fcf12a00640 1 -- 172.21.15.98:0/2466483914 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf0c0a8a20 con 0x7fcf1c070660 2024-05-27T00:19:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.583+0000 7fcf12a00640 1 -- 172.21.15.98:0/2466483914 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0b5040 con 0x7fcf1c070660 2024-05-27T00:19:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.583+0000 7fcf1be00640 1 -- 172.21.15.98:0/2466483914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1c1e60e0 con 0x7fcf1c070660 2024-05-27T00:19:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.583+0000 7fcf1be00640 1 -- 172.21.15.98:0/2466483914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf1c1e66a0 con 0x7fcf1c070660 2024-05-27T00:19:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.583+0000 7fcf07e00640 1 -- 172.21.15.98:0/2466483914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf1c075c40 con 0x7fcf1c070660 2024-05-27T00:19:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.583+0000 7fcf12a00640 1 -- 172.21.15.98:0/2466483914 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcf0c003620 con 0x7fcf1c070660 2024-05-27T00:19:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.587+0000 7fcf12a00640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcefc0678c0 0x7fcefc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.587+0000 7fcf12a00640 1 -- 172.21.15.98:0/2466483914 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7fcf0c122410 con 0x7fcf1c070660 2024-05-27T00:19:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.587+0000 7fcf1a000640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcefc0678c0 0x7fcefc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.587+0000 7fcf1a000640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcefc0678c0 0x7fcefc069d80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0cddd0 tx=0x7fcf08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:40.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.595+0000 7fcf12a00640 1 -- 172.21.15.98:0/2466483914 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fcf0c018750 con 0x7fcf1c070660 2024-05-27T00:19:40.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.819+0000 7fcf07e00640 1 -- 172.21.15.98:0/2466483914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf1c0c5480 con 0x7fcf1c070660 2024-05-27T00:19:40.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf12a00640 1 -- 172.21.15.98:0/2466483914 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6006 (secure 0 0 0) 0x7fcf0c0f2e60 con 0x7fcf1c070660 2024-05-27T00:19:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:40.826 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:35.190051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:19:40.826 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 19 2024-05-27T00:19:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 -- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcefc0678c0 msgr2=0x7fcefc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcefc0678c0 0x7fcefc069d80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0cddd0 tx=0x7fcf08009040 comp rx=0 tx=0).stop 2024-05-27T00:19:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 -- 172.21.15.98:0/2466483914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf1c070660 msgr2=0x7fcf1c0cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf1c070660 0x7fcf1c0cd070 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0b3040 tx=0x7fcf0c01b990 comp rx=0 tx=0).stop 2024-05-27T00:19:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 -- 172.21.15.98:0/2466483914 shutdown_connections 2024-05-27T00:19:40.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf1c1e4cd0 0x7fcf1c06b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcefc0678c0 0x7fcefc069d80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf1c070aa0 0x7fcf1c1e6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 --2- 172.21.15.98:0/2466483914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf1c070660 0x7fcf1c0cd070 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 -- 172.21.15.98:0/2466483914 >> 172.21.15.98:0/2466483914 conn(0x7fcf1c0cdeb0 msgr2=0x7fcf1c061f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 -- 172.21.15.98:0/2466483914 shutdown_connections 2024-05-27T00:19:40.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:40.823+0000 7fcf07e00640 1 -- 172.21.15.98:0/2466483914 wait complete. 2024-05-27T00:19:40.847 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:19:42.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:42.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.979+0000 7efd41400640 1 Processor -- start 2024-05-27T00:19:42.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.979+0000 7efd41400640 1 -- start start 2024-05-27T00:19:42.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd41400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efd3c0735d0 0x7efd3c151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:42.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd41400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7efd3c0c79f0 0x7efd3c151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:42.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd41400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd3c0c17d0 con 0x7efd3c0731f0 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd41400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7efd3c071990 con 0x7efd3c0735d0 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd41400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7efd3c070e10 con 0x7efd3c0c79f0 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd3a800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efd3c0731f0 0x7efd3c0c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59426/0 (socket says 172.21.15.98:59426) 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd3a800640 1 -- 172.21.15.98:0/2566878361 learned_addr learned my addr 172.21.15.98:0/2566878361 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd31e00640 1 --2- 172.21.15.98:0/2566878361 >> v2:172.21.15.98:3300/0 conn(0x7efd3c0735d0 0x7efd3c151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd3b200640 1 --2- 172.21.15.98:0/2566878361 >> v2:172.21.15.204:3300/0 conn(0x7efd3c0c79f0 0x7efd3c151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd31e00640 1 -- 172.21.15.98:0/2566878361 >> v2:172.21.15.204:3300/0 conn(0x7efd3c0c79f0 msgr2=0x7efd3c151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd31e00640 1 --2- 172.21.15.98:0/2566878361 >> v2:172.21.15.204:3300/0 conn(0x7efd3c0c79f0 0x7efd3c151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd31e00640 1 -- 172.21.15.98:0/2566878361 >> v1:172.21.15.98:6789/0 conn(0x7efd3c0731f0 legacy=0x7efd3c0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd31e00640 1 -- 172.21.15.98:0/2566878361 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3c0710f0 con 0x7efd3c0735d0 2024-05-27T00:19:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd39e00640 1 -- 172.21.15.98:0/2566878361 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2500937315 0 0) 0x7efd3c0c17d0 con 0x7efd3c0731f0 2024-05-27T00:19:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.983+0000 7efd31e00640 1 --2- 172.21.15.98:0/2566878361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0735d0 0x7efd3c151430 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7efd3c06c530 tx=0x7efd2401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a465cc0da09d05 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd39e00640 1 -- 172.21.15.98:0/2566878361 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd240b0070 con 0x7efd3c0735d0 2024-05-27T00:19:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd39e00640 1 -- 172.21.15.98:0/2566878361 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd240ab600 con 0x7efd3c0735d0 2024-05-27T00:19:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd39e00640 1 -- 172.21.15.98:0/2566878361 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd240ab9d0 con 0x7efd3c0735d0 2024-05-27T00:19:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 -- 172.21.15.98:0/2566878361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0735d0 msgr2=0x7efd3c151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 --2- 172.21.15.98:0/2566878361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0735d0 0x7efd3c151430 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7efd3c06c530 tx=0x7efd2401c480 comp rx=0 tx=0).stop 2024-05-27T00:19:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 -- 172.21.15.98:0/2566878361 shutdown_connections 2024-05-27T00:19:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 --2- 172.21.15.98:0/2566878361 >> v2:172.21.15.204:3300/0 conn(0x7efd3c0c79f0 0x7efd3c151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 --2- 172.21.15.98:0/2566878361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0735d0 0x7efd3c151430 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 -- 172.21.15.98:0/2566878361 >> 172.21.15.98:0/2566878361 conn(0x7efd3c067530 msgr2=0x7efd3c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 -- 172.21.15.98:0/2566878361 shutdown_connections 2024-05-27T00:19:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 -- 172.21.15.98:0/2566878361 wait complete. 2024-05-27T00:19:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 Processor -- start 2024-05-27T00:19:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 -- start start 2024-05-27T00:19:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0731f0 0x7efd3c0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c0735d0 0x7efd3c13c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:42.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd3c0c79f0 0x7efd3c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c162090 con 0x7efd3c0731f0 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c161f10 con 0x7efd3c0c79f0 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd41400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efd3c162210 con 0x7efd3c0735d0 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd3a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0731f0 0x7efd3c0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd3a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0731f0 0x7efd3c0cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57816/0 (socket says 172.21.15.98:57816) 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd3a800640 1 -- 172.21.15.98:0/3623245734 learned_addr learned my addr 172.21.15.98:0/3623245734 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd31e00640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c0735d0 0x7efd3c13c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd3b200640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd3c0c79f0 0x7efd3c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.987+0000 7efd3a800640 1 -- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c0735d0 msgr2=0x7efd3c13c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd3a800640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c0735d0 0x7efd3c13c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd3a800640 1 -- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd3c0c79f0 msgr2=0x7efd3c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd3a800640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd3c0c79f0 0x7efd3c13c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:42.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd3a800640 1 -- 172.21.15.98:0/3623245734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3c153de0 con 0x7efd3c0731f0 2024-05-27T00:19:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd31e00640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c0735d0 0x7efd3c13c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:19:42.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd3a800640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0731f0 0x7efd3c0cb5e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00c890 tx=0x7efd2c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd33400640 1 -- 172.21.15.98:0/3623245734 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c018070 con 0x7efd3c0731f0 2024-05-27T00:19:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd33400640 1 -- 172.21.15.98:0/3623245734 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd2c00a640 con 0x7efd3c0731f0 2024-05-27T00:19:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd33400640 1 -- 172.21.15.98:0/3623245734 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c00aa30 con 0x7efd3c0731f0 2024-05-27T00:19:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd41400640 1 -- 172.21.15.98:0/3623245734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3c15b710 con 0x7efd3c0731f0 2024-05-27T00:19:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd41400640 1 -- 172.21.15.98:0/3623245734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd3c154330 con 0x7efd3c0731f0 2024-05-27T00:19:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd41400640 1 -- 172.21.15.98:0/3623245734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd3c075cb0 con 0x7efd3c0731f0 2024-05-27T00:19:42.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd33400640 1 -- 172.21.15.98:0/3623245734 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efd2c00abd0 con 0x7efd3c0731f0 2024-05-27T00:19:42.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd33400640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd180678f0 0x7efd18069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:42.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd33400640 1 -- 172.21.15.98:0/3623245734 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7efd2c085530 con 0x7efd3c0731f0 2024-05-27T00:19:42.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.991+0000 7efd31e00640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd180678f0 0x7efd18069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:42.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.995+0000 7efd31e00640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd180678f0 0x7efd18069db0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efd240ad000 tx=0x7efd240a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:43.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:42.995+0000 7efd33400640 1 -- 172.21.15.98:0/3623245734 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7efd2c055e50 con 0x7efd3c0731f0 2024-05-27T00:19:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.183+0000 7efd41400640 1 -- 172.21.15.98:0/3623245734 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd3c158510 con 0x7efd3c0731f0 2024-05-27T00:19:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd33400640 1 -- 172.21.15.98:0/3623245734 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6006 (secure 0 0 0) 0x7efd2c055750 con 0x7efd3c0731f0 2024-05-27T00:19:43.190 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:43.190 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:35.190051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:19:43.190 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 19 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 -- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd180678f0 msgr2=0x7efd18069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd180678f0 0x7efd18069db0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efd240ad000 tx=0x7efd240a6000 comp rx=0 tx=0).stop 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 -- 172.21.15.98:0/3623245734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0731f0 msgr2=0x7efd3c0cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0731f0 0x7efd3c0cb5e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00c890 tx=0x7efd2c00cd50 comp rx=0 tx=0).stop 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 -- 172.21.15.98:0/3623245734 shutdown_connections 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd180678f0 0x7efd18069db0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd3c0c79f0 0x7efd3c13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd3c0735d0 0x7efd3c13c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 --2- 172.21.15.98:0/3623245734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd3c0731f0 0x7efd3c0cb5e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 -- 172.21.15.98:0/3623245734 >> 172.21.15.98:0/3623245734 conn(0x7efd3c067530 msgr2=0x7efd3c14fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:43.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 -- 172.21.15.98:0/3623245734 shutdown_connections 2024-05-27T00:19:43.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:43.187+0000 7efd41400640 1 -- 172.21.15.98:0/3623245734 wait complete. 2024-05-27T00:19:43.211 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:19:45.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:19:45.031 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:19:45.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.351+0000 7fa30e200640 1 Processor -- start 2024-05-27T00:19:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.351+0000 7fa30e200640 1 -- start start 2024-05-27T00:19:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.351+0000 7fa30e200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa308070e10 0x7fa3081627e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.351+0000 7fa30e200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3080c1000 con 0x7fa3080cc8f0 2024-05-27T00:19:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.351+0000 7fa30e200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3081515b0 con 0x7fa3081511d0 2024-05-27T00:19:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.351+0000 7fa30e200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa308071240 con 0x7fa308070e10 2024-05-27T00:19:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.351+0000 7fa2fea00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa3080cc8f0 0x7fa3080c8670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43324/0 (socket says 172.21.15.98:43324) 2024-05-27T00:19:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.351+0000 7fa2fea00640 1 -- 172.21.15.98:0/2727102172 learned_addr learned my addr 172.21.15.98:0/2727102172 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:19:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.351+0000 7fa307e00640 1 --2- 172.21.15.98:0/2727102172 >> v2:172.21.15.98:3300/0 conn(0x7fa308070e10 0x7fa3081627e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa307e00640 1 -- 172.21.15.98:0/2727102172 >> v1:172.21.15.204:6789/0 conn(0x7fa3081511d0 legacy=0x7fa3080c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa307e00640 1 -- 172.21.15.98:0/2727102172 >> v1:172.21.15.98:6789/0 conn(0x7fa3080cc8f0 legacy=0x7fa3080c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:45.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa307e00640 1 -- 172.21.15.98:0/2727102172 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3080c83c0 con 0x7fa308070e10 2024-05-27T00:19:45.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa307e00640 1 --2- 172.21.15.98:0/2727102172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 0x7fa3081627e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa30000aea0 tx=0x7fa30001d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d26b86ffd9468fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa306a00640 1 -- 172.21.15.98:0/2727102172 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3000a6640 con 0x7fa308070e10 2024-05-27T00:19:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa306a00640 1 -- 172.21.15.98:0/2727102172 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3000a67e0 con 0x7fa308070e10 2024-05-27T00:19:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa306a00640 1 -- 172.21.15.98:0/2727102172 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3000a6bb0 con 0x7fa308070e10 2024-05-27T00:19:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa30e200640 1 -- 172.21.15.98:0/2727102172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 msgr2=0x7fa3081627e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:45.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa30e200640 1 --2- 172.21.15.98:0/2727102172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 0x7fa3081627e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa30000aea0 tx=0x7fa30001d9e0 comp rx=0 tx=0).stop 2024-05-27T00:19:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa30e200640 1 -- 172.21.15.98:0/2727102172 shutdown_connections 2024-05-27T00:19:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa30e200640 1 --2- 172.21.15.98:0/2727102172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 0x7fa3081627e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.355+0000 7fa30e200640 1 -- 172.21.15.98:0/2727102172 >> 172.21.15.98:0/2727102172 conn(0x7fa308067530 msgr2=0x7fa30806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 -- 172.21.15.98:0/2727102172 shutdown_connections 2024-05-27T00:19:45.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 -- 172.21.15.98:0/2727102172 wait complete. 2024-05-27T00:19:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 Processor -- start 2024-05-27T00:19:45.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 -- start start 2024-05-27T00:19:45.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 0x7fa308143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:45.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa3080cc8f0 0x7fa308154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa3081511d0 0x7fa308155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa3080c1a20 con 0x7fa308070e10 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa3080b1dd0 con 0x7fa3080cc8f0 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa3080c2150 con 0x7fa3081511d0 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa307400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 0x7fa308143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa307400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 0x7fa308143970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35388/0 (socket says 172.21.15.98:35388) 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa307400640 1 -- 172.21.15.98:0/1716275362 learned_addr learned my addr 172.21.15.98:0/1716275362 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa2fea00640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa3080cc8f0 0x7fa308154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa307e00640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa3081511d0 0x7fa308155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa307400640 1 -- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa3081511d0 msgr2=0x7fa308155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa307400640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa3081511d0 0x7fa308155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa307400640 1 -- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa3080cc8f0 msgr2=0x7fa308154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:45.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa307400640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa3080cc8f0 0x7fa308154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa307400640 1 -- 172.21.15.98:0/1716275362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3081f13e0 con 0x7fa308070e10 2024-05-27T00:19:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa2fea00640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa3080cc8f0 0x7fa308154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:19:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa307400640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 0x7fa308143970 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800c370 tx=0x7fa2f800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa2ffe00640 1 -- 172.21.15.98:0/1716275362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f80052d0 con 0x7fa308070e10 2024-05-27T00:19:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa2ffe00640 1 -- 172.21.15.98:0/1716275362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f8005470 con 0x7fa308070e10 2024-05-27T00:19:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa30e200640 1 -- 172.21.15.98:0/1716275362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa308162910 con 0x7fa308070e10 2024-05-27T00:19:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.359+0000 7fa2ffe00640 1 -- 172.21.15.98:0/1716275362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8007260 con 0x7fa308070e10 2024-05-27T00:19:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.363+0000 7fa30e200640 1 -- 172.21.15.98:0/1716275362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3081f1660 con 0x7fa308070e10 2024-05-27T00:19:45.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.363+0000 7fa30e200640 1 -- 172.21.15.98:0/1716275362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa308075cb0 con 0x7fa308070e10 2024-05-27T00:19:45.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.363+0000 7fa2ffe00640 1 -- 172.21.15.98:0/1716275362 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa2f8007400 con 0x7fa308070e10 2024-05-27T00:19:45.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.363+0000 7fa2ffe00640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2e40678f0 0x7fa2e4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:45.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.363+0000 7fa2ffe00640 1 -- 172.21.15.98:0/1716275362 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7fa2f8085390 con 0x7fa308070e10 2024-05-27T00:19:45.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.363+0000 7fa2fea00640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2e40678f0 0x7fa2e4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:45.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.367+0000 7fa2fea00640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2e40678f0 0x7fa2e4069db0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa308162800 tx=0x7fa2f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:45.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.367+0000 7fa2ffe00640 1 -- 172.21.15.98:0/1716275362 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa3081f13e0 con 0x7fa308070e10 2024-05-27T00:19:45.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 -- 172.21.15.98:0/1716275362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa30814e5b0 con 0x7fa308070e10 2024-05-27T00:19:45.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa2ffe00640 1 -- 172.21.15.98:0/1716275362 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6010 (secure 0 0 0) 0x7fa2f8055c20 con 0x7fa308070e10 2024-05-27T00:19:45.563 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:45.563 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:42.830167+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:19:45.563 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 20 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 -- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2e40678f0 msgr2=0x7fa2e4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2e40678f0 0x7fa2e4069db0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa308162800 tx=0x7fa2f0009040 comp rx=0 tx=0).stop 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 -- 172.21.15.98:0/1716275362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 msgr2=0x7fa308143970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 0x7fa308143970 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800c370 tx=0x7fa2f800c830 comp rx=0 tx=0).stop 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 -- 172.21.15.98:0/1716275362 shutdown_connections 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa3081511d0 0x7fa308155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2e40678f0 0x7fa2e4069db0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa3080cc8f0 0x7fa308154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 --2- 172.21.15.98:0/1716275362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa308070e10 0x7fa308143970 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 -- 172.21.15.98:0/1716275362 >> 172.21.15.98:0/1716275362 conn(0x7fa308067530 msgr2=0x7fa3080c6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 -- 172.21.15.98:0/1716275362 shutdown_connections 2024-05-27T00:19:45.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:45.559+0000 7fa30e200640 1 -- 172.21.15.98:0/1716275362 wait complete. 2024-05-27T00:19:45.583 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:19:47.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:47.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe777400640 1 Processor -- start 2024-05-27T00:19:47.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe777400640 1 -- start start 2024-05-27T00:19:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe777400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe7700cbe10 0x7fe7701592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe777400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe770073570 0x7fe7701559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe777400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7700c1f60 con 0x7fe7700cc1f0 2024-05-27T00:19:47.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe777400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe770159810 con 0x7fe770073570 2024-05-27T00:19:47.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe777400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fe7700c6dd0 con 0x7fe7700cbe10 2024-05-27T00:19:47.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe775400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe770073570 0x7fe7701559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:47.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe775400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe770073570 0x7fe7701559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35394/0 (socket says 172.21.15.98:35394) 2024-05-27T00:19:47.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe775400640 1 -- 172.21.15.98:0/2994399400 learned_addr learned my addr 172.21.15.98:0/2994399400 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:47.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe774a00640 1 --2- 172.21.15.98:0/2994399400 >> v2:172.21.15.204:3300/0 conn(0x7fe7700cbe10 0x7fe7701592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:47.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe775400640 1 -- 172.21.15.98:0/2994399400 >> v2:172.21.15.204:3300/0 conn(0x7fe7700cbe10 msgr2=0x7fe7701592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe775400640 1 --2- 172.21.15.98:0/2994399400 >> v2:172.21.15.204:3300/0 conn(0x7fe7700cbe10 0x7fe7701592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe775400640 1 -- 172.21.15.98:0/2994399400 >> v1:172.21.15.204:6789/0 conn(0x7fe7700cc1f0 legacy=0x7fe7700c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe775400640 1 -- 172.21.15.98:0/2994399400 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7700c70b0 con 0x7fe770073570 2024-05-27T00:19:47.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.727+0000 7fe774a00640 1 --2- 172.21.15.98:0/2994399400 >> v2:172.21.15.204:3300/0 conn(0x7fe7700cbe10 0x7fe7701592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:19:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe775400640 1 --2- 172.21.15.98:0/2994399400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 0x7fe7701559e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe7700cc760 tx=0x7fe76801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf646fb2586899 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe76f400640 1 -- 172.21.15.98:0/2994399400 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7680b4070 con 0x7fe770073570 2024-05-27T00:19:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe76f400640 1 -- 172.21.15.98:0/2994399400 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7680af400 con 0x7fe770073570 2024-05-27T00:19:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe76f400640 1 -- 172.21.15.98:0/2994399400 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7680af7d0 con 0x7fe770073570 2024-05-27T00:19:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe777400640 1 -- 172.21.15.98:0/2994399400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 msgr2=0x7fe7701559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe777400640 1 --2- 172.21.15.98:0/2994399400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 0x7fe7701559e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe7700cc760 tx=0x7fe76801f1e0 comp rx=0 tx=0).stop 2024-05-27T00:19:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe777400640 1 -- 172.21.15.98:0/2994399400 shutdown_connections 2024-05-27T00:19:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe777400640 1 --2- 172.21.15.98:0/2994399400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 0x7fe7701559e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe777400640 1 --2- 172.21.15.98:0/2994399400 >> v2:172.21.15.204:3300/0 conn(0x7fe7700cbe10 0x7fe7701592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:47.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe777400640 1 -- 172.21.15.98:0/2994399400 >> 172.21.15.98:0/2994399400 conn(0x7fe770067530 msgr2=0x7fe77006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe777400640 1 -- 172.21.15.98:0/2994399400 shutdown_connections 2024-05-27T00:19:47.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.731+0000 7fe777400640 1 -- 172.21.15.98:0/2994399400 wait complete. 2024-05-27T00:19:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe777400640 1 Processor -- start 2024-05-27T00:19:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe777400640 1 -- start start 2024-05-27T00:19:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe777400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 0x7fe7701f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe777400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7700cbe10 0x7fe7700c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe777400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7700cc1f0 0x7fe77014dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:47.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe777400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7700c1330 con 0x7fe770073570 2024-05-27T00:19:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe777400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe7700c1050 con 0x7fe7700cbe10 2024-05-27T00:19:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe777400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe7700c79d0 con 0x7fe7700cc1f0 2024-05-27T00:19:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe774a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 0x7fe7701f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe774a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 0x7fe7701f38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35410/0 (socket says 172.21.15.98:35410) 2024-05-27T00:19:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe774a00640 1 -- 172.21.15.98:0/323836696 learned_addr learned my addr 172.21.15.98:0/323836696 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe76fe00640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7700cbe10 0x7fe7700c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:47.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe775400640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7700cc1f0 0x7fe77014dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe774a00640 1 -- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7700cc1f0 msgr2=0x7fe77014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe774a00640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7700cc1f0 0x7fe77014dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe774a00640 1 -- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7700cbe10 msgr2=0x7fe7700c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe774a00640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7700cbe10 0x7fe7700c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe774a00640 1 -- 172.21.15.98:0/323836696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe770071fa0 con 0x7fe770073570 2024-05-27T00:19:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe76fe00640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7700cbe10 0x7fe7700c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:19:47.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe774a00640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 0x7fe7701f38d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe76000e840 tx=0x7fe76000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe76cc00640 1 -- 172.21.15.98:0/323836696 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe760017070 con 0x7fe770073570 2024-05-27T00:19:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe76cc00640 1 -- 172.21.15.98:0/323836696 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe76000c8d0 con 0x7fe770073570 2024-05-27T00:19:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe777400640 1 -- 172.21.15.98:0/323836696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7701f4510 con 0x7fe770073570 2024-05-27T00:19:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe76cc00640 1 -- 172.21.15.98:0/323836696 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76000f040 con 0x7fe770073570 2024-05-27T00:19:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.735+0000 7fe777400640 1 -- 172.21.15.98:0/323836696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe770072940 con 0x7fe770073570 2024-05-27T00:19:47.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.739+0000 7fe777400640 1 -- 172.21.15.98:0/323836696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe770075cb0 con 0x7fe770073570 2024-05-27T00:19:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.739+0000 7fe76cc00640 1 -- 172.21.15.98:0/323836696 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe76001a070 con 0x7fe770073570 2024-05-27T00:19:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.739+0000 7fe76cc00640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe7500678c0 0x7fe750069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.739+0000 7fe76cc00640 1 -- 172.21.15.98:0/323836696 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7fe760085b10 con 0x7fe770073570 2024-05-27T00:19:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.739+0000 7fe76fe00640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe7500678c0 0x7fe750069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:47.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.739+0000 7fe76fe00640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe7500678c0 0x7fe750069d80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe770157db0 tx=0x7fe75c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:47.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.743+0000 7fe76cc00640 1 -- 172.21.15.98:0/323836696 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fe760056430 con 0x7fe770073570 2024-05-27T00:19:47.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 -- 172.21.15.98:0/323836696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7700b1370 con 0x7fe770073570 2024-05-27T00:19:47.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe76cc00640 1 -- 172.21.15.98:0/323836696 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6010 (secure 0 0 0) 0x7fe760055d30 con 0x7fe770073570 2024-05-27T00:19:47.935 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:47.935 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:42.830167+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:19:47.936 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 20 2024-05-27T00:19:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 -- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe7500678c0 msgr2=0x7fe750069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe7500678c0 0x7fe750069d80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe770157db0 tx=0x7fe75c009040 comp rx=0 tx=0).stop 2024-05-27T00:19:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 -- 172.21.15.98:0/323836696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 msgr2=0x7fe7701f38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 0x7fe7701f38d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe76000e840 tx=0x7fe76000ed00 comp rx=0 tx=0).stop 2024-05-27T00:19:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 -- 172.21.15.98:0/323836696 shutdown_connections 2024-05-27T00:19:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7700cc1f0 0x7fe77014dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe7500678c0 0x7fe750069d80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7700cbe10 0x7fe7700c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 --2- 172.21.15.98:0/323836696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe770073570 0x7fe7701f38d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 -- 172.21.15.98:0/323836696 >> 172.21.15.98:0/323836696 conn(0x7fe770067530 msgr2=0x7fe770158ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 -- 172.21.15.98:0/323836696 shutdown_connections 2024-05-27T00:19:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:47.931+0000 7fe777400640 1 -- 172.21.15.98:0/323836696 wait complete. 2024-05-27T00:19:47.956 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 20, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5073": { "addr": "172.21.15.98:6834/3247908011", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 3247908011, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 3247908011, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5073, "incarnation": 19, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:19:42.830167+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5073 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:19:49.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.099+0000 7f4d5fa00640 1 Processor -- start 2024-05-27T00:19:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.099+0000 7f4d5fa00640 1 -- start start 2024-05-27T00:19:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5fa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d580731f0 0x7f4d580c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5fa00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4d580c79f0 0x7f4d58157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5fa00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d580c17d0 con 0x7f4d580735d0 2024-05-27T00:19:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5fa00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4d58071990 con 0x7f4d580731f0 2024-05-27T00:19:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5fa00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f4d58070e10 con 0x7f4d580c79f0 2024-05-27T00:19:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5d000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d580731f0 0x7f4d580c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5d000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d580731f0 0x7f4d580c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35424/0 (socket says 172.21.15.98:35424) 2024-05-27T00:19:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5d000640 1 -- 172.21.15.98:0/2843378219 learned_addr learned my addr 172.21.15.98:0/2843378219 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5da00640 1 --2- 172.21.15.98:0/2843378219 >> v2:172.21.15.204:3300/0 conn(0x7f4d580c79f0 0x7f4d58157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5d000640 1 -- 172.21.15.98:0/2843378219 >> v2:172.21.15.204:3300/0 conn(0x7f4d580c79f0 msgr2=0x7f4d58157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5d000640 1 --2- 172.21.15.98:0/2843378219 >> v2:172.21.15.204:3300/0 conn(0x7f4d580c79f0 0x7f4d58157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5d000640 1 -- 172.21.15.98:0/2843378219 >> v1:172.21.15.98:6789/0 conn(0x7f4d580735d0 legacy=0x7f4d58151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d5d000640 1 -- 172.21.15.98:0/2843378219 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d580710f0 con 0x7f4d580731f0 2024-05-27T00:19:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.103+0000 7f4d57400640 1 -- 172.21.15.98:0/2843378219 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335907226 0 0) 0x7f4d580c17d0 con 0x7f4d580735d0 2024-05-27T00:19:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5d000640 1 --2- 172.21.15.98:0/2843378219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580731f0 0x7f4d580c0eb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4d5806c530 tx=0x7f4d4c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37389fc9f04ac9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d57400640 1 -- 172.21.15.98:0/2843378219 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c0aa650 con 0x7f4d580731f0 2024-05-27T00:19:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d57400640 1 -- 172.21.15.98:0/2843378219 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d4c0af070 con 0x7f4d580731f0 2024-05-27T00:19:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d57400640 1 -- 172.21.15.98:0/2843378219 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c0aaa10 con 0x7f4d580731f0 2024-05-27T00:19:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2843378219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580731f0 msgr2=0x7f4d580c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5fa00640 1 --2- 172.21.15.98:0/2843378219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580731f0 0x7f4d580c0eb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4d5806c530 tx=0x7f4d4c01c4f0 comp rx=0 tx=0).stop 2024-05-27T00:19:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2843378219 shutdown_connections 2024-05-27T00:19:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5fa00640 1 --2- 172.21.15.98:0/2843378219 >> v2:172.21.15.204:3300/0 conn(0x7f4d580c79f0 0x7f4d58157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5fa00640 1 --2- 172.21.15.98:0/2843378219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580731f0 0x7f4d580c0eb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2843378219 >> 172.21.15.98:0/2843378219 conn(0x7f4d58067530 msgr2=0x7f4d5806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2843378219 shutdown_connections 2024-05-27T00:19:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2843378219 wait complete. 2024-05-27T00:19:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5fa00640 1 Processor -- start 2024-05-27T00:19:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.107+0000 7f4d5fa00640 1 -- start start 2024-05-27T00:19:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5fa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d580731f0 0x7f4d581532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5fa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580735d0 0x7f4d5815ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5fa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d580c79f0 0x7f4d581f6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5fa00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d580c13f0 con 0x7f4d580735d0 2024-05-27T00:19:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5fa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4d580c8400 con 0x7f4d580c79f0 2024-05-27T00:19:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5fa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4d580c1570 con 0x7f4d580731f0 2024-05-27T00:19:50.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d57e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580735d0 0x7f4d5815ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d57e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580735d0 0x7f4d5815ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35430/0 (socket says 172.21.15.98:35430) 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d57e00640 1 -- 172.21.15.98:0/2577301636 learned_addr learned my addr 172.21.15.98:0/2577301636 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5d000640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d580731f0 0x7f4d581532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5da00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d580c79f0 0x7f4d581f6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d57e00640 1 -- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d580731f0 msgr2=0x7f4d581532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d57e00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d580731f0 0x7f4d581532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d57e00640 1 -- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d580c79f0 msgr2=0x7f4d581f6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d57e00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d580c79f0 0x7f4d581f6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d57e00640 1 -- 172.21.15.98:0/2577301636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5814d840 con 0x7f4d580735d0 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d57e00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580735d0 0x7f4d5815ab80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400c550 tx=0x7f4d4400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d54c00640 1 -- 172.21.15.98:0/2577301636 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44017070 con 0x7f4d580735d0 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d54c00640 1 -- 172.21.15.98:0/2577301636 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d44003600 con 0x7f4d580735d0 2024-05-27T00:19:50.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2577301636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d580c9be0 con 0x7f4d580735d0 2024-05-27T00:19:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d54c00640 1 -- 172.21.15.98:0/2577301636 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d440039f0 con 0x7f4d580735d0 2024-05-27T00:19:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2577301636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d58151dd0 con 0x7f4d580735d0 2024-05-27T00:19:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2577301636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d58075cb0 con 0x7f4d580735d0 2024-05-27T00:19:50.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.111+0000 7f4d54c00640 1 -- 172.21.15.98:0/2577301636 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4d4401a070 con 0x7f4d580735d0 2024-05-27T00:19:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.115+0000 7f4d54c00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d380678c0 0x7f4d38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.115+0000 7f4d54c00640 1 -- 172.21.15.98:0/2577301636 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7f4d440851c0 con 0x7f4d580735d0 2024-05-27T00:19:50.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.115+0000 7f4d5d000640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d380678c0 0x7f4d38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:50.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.115+0000 7f4d5d000640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d380678c0 0x7f4d38069d80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0ae000 tx=0x7f4d4c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:50.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.119+0000 7f4d54c00640 1 -- 172.21.15.98:0/2577301636 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4d44055520 con 0x7f4d580735d0 2024-05-27T00:19:50.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.303+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2577301636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d5814f060 con 0x7f4d580735d0 2024-05-27T00:19:50.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.303+0000 7f4d54c00640 1 -- 172.21.15.98:0/2577301636 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6007 (secure 0 0 0) 0x7f4d44059820 con 0x7f4d580735d0 2024-05-27T00:19:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:50.312 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:48.442931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:19:50.313 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 21 2024-05-27T00:19:50.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d380678c0 msgr2=0x7f4d38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:50.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d380678c0 0x7f4d38069d80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0ae000 tx=0x7f4d4c0a7000 comp rx=0 tx=0).stop 2024-05-27T00:19:50.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2577301636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580735d0 msgr2=0x7f4d5815ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:50.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580735d0 0x7f4d5815ab80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400c550 tx=0x7f4d4400ca10 comp rx=0 tx=0).stop 2024-05-27T00:19:50.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2577301636 shutdown_connections 2024-05-27T00:19:50.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d380678c0 0x7f4d38069d80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:50.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d580c79f0 0x7f4d581f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:50.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d580735d0 0x7f4d5815ab80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:50.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 --2- 172.21.15.98:0/2577301636 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d580731f0 0x7f4d581532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:50.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2577301636 >> 172.21.15.98:0/2577301636 conn(0x7f4d58067530 msgr2=0x7f4d581468a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:50.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2577301636 shutdown_connections 2024-05-27T00:19:50.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:50.307+0000 7f4d5fa00640 1 -- 172.21.15.98:0/2577301636 wait complete. 2024-05-27T00:19:50.329 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:19:52.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef7d200640 1 Processor -- start 2024-05-27T00:19:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef7d200640 1 -- start start 2024-05-27T00:19:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef7d200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fef78152a50 0x7fef780709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef7d200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7815be40 con 0x7fef78158e60 2024-05-27T00:19:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef7d200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef780c50e0 con 0x7fef780cb7e0 2024-05-27T00:19:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef7d200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fef780c8a00 con 0x7fef78152a50 2024-05-27T00:19:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef76800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fef78158e60 0x7fef78162ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43348/0 (socket says 172.21.15.98:43348) 2024-05-27T00:19:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef76800640 1 -- 172.21.15.98:0/2956928928 learned_addr learned my addr 172.21.15.98:0/2956928928 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:19:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75e00640 1 --2- 172.21.15.98:0/2956928928 >> v2:172.21.15.204:3300/0 conn(0x7fef78152a50 0x7fef780709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 474933615 0 0) 0x7fef7815be40 con 0x7fef78158e60 2024-05-27T00:19:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef5c003530 con 0x7fef78158e60 2024-05-27T00:19:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 544839569 0 0) 0x7fef5c003530 con 0x7fef78158e60 2024-05-27T00:19:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 >> v2:172.21.15.204:3300/0 conn(0x7fef78152a50 msgr2=0x7fef780709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 --2- 172.21.15.98:0/2956928928 >> v2:172.21.15.204:3300/0 conn(0x7fef78152a50 0x7fef780709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 >> v1:172.21.15.204:6789/0 conn(0x7fef780cb7e0 legacy=0x7fef78158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef780c8c50 con 0x7fef78158e60 2024-05-27T00:19:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fef700016a0 con 0x7fef78158e60 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 >> v1:172.21.15.98:6789/0 conn(0x7fef78158e60 legacy=0x7fef78162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5c0039c0 0x7fef5c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fef5c004810 0x7fef5c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef5c0051d0 0x7fef5c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef780c8a00 con 0x7fef5c0051d0 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c00c170 con 0x7fef5c0039c0 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fef5c00c370 con 0x7fef5c004810 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.475+0000 7fef76800640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef5c0051d0 0x7fef5c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef75e00640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5c0039c0 0x7fef5c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef6d400640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fef5c004810 0x7fef5c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef76800640 1 -- 172.21.15.98:0/2956928928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fef5c004810 msgr2=0x7fef5c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef76800640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fef5c004810 0x7fef5c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef76800640 1 -- 172.21.15.98:0/2956928928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5c0039c0 msgr2=0x7fef5c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef76800640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5c0039c0 0x7fef5c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef76800640 1 -- 172.21.15.98:0/2956928928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c003530 con 0x7fef5c0051d0 2024-05-27T00:19:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef6d400640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fef5c004810 0x7fef5c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:19:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef76800640 1 -- 172.21.15.98:0/2956928928 reap_dead start 2024-05-27T00:19:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef76800640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef5c0051d0 0x7fef5c00b9d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fef7815c010 tx=0x7fef7001f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413feab811974ab7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70007d60 con 0x7fef5c0051d0 2024-05-27T00:19:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef70005b00 con 0x7fef5c0051d0 2024-05-27T00:19:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef75400640 1 -- 172.21.15.98:0/2956928928 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70007070 con 0x7fef5c0051d0 2024-05-27T00:19:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef7d200640 1 -- 172.21.15.98:0/2956928928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef5c0051d0 msgr2=0x7fef5c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.479+0000 7fef7d200640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef5c0051d0 0x7fef5c00b9d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fef7815c010 tx=0x7fef7001f430 comp rx=0 tx=0).stop 2024-05-27T00:19:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 -- 172.21.15.98:0/2956928928 shutdown_connections 2024-05-27T00:19:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 --2- 172.21.15.98:0/2956928928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef5c0051d0 0x7fef5c00b9d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 -- 172.21.15.98:0/2956928928 >> 172.21.15.98:0/2956928928 conn(0x7fef78072f90 msgr2=0x7fef780733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 -- 172.21.15.98:0/2956928928 shutdown_connections 2024-05-27T00:19:52.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 -- 172.21.15.98:0/2956928928 wait complete. 2024-05-27T00:19:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 Processor -- start 2024-05-27T00:19:52.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 -- start start 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5c0051d0 0x7fef78153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78154730 0x7fef78157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fef781550d0 0x7fef78155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef780c9210 con 0x7fef78154730 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fef7815bcc0 con 0x7fef5c0051d0 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fef780c94f0 con 0x7fef781550d0 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78154730 0x7fef78157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78154730 0x7fef78157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35442/0 (socket says 172.21.15.98:35442) 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6d400640 1 -- 172.21.15.98:0/1861899805 learned_addr learned my addr 172.21.15.98:0/1861899805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef75e00640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5c0051d0 0x7fef78153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef76800640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fef781550d0 0x7fef78155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6d400640 1 -- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fef781550d0 msgr2=0x7fef78155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6d400640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fef781550d0 0x7fef78155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6d400640 1 -- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5c0051d0 msgr2=0x7fef78153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6d400640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5c0051d0 0x7fef78153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6d400640 1 -- 172.21.15.98:0/1861899805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef78155a90 con 0x7fef78154730 2024-05-27T00:19:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6d400640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78154730 0x7fef78157970 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fef60000c10 tx=0x7fef6000e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6ea00640 1 -- 172.21.15.98:0/1861899805 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60019070 con 0x7fef78154730 2024-05-27T00:19:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6ea00640 1 -- 172.21.15.98:0/1861899805 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef600052e0 con 0x7fef78154730 2024-05-27T00:19:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef6ea00640 1 -- 172.21.15.98:0/1861899805 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60003840 con 0x7fef78154730 2024-05-27T00:19:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 -- 172.21.15.98:0/1861899805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef781eeab0 con 0x7fef78154730 2024-05-27T00:19:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.483+0000 7fef7d200640 1 -- 172.21.15.98:0/1861899805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef781534d0 con 0x7fef78154730 2024-05-27T00:19:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.487+0000 7fef7d200640 1 -- 172.21.15.98:0/1861899805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef78075cb0 con 0x7fef78154730 2024-05-27T00:19:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.487+0000 7fef6ea00640 1 -- 172.21.15.98:0/1861899805 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef6001c070 con 0x7fef78154730 2024-05-27T00:19:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.487+0000 7fef6ea00640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fef540678f0 0x7fef54069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.487+0000 7fef6ea00640 1 -- 172.21.15.98:0/1861899805 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7fef600853c0 con 0x7fef78154730 2024-05-27T00:19:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.487+0000 7fef75e00640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fef540678f0 0x7fef54069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:52.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.491+0000 7fef75e00640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fef540678f0 0x7fef54069db0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fef68014450 tx=0x7fef6800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:52.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.491+0000 7fef6ea00640 1 -- 172.21.15.98:0/1861899805 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fef78155a90 con 0x7fef78154730 2024-05-27T00:19:52.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.679+0000 7fef7d200640 1 -- 172.21.15.98:0/1861899805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef780710d0 con 0x7fef78154730 2024-05-27T00:19:52.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef6ea00640 1 -- 172.21.15.98:0/1861899805 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6007 (secure 0 0 0) 0x7fef60055720 con 0x7fef78154730 2024-05-27T00:19:52.686 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:52.686 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:48.442931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:19:52.687 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 21 2024-05-27T00:19:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 -- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fef540678f0 msgr2=0x7fef54069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fef540678f0 0x7fef54069db0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fef68014450 tx=0x7fef6800b040 comp rx=0 tx=0).stop 2024-05-27T00:19:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 -- 172.21.15.98:0/1861899805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78154730 msgr2=0x7fef78157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78154730 0x7fef78157970 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fef60000c10 tx=0x7fef6000e910 comp rx=0 tx=0).stop 2024-05-27T00:19:52.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 -- 172.21.15.98:0/1861899805 shutdown_connections 2024-05-27T00:19:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fef781550d0 0x7fef78155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78154730 0x7fef78157970 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fef540678f0 0x7fef54069db0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 --2- 172.21.15.98:0/1861899805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5c0051d0 0x7fef78153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 -- 172.21.15.98:0/1861899805 >> 172.21.15.98:0/1861899805 conn(0x7fef78072f90 msgr2=0x7fef7815b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 -- 172.21.15.98:0/1861899805 shutdown_connections 2024-05-27T00:19:52.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:52.683+0000 7fef7d200640 1 -- 172.21.15.98:0/1861899805 wait complete. 2024-05-27T00:19:52.707 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:19:54.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:19:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.847+0000 7f9e55000640 1 Processor -- start 2024-05-27T00:19:54.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e55000640 1 -- start start 2024-05-27T00:19:54.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e55000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e501525a0 0x7f9e500629d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e55000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9e500c90f0 0x7f9e5006f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e55000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e500c62c0 con 0x7f9e500c76b0 2024-05-27T00:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e55000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9e500c7b80 con 0x7f9e501525a0 2024-05-27T00:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e55000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9e50159190 con 0x7f9e500c90f0 2024-05-27T00:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e4f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e501525a0 0x7f9e500629d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e4f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e501525a0 0x7f9e500629d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42976/0 (socket says 172.21.15.98:42976) 2024-05-27T00:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e4f400640 1 -- 172.21.15.98:0/3346260943 learned_addr learned my addr 172.21.15.98:0/3346260943 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e4fe00640 1 --2- 172.21.15.98:0/3346260943 >> v2:172.21.15.204:3300/0 conn(0x7f9e500c90f0 0x7f9e5006f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e4f400640 1 -- 172.21.15.98:0/3346260943 >> v2:172.21.15.204:3300/0 conn(0x7f9e500c90f0 msgr2=0x7f9e5006f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e4f400640 1 --2- 172.21.15.98:0/3346260943 >> v2:172.21.15.204:3300/0 conn(0x7f9e500c90f0 0x7f9e5006f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e4f400640 1 -- 172.21.15.98:0/3346260943 >> v1:172.21.15.204:6789/0 conn(0x7f9e500c76b0 legacy=0x7f9e500c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.851+0000 7f9e4f400640 1 -- 172.21.15.98:0/3346260943 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e50159430 con 0x7f9e501525a0 2024-05-27T00:19:54.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e4f400640 1 --2- 172.21.15.98:0/3346260943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 0x7f9e500629d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9e5006c340 tx=0x7f9e4001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0c3600d92bd971 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e4ea00640 1 -- 172.21.15.98:0/3346260943 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e400b0070 con 0x7f9e501525a0 2024-05-27T00:19:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e4ea00640 1 -- 172.21.15.98:0/3346260943 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e400ab600 con 0x7f9e501525a0 2024-05-27T00:19:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e4ea00640 1 -- 172.21.15.98:0/3346260943 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e400ab9d0 con 0x7f9e501525a0 2024-05-27T00:19:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e55000640 1 -- 172.21.15.98:0/3346260943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 msgr2=0x7f9e500629d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:54.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e55000640 1 --2- 172.21.15.98:0/3346260943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 0x7f9e500629d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9e5006c340 tx=0x7f9e4001c520 comp rx=0 tx=0).stop 2024-05-27T00:19:54.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e55000640 1 -- 172.21.15.98:0/3346260943 shutdown_connections 2024-05-27T00:19:54.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e55000640 1 --2- 172.21.15.98:0/3346260943 >> v2:172.21.15.204:3300/0 conn(0x7f9e500c90f0 0x7f9e5006f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:54.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e55000640 1 --2- 172.21.15.98:0/3346260943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 0x7f9e500629d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:54.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e55000640 1 -- 172.21.15.98:0/3346260943 >> 172.21.15.98:0/3346260943 conn(0x7f9e50067730 msgr2=0x7f9e5006b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:54.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e55000640 1 -- 172.21.15.98:0/3346260943 shutdown_connections 2024-05-27T00:19:54.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e55000640 1 -- 172.21.15.98:0/3346260943 wait complete. 2024-05-27T00:19:54.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e55000640 1 Processor -- start 2024-05-27T00:19:54.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.855+0000 7f9e55000640 1 -- start start 2024-05-27T00:19:54.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e55000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e500c76b0 0x7f9e50145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:54.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e55000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e500c90f0 0x7f9e501f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:54.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e55000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 0x7f9e5013d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:54.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e55000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501650f0 con 0x7f9e501525a0 2024-05-27T00:19:54.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e55000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50164f70 con 0x7f9e500c90f0 2024-05-27T00:19:54.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e55000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9e50165270 con 0x7f9e500c76b0 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e4fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 0x7f9e5013d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e4fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 0x7f9e5013d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42984/0 (socket says 172.21.15.98:42984) 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e4fe00640 1 -- 172.21.15.98:0/1866471256 learned_addr learned my addr 172.21.15.98:0/1866471256 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e4f400640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e500c76b0 0x7f9e50145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e46a00640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e500c90f0 0x7f9e501f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e4fe00640 1 -- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e500c76b0 msgr2=0x7f9e50145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e4fe00640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e500c76b0 0x7f9e50145540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e4fe00640 1 -- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e500c90f0 msgr2=0x7f9e501f2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e4fe00640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e500c90f0 0x7f9e501f2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e4fe00640 1 -- 172.21.15.98:0/1866471256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e50140450 con 0x7f9e501525a0 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e46a00640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e500c90f0 0x7f9e501f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e4fe00640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 0x7f9e5013d340 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9e48007e70 tx=0x7f9e4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e47e00640 1 -- 172.21.15.98:0/1866471256 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48014620 con 0x7f9e501525a0 2024-05-27T00:19:54.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e47e00640 1 -- 172.21.15.98:0/1866471256 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e48019070 con 0x7f9e501525a0 2024-05-27T00:19:54.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e55000640 1 -- 172.21.15.98:0/1866471256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e5013f3b0 con 0x7f9e501525a0 2024-05-27T00:19:54.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e47e00640 1 -- 172.21.15.98:0/1866471256 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48014a20 con 0x7f9e501525a0 2024-05-27T00:19:54.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e55000640 1 -- 172.21.15.98:0/1866471256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e5013f970 con 0x7f9e501525a0 2024-05-27T00:19:54.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e55000640 1 -- 172.21.15.98:0/1866471256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e50075c40 con 0x7f9e501525a0 2024-05-27T00:19:54.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.859+0000 7f9e47e00640 1 -- 172.21.15.98:0/1866471256 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9e48014bc0 con 0x7f9e501525a0 2024-05-27T00:19:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.863+0000 7f9e47e00640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e2c0678c0 0x7f9e2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:19:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.863+0000 7f9e47e00640 1 -- 172.21.15.98:0/1866471256 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7f9e48086740 con 0x7f9e501525a0 2024-05-27T00:19:54.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.863+0000 7f9e4f400640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e2c0678c0 0x7f9e2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:19:54.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.863+0000 7f9e4f400640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e2c0678c0 0x7f9e2c069d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9e400ad000 tx=0x7f9e40008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:19:54.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:54.867+0000 7f9e47e00640 1 -- 172.21.15.98:0/1866471256 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9e48011260 con 0x7f9e501525a0 2024-05-27T00:19:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 -- 172.21.15.98:0/1866471256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e5014d6b0 con 0x7f9e501525a0 2024-05-27T00:19:55.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e47e00640 1 -- 172.21.15.98:0/1866471256 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6014 (secure 0 0 0) 0x7f9e480570f0 con 0x7f9e501525a0 2024-05-27T00:19:55.059 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:19:55.060 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:54.483605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":17,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5073,"qdb_cluster":[5073]},"id":1}]} 2024-05-27T00:19:55.060 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 22 2024-05-27T00:19:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 -- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e2c0678c0 msgr2=0x7f9e2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e2c0678c0 0x7f9e2c069d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9e400ad000 tx=0x7f9e40008040 comp rx=0 tx=0).stop 2024-05-27T00:19:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 -- 172.21.15.98:0/1866471256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 msgr2=0x7f9e5013d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:19:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 0x7f9e5013d340 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9e48007e70 tx=0x7f9e4800e4b0 comp rx=0 tx=0).stop 2024-05-27T00:19:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 -- 172.21.15.98:0/1866471256 shutdown_connections 2024-05-27T00:19:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e501525a0 0x7f9e5013d340 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9e2c0678c0 0x7f9e2c069d80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9e500c90f0 0x7f9e501f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 --2- 172.21.15.98:0/1866471256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9e500c76b0 0x7f9e50145540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:19:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 -- 172.21.15.98:0/1866471256 >> 172.21.15.98:0/1866471256 conn(0x7f9e50067730 msgr2=0x7f9e5014fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:19:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 -- 172.21.15.98:0/1866471256 shutdown_connections 2024-05-27T00:19:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:19:55.055+0000 7f9e55000640 1 -- 172.21.15.98:0/1866471256 wait complete. 2024-05-27T00:19:55.080 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=4894 2024-05-27T00:19:55.081 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.c 2024-05-27T00:20:00.527 INFO:tasks.ceph.mds.a.smithi098.stderr:2024-05-27T00:20:00.523+0000 7fdc98400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:20:01.081 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-05-27T00:20:01.082 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-27T00:20:01.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-27T00:20:01.086 INFO:tasks.ceph.mds.c:Started 2024-05-27T00:20:01.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:20:01.179 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2024-05-27T00:20:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32ca800640 1 Processor -- start 2024-05-27T00:20:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32ca800640 1 -- start start 2024-05-27T00:20:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32ca800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 0x7f32c406f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:20:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32ca800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f32c40c4fd0 0x7f32c40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:20:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32ca800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32c406cfb0 con 0x7f32c40c8fd0 2024-05-27T00:20:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32ca800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f32c406d500 con 0x7f32c40c4fd0 2024-05-27T00:20:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32ca800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f32c406ef50 con 0x7f32c40cdf90 2024-05-27T00:20:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32c3e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f32c40c8fd0 0x7f32c40caa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40656/0 (socket says 172.21.15.98:40656) 2024-05-27T00:20:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32c3e00640 1 -- 172.21.15.98:0/2284770859 learned_addr learned my addr 172.21.15.98:0/2284770859 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:20:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32c3400640 1 --2- 172.21.15.98:0/2284770859 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 0x7f32c406f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:20:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.231+0000 7f32bbe00640 1 --2- 172.21.15.98:0/2284770859 >> v2:172.21.15.204:3300/0 conn(0x7f32c40c4fd0 0x7f32c40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:20:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.235+0000 7f32c2a00640 1 -- 172.21.15.98:0/2284770859 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1263403637 0 0) 0x7f32c406cfb0 con 0x7f32c40c8fd0 2024-05-27T00:20:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.235+0000 7f32c2a00640 1 -- 172.21.15.98:0/2284770859 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32a8003530 con 0x7f32c40c8fd0 2024-05-27T00:20:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.235+0000 7f32c3400640 1 -- 172.21.15.98:0/2284770859 >> v2:172.21.15.204:3300/0 conn(0x7f32c40c4fd0 msgr2=0x7f32c40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:20:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.235+0000 7f32c2a00640 1 -- 172.21.15.98:0/2284770859 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3775925331 0 0) 0x7f32a8003530 con 0x7f32c40c8fd0 2024-05-27T00:20:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.235+0000 7f32c3400640 1 --2- 172.21.15.98:0/2284770859 >> v2:172.21.15.204:3300/0 conn(0x7f32c40c4fd0 0x7f32c40ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:20:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.235+0000 7f32c3400640 1 -- 172.21.15.98:0/2284770859 >> v1:172.21.15.98:6789/0 conn(0x7f32c40c8fd0 legacy=0x7f32c40caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:20:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.235+0000 7f32c3400640 1 -- 172.21.15.98:0/2284770859 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c40ca840 con 0x7f32c40cdf90 2024-05-27T00:20:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.235+0000 7f32bbe00640 1 --2- 172.21.15.98:0/2284770859 >> v2:172.21.15.204:3300/0 conn(0x7f32c40c4fd0 0x7f32c40ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.235+0000 7f32c3400640 1 --2- 172.21.15.98:0/2284770859 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 0x7f32c406f760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f32b40097c0 tx=0x7f32b401ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5caeebe44d7fac4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32c2a00640 1 -- 172.21.15.98:0/2284770859 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b40b5070 con 0x7f32c40cdf90 2024-05-27T00:20:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32c2a00640 1 -- 172.21.15.98:0/2284770859 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32b40a85f0 con 0x7f32c40cdf90 2024-05-27T00:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32c2a00640 1 -- 172.21.15.98:0/2284770859 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32bc001a00 con 0x7f32c40cdf90 2024-05-27T00:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 -- 172.21.15.98:0/2284770859 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 msgr2=0x7f32c406f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 --2- 172.21.15.98:0/2284770859 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 0x7f32c406f760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f32b40097c0 tx=0x7f32b401ebb0 comp rx=0 tx=0).stop 2024-05-27T00:20:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 -- 172.21.15.98:0/2284770859 shutdown_connections 2024-05-27T00:20:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 --2- 172.21.15.98:0/2284770859 >> v2:172.21.15.204:3300/0 conn(0x7f32c40c4fd0 0x7f32c40ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:20:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 --2- 172.21.15.98:0/2284770859 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 0x7f32c406f760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:20:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 -- 172.21.15.98:0/2284770859 >> 172.21.15.98:0/2284770859 conn(0x7f32c406d680 msgr2=0x7f32c406da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:20:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 -- 172.21.15.98:0/2284770859 shutdown_connections 2024-05-27T00:20:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 -- 172.21.15.98:0/2284770859 wait complete. 2024-05-27T00:20:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 Processor -- start 2024-05-27T00:20:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 -- start start 2024-05-27T00:20:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32c40c4fd0 0x7f32c414fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:20:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32c40c8fd0 0x7f32c41504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:20:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 0x7f32c4070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:20:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f32c406d220 con 0x7f32c40c8fd0 2024-05-27T00:20:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f32c406b610 con 0x7f32c40c4fd0 2024-05-27T00:20:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32ca800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f32c406b8b0 con 0x7f32c40cdf90 2024-05-27T00:20:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32bbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32c40c8fd0 0x7f32c41504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:20:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32bbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32c40c8fd0 0x7f32c41504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43008/0 (socket says 172.21.15.98:43008) 2024-05-27T00:20:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32bbe00640 1 -- 172.21.15.98:0/1296488145 learned_addr learned my addr 172.21.15.98:0/1296488145 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:20:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32c3400640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32c40c4fd0 0x7f32c414fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:20:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.239+0000 7f32c3e00640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 0x7f32c4070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:20:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32bbe00640 1 -- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 msgr2=0x7f32c4070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:20:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32bbe00640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 0x7f32c4070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:20:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32bbe00640 1 -- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32c40c4fd0 msgr2=0x7f32c414fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:20:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32bbe00640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32c40c4fd0 0x7f32c414fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:20:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32bbe00640 1 -- 172.21.15.98:0/1296488145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c4150290 con 0x7f32c40c8fd0 2024-05-27T00:20:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32bbe00640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32c40c8fd0 0x7f32c41504a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00e9d0 tx=0x7f32ac00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:20:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32bb400640 1 -- 172.21.15.98:0/1296488145 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac017070 con 0x7f32c40c8fd0 2024-05-27T00:20:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32bb400640 1 -- 172.21.15.98:0/1296488145 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32ac00ca60 con 0x7f32c40c8fd0 2024-05-27T00:20:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32bb400640 1 -- 172.21.15.98:0/1296488145 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac00ce50 con 0x7f32c40c8fd0 2024-05-27T00:20:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32ca800640 1 -- 172.21.15.98:0/1296488145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32c4157060 con 0x7f32c40c8fd0 2024-05-27T00:20:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32ca800640 1 -- 172.21.15.98:0/1296488145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32c41f4150 con 0x7f32c40c8fd0 2024-05-27T00:20:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32ca800640 1 -- 172.21.15.98:0/1296488145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32c4075cb0 con 0x7f32c40c8fd0 2024-05-27T00:20:01.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.243+0000 7f32bb400640 1 -- 172.21.15.98:0/1296488145 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f32ac01a070 con 0x7f32c40c8fd0 2024-05-27T00:20:01.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.247+0000 7f32bb400640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32a0067a90 0x7f32a0069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:20:01.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.247+0000 7f32bb400640 1 -- 172.21.15.98:0/1296488145 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7f32ac085810 con 0x7f32c40c8fd0 2024-05-27T00:20:01.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.247+0000 7f32c3400640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32a0067a90 0x7f32a0069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:20:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.247+0000 7f32c3400640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32a0067a90 0x7f32a0069f50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f32b40b2000 tx=0x7f32b40aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:20:01.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.251+0000 7f32bb400640 1 -- 172.21.15.98:0/1296488145 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f32ac055c10 con 0x7f32c40c8fd0 2024-05-27T00:20:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.611+0000 7f32ca800640 1 -- 172.21.15.98:0/1296488145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32c406f540 con 0x7f32c40c8fd0 2024-05-27T00:20:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.611+0000 7f32bb400640 1 -- 172.21.15.98:0/1296488145 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6839 (secure 0 0 0) 0x7f32ac059f10 con 0x7f32c40c8fd0 2024-05-27T00:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:20:01.616 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5202,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:54.483605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":17,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5073,"qdb_cluster":[5073]},"id":1}]} 2024-05-27T00:20:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-05-27T00:20:01.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 -- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32a0067a90 msgr2=0x7f32a0069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:20:01.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32a0067a90 0x7f32a0069f50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f32b40b2000 tx=0x7f32b40aa070 comp rx=0 tx=0).stop 2024-05-27T00:20:01.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 -- 172.21.15.98:0/1296488145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32c40c8fd0 msgr2=0x7f32c41504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:20:01.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32c40c8fd0 0x7f32c41504a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00e9d0 tx=0x7f32ac00ee90 comp rx=0 tx=0).stop 2024-05-27T00:20:01.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 -- 172.21.15.98:0/1296488145 shutdown_connections 2024-05-27T00:20:01.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32c40cdf90 0x7f32c4070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:20:01.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32c40c8fd0 0x7f32c41504a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:20:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f32a0067a90 0x7f32a0069f50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:20:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 --2- 172.21.15.98:0/1296488145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32c40c4fd0 0x7f32c414fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:20:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 -- 172.21.15.98:0/1296488145 >> 172.21.15.98:0/1296488145 conn(0x7f32c406d680 msgr2=0x7f32c40cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:20:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 -- 172.21.15.98:0/1296488145 shutdown_connections 2024-05-27T00:20:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:20:01.615+0000 7f32ca800640 1 -- 172.21.15.98:0/1296488145 wait complete. 2024-05-27T00:20:01.638 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-05-27T00:20:01.638 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2024-05-27T00:20:15.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:20:15.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:20:21.336 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 6% (827/12563) Updating files: 7% (880/12563) Updating files: 8% (1006/12563) Updating files: 9% (1131/12563) Updating files: 10% (1257/12563) Updating files: 11% (1382/12563) Updating files: 12% (1508/12563) Updating files: 12% (1555/12563) Updating files: 13% (1634/12563) Updating files: 14% (1759/12563) Updating files: 15% (1885/12563) Updating files: 16% (2011/12563) Updating files: 17% (2136/12563) Updating files: 17% (2140/12563) Updating files: 18% (2262/12563) Updating files: 19% (2387/12563) Updating files: 20% (2513/12563) Updating files: 21% (2639/12563) Updating files: 22% (2764/12563) Updating files: 22% (2833/12563) Updating files: 23% (2890/12563) Updating files: 24% (3016/12563) Updating files: 25% (3141/12563) Updating files: 26% (3267/12563) Updating files: 27% (3393/12563) Updating files: 28% (3518/12563) Updating files: 28% (3624/12563) Updating files: 29% (3644/12563) Updating files: 30% (3769/12563) Updating files: 31% (3895/12563) Updating files: 32% (4021/12563) Updating files: 33% (4146/12563) Updating files: 34% (4272/12563) Updating files: 35% (4398/12563) Updating files: 36% (4523/12563) Updating files: 37% (4649/12563) Updating files: 38% (4774/12563) Updating files: 39% (4900/12563) Updating files: 39% (4937/12563) Updating files: 40% (5026/12563) Updating files: 41% (5151/12563) Updating files: 42% (5277/12563) Updating files: 43% (5403/12563) Updating files: 44% (5528/12563) Updating files: 44% (5629/12563) Updating files: 45% (5654/12563) Updating files: 46% (5779/12563) Updating files: 47% (5905/12563) Updating files: 47% (6014/12563) Updating files: 48% (6031/12563) Updating files: 49% (6156/12563) Updating files: 50% (6282/12563) Updating files: 51% (6408/12563) Updating files: 52% (6533/12563) Updating files: 53% (6659/12563) Updating files: 53% (6727/12563) Updating files: 54% (6785/12563) Updating files: 55% (6910/12563) Updating files: 56% (7036/12563) Updating files: 57% (7161/12563) Updating files: 58% (7287/12563) Updating files: 59% (7413/12563) Updating files: 60% (7538/12563) Updating files: 60% (7635/12563) Updating files: 61% (7664/12563) Updating files: 61% (7713/12563) Updating files: 62% (7790/12563) Updating files: 63% (7915/12563) Updating files: 64% (8041/12563) Updating files: 65% (8166/12563) Updating files: 66% (8292/12563) Updating files: 66% (8376/12563) Updating files: 67% (8418/12563) Updating files: 68% (8543/12563) Updating files: 69% (8669/12563) Updating files: 70% (8795/12563) Updating files: 70% (8816/12563) Updating files: 71% (8920/12563) Updating files: 72% (9046/12563) Updating files: 73% (9171/12563) Updating files: 74% (9297/12563) Updating files: 74% (9386/12563) Updating files: 75% (9423/12563) Updating files: 76% (9548/12563) Updating files: 77% (9674/12563) Updating files: 78% (9800/12563) Updating files: 79% (9925/12563) Updating files: 79% (9969/12563) Updating files: 80% (10051/12563) Updating files: 81% (10177/12563) Updating files: 82% (10302/12563) Updating files: 83% (10428/12563) Updating files: 84% (10553/12563) Updating files: 84% (10625/12563) Updating files: 85% (10679/12563) Updating files: 86% (10805/12563) Updating files: 87% (10930/12563) Updating files: 88% (11056/12563) Updating files: 89% (11182/12563) Updating files: 89% (11200/12563) Updating files: 90% (11307/12563) Updating files: 91% (11433/12563) Updating files: 92% (11558/12563) Updating files: 93% (11684/12563) Updating files: 94% (11810/12563) Updating files: 94% (11886/12563) Updating files: 95% (11935/12563) Updating files: 96% (12061/12563) Updating files: 97% (12187/12563) Updating files: 97% (12220/12563) Updating files: 98% (12312/12563) Updating files: 99% (12438/12563) Updating files: 100% (12563/12563) Updating files: 100% (12563/12563), done. 2024-05-27T00:20:21.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ trap - EXIT 2024-05-27T00:20:21.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ cd ceph 2024-05-27T00:20:21.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ seq 1 90 2024-05-27T00:20:21.375 INFO:tasks.workunit.client.0.smithi098.stderr:+ versions=1 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:2 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:3 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:4 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:5 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:6 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:7 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:8 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:9 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:10 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:11 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:12 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:13 2024-05-27T00:20:21.376 INFO:tasks.workunit.client.0.smithi098.stderr:14 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:15 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:16 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:17 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:18 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:19 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:20 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:21 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:22 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:23 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:24 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:25 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:26 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:27 2024-05-27T00:20:21.377 INFO:tasks.workunit.client.0.smithi098.stderr:28 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:29 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:30 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:31 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:32 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:33 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:34 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:35 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:36 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:37 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:38 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:39 2024-05-27T00:20:21.378 INFO:tasks.workunit.client.0.smithi098.stderr:40 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:41 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:42 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:43 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:44 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:45 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:46 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:47 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:48 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:49 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:50 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:51 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:52 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:53 2024-05-27T00:20:21.379 INFO:tasks.workunit.client.0.smithi098.stderr:54 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:55 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:56 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:57 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:58 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:59 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:60 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:61 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:62 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:63 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:64 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:65 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:66 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:67 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:68 2024-05-27T00:20:21.380 INFO:tasks.workunit.client.0.smithi098.stderr:69 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:70 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:71 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:72 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:73 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:74 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:75 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:76 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:77 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:78 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:79 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:80 2024-05-27T00:20:21.381 INFO:tasks.workunit.client.0.smithi098.stderr:81 2024-05-27T00:20:21.382 INFO:tasks.workunit.client.0.smithi098.stderr:82 2024-05-27T00:20:21.382 INFO:tasks.workunit.client.0.smithi098.stderr:83 2024-05-27T00:20:21.382 INFO:tasks.workunit.client.0.smithi098.stderr:84 2024-05-27T00:20:21.382 INFO:tasks.workunit.client.0.smithi098.stderr:85 2024-05-27T00:20:21.382 INFO:tasks.workunit.client.0.smithi098.stderr:86 2024-05-27T00:20:21.382 INFO:tasks.workunit.client.0.smithi098.stderr:87 2024-05-27T00:20:21.382 INFO:tasks.workunit.client.0.smithi098.stderr:88 2024-05-27T00:20:21.382 INFO:tasks.workunit.client.0.smithi098.stderr:89 2024-05-27T00:20:21.382 INFO:tasks.workunit.client.0.smithi098.stderr:90 2024-05-27T00:20:21.382 INFO:tasks.workunit.client.0.smithi098.stdout:v0.1 2024-05-27T00:20:21.383 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 1 -eq 48 ] 2024-05-27T00:20:21.383 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.1 2024-05-27T00:20:21.383 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.1 2024-05-27T00:20:21.383 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.1 2024-05-27T00:20:45.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:20:45.447 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:21:07.621 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 3% (426/13353) Updating files: 4% (535/13353) Updating files: 5% (668/13353) Updating files: 6% (802/13353) Updating files: 6% (821/13353) Updating files: 7% (935/13353) Updating files: 8% (1069/13353) Updating files: 8% (1138/13353) Updating files: 9% (1202/13353) Updating files: 9% (1250/13353) Updating files: 9% (1251/13353) Updating files: 9% (1252/13353) Updating files: 9% (1291/13353) Updating files: 10% (1336/13353) Updating files: 11% (1469/13353) Updating files: 11% (1577/13353) Updating files: 12% (1603/13353) Updating files: 13% (1736/13353) Updating files: 13% (1809/13353) Updating files: 14% (1870/13353) Updating files: 14% (1998/13353) Updating files: 15% (2003/13353) Updating files: 16% (2137/13353) Updating files: 16% (2256/13353) Updating files: 17% (2271/13353) Updating files: 18% (2404/13353) Updating files: 18% (2491/13353) Updating files: 19% (2538/13353) Updating files: 20% (2671/13353) Updating files: 20% (2749/13353) Updating files: 21% (2805/13353) Updating files: 22% (2938/13353) Updating files: 22% (3044/13353) Updating files: 23% (3072/13353) Updating files: 24% (3205/13353) Updating files: 25% (3339/13353) Updating files: 25% (3403/13353) Updating files: 26% (3472/13353) Updating files: 27% (3606/13353) Updating files: 27% (3698/13353) Updating files: 28% (3739/13353) Updating files: 28% (3836/13353) Updating files: 29% (3873/13353) Updating files: 30% (4006/13353) Updating files: 31% (4140/13353) Updating files: 31% (4176/13353) Updating files: 32% (4273/13353) Updating files: 33% (4407/13353) Updating files: 33% (4477/13353) Updating files: 34% (4541/13353) Updating files: 35% (4674/13353) Updating files: 36% (4808/13353) Updating files: 36% (4863/13353) Updating files: 37% (4941/13353) Updating files: 38% (5075/13353) Updating files: 39% (5208/13353) Updating files: 39% (5210/13353) Updating files: 40% (5342/13353) Updating files: 40% (5467/13353) Updating files: 41% (5475/13353) Updating files: 42% (5609/13353) Updating files: 43% (5742/13353) Updating files: 44% (5876/13353) Updating files: 44% (5902/13353) Updating files: 45% (6009/13353) Updating files: 46% (6143/13353) Updating files: 47% (6276/13353) Updating files: 47% (6302/13353) Updating files: 48% (6410/13353) Updating files: 49% (6543/13353) Updating files: 50% (6677/13353) Updating files: 50% (6717/13353) Updating files: 51% (6811/13353) Updating files: 52% (6944/13353) Updating files: 53% (7078/13353) Updating files: 53% (7175/13353) Updating files: 54% (7211/13353) Updating files: 55% (7345/13353) Updating files: 56% (7478/13353) Updating files: 56% (7536/13353) Updating files: 57% (7612/13353) Updating files: 58% (7745/13353) Updating files: 59% (7879/13353) Updating files: 59% (7905/13353) Updating files: 60% (8012/13353) Updating files: 61% (8146/13353) Updating files: 61% (8241/13353) Updating files: 62% (8279/13353) Updating files: 63% (8413/13353) Updating files: 63% (8490/13353) Updating files: 64% (8546/13353) Updating files: 65% (8680/13353) Updating files: 65% (8771/13353) Updating files: 66% (8813/13353) Updating files: 67% (8947/13353) Updating files: 68% (9081/13353) Updating files: 68% (9097/13353) Updating files: 69% (9214/13353) Updating files: 69% (9303/13353) Updating files: 70% (9348/13353) Updating files: 71% (9481/13353) Updating files: 72% (9615/13353) Updating files: 72% (9710/13353) Updating files: 73% (9748/13353) Updating files: 74% (9882/13353) Updating files: 74% (9998/13353) Updating files: 75% (10015/13353) Updating files: 76% (10149/13353) Updating files: 77% (10282/13353) Updating files: 78% (10416/13353) Updating files: 78% (10498/13353) Updating files: 79% (10549/13353) Updating files: 80% (10683/13353) Updating files: 81% (10816/13353) Updating files: 81% (10887/13353) Updating files: 82% (10950/13353) Updating files: 83% (11083/13353) Updating files: 84% (11217/13353) Updating files: 84% (11229/13353) Updating files: 85% (11351/13353) Updating files: 86% (11484/13353) Updating files: 86% (11616/13353) Updating files: 87% (11618/13353) Updating files: 88% (11751/13353) Updating files: 89% (11885/13353) Updating files: 90% (12018/13353) Updating files: 90% (12059/13353) Updating files: 91% (12152/13353) Updating files: 92% (12285/13353) Updating files: 93% (12419/13353) Updating files: 93% (12444/13353) Updating files: 94% (12552/13353) Updating files: 95% (12686/13353) Updating files: 96% (12819/13353) Updating files: 97% (12953/13353) Updating files: 97% (13055/13353) Updating files: 98% (13086/13353) Updating files: 99% (13220/13353) Updating files: 100% (13353/13353) Updating files: 100% (13353/13353), done. 2024-05-27T00:21:07.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:07.657 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2024-05-27T00:21:07.666 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.1 2024-05-27T00:21:07.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 2 -eq 48 ] 2024-05-27T00:21:07.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.2 2024-05-27T00:21:07.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.2 2024-05-27T00:21:07.690 INFO:tasks.workunit.client.0.smithi098.stdout:v0.2 2024-05-27T00:21:07.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.2 2024-05-27T00:21:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.814+0000 7f1d3fa00640 1 Processor -- start 2024-05-27T00:21:07.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.814+0000 7f1d3fa00640 1 -- start start 2024-05-27T00:21:07.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.818+0000 7f1d3fa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1d380cbe10 0x7f1d381592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.818+0000 7f1d3fa00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d380c1f60 con 0x7f1d38073570 2024-05-27T00:21:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.818+0000 7f1d3fa00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d380c6dd0 con 0x7f1d380cc1f0 2024-05-27T00:21:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.818+0000 7f1d3fa00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1d38159810 con 0x7f1d380cbe10 2024-05-27T00:21:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.818+0000 7f1d3d000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1d380cbe10 0x7f1d381592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.818+0000 7f1d3d000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1d380cbe10 0x7f1d381592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42602/0 (socket says 172.21.15.98:42602) 2024-05-27T00:21:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.818+0000 7f1d3d000640 1 -- 172.21.15.98:0/188340810 learned_addr learned my addr 172.21.15.98:0/188340810 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:07.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.818+0000 7f1d3d000640 1 -- 172.21.15.98:0/188340810 >> v1:172.21.15.204:6789/0 conn(0x7f1d380cc1f0 legacy=0x7f1d380c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.818+0000 7f1d3d000640 1 -- 172.21.15.98:0/188340810 >> v1:172.21.15.98:6789/0 conn(0x7f1d38073570 legacy=0x7f1d381559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:07.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.818+0000 7f1d3d000640 1 -- 172.21.15.98:0/188340810 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d380c70b0 con 0x7f1d380cbe10 2024-05-27T00:21:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d3d000640 1 --2- 172.21.15.98:0/188340810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d380cbe10 0x7f1d381592d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009870 tx=0x7f1d2c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a140223fee131772 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d37400640 1 -- 172.21.15.98:0/188340810 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0b5070 con 0x7f1d380cbe10 2024-05-27T00:21:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d37400640 1 -- 172.21.15.98:0/188340810 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d2c0a8a10 con 0x7f1d380cbe10 2024-05-27T00:21:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d37400640 1 -- 172.21.15.98:0/188340810 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0a8de0 con 0x7f1d380cbe10 2024-05-27T00:21:07.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d3fa00640 1 -- 172.21.15.98:0/188340810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d380cbe10 msgr2=0x7f1d381592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d3fa00640 1 --2- 172.21.15.98:0/188340810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d380cbe10 0x7f1d381592d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009870 tx=0x7f1d2c01ed10 comp rx=0 tx=0).stop 2024-05-27T00:21:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d3fa00640 1 -- 172.21.15.98:0/188340810 shutdown_connections 2024-05-27T00:21:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d3fa00640 1 --2- 172.21.15.98:0/188340810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d380cbe10 0x7f1d381592d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d3fa00640 1 -- 172.21.15.98:0/188340810 >> 172.21.15.98:0/188340810 conn(0x7f1d38067530 msgr2=0x7f1d3806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d3fa00640 1 -- 172.21.15.98:0/188340810 shutdown_connections 2024-05-27T00:21:07.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d3fa00640 1 -- 172.21.15.98:0/188340810 wait complete. 2024-05-27T00:21:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d3fa00640 1 Processor -- start 2024-05-27T00:21:07.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.822+0000 7f1d3fa00640 1 -- start start 2024-05-27T00:21:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3fa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d38073570 0x7f1d380b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3fa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d380cbe10 0x7f1d381565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3fa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d380cc1f0 0x7f1d38157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3fa00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1d380c1330 con 0x7f1d38073570 2024-05-27T00:21:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3fa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1d380c1050 con 0x7f1d380cbe10 2024-05-27T00:21:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3fa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1d380c79d0 con 0x7f1d380cc1f0 2024-05-27T00:21:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d38073570 0x7f1d380b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d38073570 0x7f1d380b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42608/0 (socket says 172.21.15.98:42608) 2024-05-27T00:21:07.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3d000640 1 -- 172.21.15.98:0/1908880192 learned_addr learned my addr 172.21.15.98:0/1908880192 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d37e00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d380cbe10 0x7f1d381565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3da00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d380cc1f0 0x7f1d38157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3d000640 1 -- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d380cc1f0 msgr2=0x7f1d38157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3d000640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d380cc1f0 0x7f1d38157220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3d000640 1 -- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d380cbe10 msgr2=0x7f1d381565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3d000640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d380cbe10 0x7f1d381565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3d000640 1 -- 172.21.15.98:0/1908880192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d380c4850 con 0x7f1d38073570 2024-05-27T00:21:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d37e00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d380cbe10 0x7f1d381565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:21:07.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3da00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d380cc1f0 0x7f1d38157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:21:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3d000640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d38073570 0x7f1d380b0d10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0b3040 tx=0x7f1d2c01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d34c00640 1 -- 172.21.15.98:0/1908880192 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0c0070 con 0x7f1d38073570 2024-05-27T00:21:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d34c00640 1 -- 172.21.15.98:0/1908880192 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d2c0b0b80 con 0x7f1d38073570 2024-05-27T00:21:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d34c00640 1 -- 172.21.15.98:0/1908880192 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0b5070 con 0x7f1d38073570 2024-05-27T00:21:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3fa00640 1 -- 172.21.15.98:0/1908880192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d38155480 con 0x7f1d38073570 2024-05-27T00:21:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3fa00640 1 -- 172.21.15.98:0/1908880192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d380c4dd0 con 0x7f1d38073570 2024-05-27T00:21:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.826+0000 7f1d3fa00640 1 -- 172.21.15.98:0/1908880192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d38075cb0 con 0x7f1d38073570 2024-05-27T00:21:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.830+0000 7f1d34c00640 1 -- 172.21.15.98:0/1908880192 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1d2c0ba3e0 con 0x7f1d38073570 2024-05-27T00:21:07.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.830+0000 7f1d34c00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1d180678c0 0x7f1d18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.830+0000 7f1d34c00640 1 -- 172.21.15.98:0/1908880192 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7f1d2c1232e0 con 0x7f1d38073570 2024-05-27T00:21:07.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.830+0000 7f1d37e00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1d180678c0 0x7f1d18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:07.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.830+0000 7f1d37e00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1d180678c0 0x7f1d18069d80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1d3806ec20 tx=0x7f1d24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:07.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:07.834+0000 7f1d34c00640 1 -- 172.21.15.98:0/1908880192 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1d2c018870 con 0x7f1d38073570 2024-05-27T00:21:08.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.050+0000 7f1d3fa00640 1 -- 172.21.15.98:0/1908880192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d380c6630 con 0x7f1d38073570 2024-05-27T00:21:08.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.050+0000 7f1d34c00640 1 -- 172.21.15.98:0/1908880192 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6839 (secure 0 0 0) 0x7f1d2c0f3b70 con 0x7f1d38073570 2024-05-27T00:21:08.057 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:08.057 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5202,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:54.483605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":17,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5073,"qdb_cluster":[5073]},"id":1}]} 2024-05-27T00:21:08.057 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-05-27T00:21:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 -- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1d180678c0 msgr2=0x7f1d18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1d180678c0 0x7f1d18069d80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1d3806ec20 tx=0x7f1d24008040 comp rx=0 tx=0).stop 2024-05-27T00:21:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 -- 172.21.15.98:0/1908880192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d38073570 msgr2=0x7f1d380b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d38073570 0x7f1d380b0d10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0b3040 tx=0x7f1d2c01bab0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 -- 172.21.15.98:0/1908880192 shutdown_connections 2024-05-27T00:21:08.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d380cc1f0 0x7f1d38157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1d180678c0 0x7f1d18069d80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d380cbe10 0x7f1d381565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 --2- 172.21.15.98:0/1908880192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d38073570 0x7f1d380b0d10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 -- 172.21.15.98:0/1908880192 >> 172.21.15.98:0/1908880192 conn(0x7f1d38067530 msgr2=0x7f1d380c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:08.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 -- 172.21.15.98:0/1908880192 shutdown_connections 2024-05-27T00:21:08.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.054+0000 7f1d3fa00640 1 -- 172.21.15.98:0/1908880192 wait complete. 2024-05-27T00:21:08.080 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2024-05-27T00:21:08.080 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-27T00:21:08.080 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:21:08.144 INFO:tasks.ceph.mds.a:Stopped 2024-05-27T00:21:08.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:08.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.314+0000 7fd0e2a00640 1 Processor -- start 2024-05-27T00:21:08.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.314+0000 7fd0e2a00640 1 -- start start 2024-05-27T00:21:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc157cd0 0x7fd0dc0c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e2a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd0dc0b3cb0 0x7fd0dc0c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e2a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0dc0c59d0 con 0x7fd0dc1580b0 2024-05-27T00:21:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e2a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd0dc0c51e0 con 0x7fd0dc0b3cb0 2024-05-27T00:21:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e2a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd0dc0cabb0 con 0x7fd0dc157cd0 2024-05-27T00:21:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e0c00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd0dc1580b0 0x7fd0dc15e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58806/0 (socket says 172.21.15.98:58806) 2024-05-27T00:21:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e0c00640 1 -- 172.21.15.98:0/411234644 learned_addr learned my addr 172.21.15.98:0/411234644 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e2000640 1 --2- 172.21.15.98:0/411234644 >> v2:172.21.15.98:3300/0 conn(0x7fd0dc0b3cb0 0x7fd0dc0c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e1600640 1 --2- 172.21.15.98:0/411234644 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc157cd0 0x7fd0dc0c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0dbe00640 1 -- 172.21.15.98:0/411234644 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157808185 0 0) 0x7fd0dc0c59d0 con 0x7fd0dc1580b0 2024-05-27T00:21:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0dbe00640 1 -- 172.21.15.98:0/411234644 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0c4003530 con 0x7fd0dc1580b0 2024-05-27T00:21:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e2000640 1 -- 172.21.15.98:0/411234644 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc157cd0 msgr2=0x7fd0dc0c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e2000640 1 --2- 172.21.15.98:0/411234644 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc157cd0 0x7fd0dc0c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e2000640 1 -- 172.21.15.98:0/411234644 >> v1:172.21.15.98:6789/0 conn(0x7fd0dc1580b0 legacy=0x7fd0dc15e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e2000640 1 -- 172.21.15.98:0/411234644 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc0cae50 con 0x7fd0dc0b3cb0 2024-05-27T00:21:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.318+0000 7fd0e1600640 1 --2- 172.21.15.98:0/411234644 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc157cd0 0x7fd0dc0c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2000640 1 --2- 172.21.15.98:0/411234644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc0b3cb0 0x7fd0dc0c9a50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009d90 tx=0x7fd0d401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a10121861e9799f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0dbe00640 1 -- 172.21.15.98:0/411234644 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40b2070 con 0x7fd0dc0b3cb0 2024-05-27T00:21:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0dbe00640 1 -- 172.21.15.98:0/411234644 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d40a5830 con 0x7fd0dc0b3cb0 2024-05-27T00:21:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0dbe00640 1 -- 172.21.15.98:0/411234644 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40a5c00 con 0x7fd0dc0b3cb0 2024-05-27T00:21:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 -- 172.21.15.98:0/411234644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc0b3cb0 msgr2=0x7fd0dc0c9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 --2- 172.21.15.98:0/411234644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc0b3cb0 0x7fd0dc0c9a50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009d90 tx=0x7fd0d401c970 comp rx=0 tx=0).stop 2024-05-27T00:21:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 -- 172.21.15.98:0/411234644 shutdown_connections 2024-05-27T00:21:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 --2- 172.21.15.98:0/411234644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc0b3cb0 0x7fd0dc0c9a50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 --2- 172.21.15.98:0/411234644 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc157cd0 0x7fd0dc0c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 -- 172.21.15.98:0/411234644 >> 172.21.15.98:0/411234644 conn(0x7fd0dc070b30 msgr2=0x7fd0dc070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 -- 172.21.15.98:0/411234644 shutdown_connections 2024-05-27T00:21:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 -- 172.21.15.98:0/411234644 wait complete. 2024-05-27T00:21:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 Processor -- start 2024-05-27T00:21:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 -- start start 2024-05-27T00:21:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc0b3cb0 0x7fd0dc156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc157cd0 0x7fd0dc1495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.322+0000 7fd0e2a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0dc1580b0 0x7fd0dc1f6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e2a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc15a830 con 0x7fd0dc157cd0 2024-05-27T00:21:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e2a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc0cb410 con 0x7fd0dc1580b0 2024-05-27T00:21:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e2a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd0dc15a9b0 con 0x7fd0dc0b3cb0 2024-05-27T00:21:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e0c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc157cd0 0x7fd0dc1495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e1600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc0b3cb0 0x7fd0dc156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e0c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc157cd0 0x7fd0dc1495b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42636/0 (socket says 172.21.15.98:42636) 2024-05-27T00:21:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e0c00640 1 -- 172.21.15.98:0/2884714697 learned_addr learned my addr 172.21.15.98:0/2884714697 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e2000640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0dc1580b0 0x7fd0dc1f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e0c00640 1 -- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc0b3cb0 msgr2=0x7fd0dc156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e0c00640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc0b3cb0 0x7fd0dc156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e0c00640 1 -- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0dc1580b0 msgr2=0x7fd0dc1f6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e0c00640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0dc1580b0 0x7fd0dc1f6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e0c00640 1 -- 172.21.15.98:0/2884714697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc1527f0 con 0x7fd0dc157cd0 2024-05-27T00:21:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e0c00640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc157cd0 0x7fd0dc1495b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800c3c0 tx=0x7fd0c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0d9600640 1 -- 172.21.15.98:0/2884714697 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8017070 con 0x7fd0dc157cd0 2024-05-27T00:21:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0d9600640 1 -- 172.21.15.98:0/2884714697 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0c8005cf0 con 0x7fd0dc157cd0 2024-05-27T00:21:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0d9600640 1 -- 172.21.15.98:0/2884714697 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80072f0 con 0x7fd0dc157cd0 2024-05-27T00:21:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e2a00640 1 -- 172.21.15.98:0/2884714697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0dc155b80 con 0x7fd0dc157cd0 2024-05-27T00:21:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e2a00640 1 -- 172.21.15.98:0/2884714697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0dc1540e0 con 0x7fd0dc157cd0 2024-05-27T00:21:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0e2a00640 1 -- 172.21.15.98:0/2884714697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0dc075c40 con 0x7fd0dc157cd0 2024-05-27T00:21:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.326+0000 7fd0d9600640 1 -- 172.21.15.98:0/2884714697 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd0c801a070 con 0x7fd0dc157cd0 2024-05-27T00:21:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.330+0000 7fd0d9600640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0bc0678c0 0x7fd0bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.330+0000 7fd0d9600640 1 -- 172.21.15.98:0/2884714697 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7fd0c8085120 con 0x7fd0dc157cd0 2024-05-27T00:21:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.330+0000 7fd0e1600640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0bc0678c0 0x7fd0bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.330+0000 7fd0e1600640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0bc0678c0 0x7fd0bc069d80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009950 tx=0x7fd0d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.334+0000 7fd0d9600640 1 -- 172.21.15.98:0/2884714697 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd0c8055480 con 0x7fd0dc157cd0 2024-05-27T00:21:08.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0e2a00640 1 -- 172.21.15.98:0/2884714697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0dc0ca570 con 0x7fd0dc157cd0 2024-05-27T00:21:08.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0d9600640 1 -- 172.21.15.98:0/2884714697 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6839 (secure 0 0 0) 0x7fd0c8059780 con 0x7fd0dc157cd0 2024-05-27T00:21:08.527 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:08.527 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5202,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:54.483605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":17,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5073,"qdb_cluster":[5073]},"id":1}]} 2024-05-27T00:21:08.528 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-05-27T00:21:08.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0e2a00640 1 -- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0bc0678c0 msgr2=0x7fd0bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0e2a00640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0bc0678c0 0x7fd0bc069d80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009950 tx=0x7fd0d0008040 comp rx=0 tx=0).stop 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0e2a00640 1 -- 172.21.15.98:0/2884714697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc157cd0 msgr2=0x7fd0dc1495b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0e2a00640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc157cd0 0x7fd0dc1495b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800c3c0 tx=0x7fd0c800c880 comp rx=0 tx=0).stop 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0e2a00640 1 -- 172.21.15.98:0/2884714697 shutdown_connections 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0e2a00640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0bc0678c0 0x7fd0bc069d80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0e2a00640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0dc1580b0 0x7fd0dc1f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0e2a00640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0dc157cd0 0x7fd0dc1495b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.522+0000 7fd0e2a00640 1 --2- 172.21.15.98:0/2884714697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0dc0b3cb0 0x7fd0dc156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.526+0000 7fd0e2a00640 1 -- 172.21.15.98:0/2884714697 >> 172.21.15.98:0/2884714697 conn(0x7fd0dc070b30 msgr2=0x7fd0dc158d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.526+0000 7fd0e2a00640 1 -- 172.21.15.98:0/2884714697 shutdown_connections 2024-05-27T00:21:08.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:08.526+0000 7fd0e2a00640 1 -- 172.21.15.98:0/2884714697 wait complete. 2024-05-27T00:21:08.549 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:21:10.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:10.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.698+0000 7fa9da800640 1 Processor -- start 2024-05-27T00:21:10.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.698+0000 7fa9da800640 1 -- start start 2024-05-27T00:21:10.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.698+0000 7fa9da800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9d40b1a40 0x7fa9d40c2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:10.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.698+0000 7fa9da800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9d416b5b0 con 0x7fa9d40c0eb0 2024-05-27T00:21:10.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.698+0000 7fa9da800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9d416b890 con 0x7fa9d40b1e20 2024-05-27T00:21:10.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.698+0000 7fa9da800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa9d40c1900 con 0x7fa9d40b1a40 2024-05-27T00:21:10.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.698+0000 7fa9d3e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa9d40c0eb0 0x7fa9d40c68c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58822/0 (socket says 172.21.15.98:58822) 2024-05-27T00:21:10.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.698+0000 7fa9d3e00640 1 -- 172.21.15.98:0/1403102693 learned_addr learned my addr 172.21.15.98:0/1403102693 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:10.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.698+0000 7fa9d3400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9d40b1a40 0x7fa9d40c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.698+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1242611164 0 0) 0x7fa9d416b5b0 con 0x7fa9d40c0eb0 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.914+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9b8003510 con 0x7fa9d40c0eb0 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.914+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1173897465 0 0) 0x7fa9d416b890 con 0x7fa9d40b1e20 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.914+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9d416b5b0 con 0x7fa9d40b1e20 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.918+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 239591433 0 0) 0x7fa9b8003510 con 0x7fa9d40c0eb0 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 >> v2:172.21.15.98:3300/0 conn(0x7fa9d40b1a40 msgr2=0x7fa9d40c2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 --2- 172.21.15.98:0/1403102693 >> v2:172.21.15.98:3300/0 conn(0x7fa9d40b1a40 0x7fa9d40c2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 >> v1:172.21.15.204:6789/0 conn(0x7fa9d40b1e20 legacy=0x7fa9d416aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d40c1b50 con 0x7fa9d40c0eb0 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fa9cc001a00 con 0x7fa9d40c0eb0 2024-05-27T00:21:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 >> v1:172.21.15.98:6789/0 conn(0x7fa9d40c0eb0 legacy=0x7fa9d40c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 --2- 172.21.15.98:0/1403102693 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9b8003ce0 0x7fa9b80040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 --2- 172.21.15.98:0/1403102693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9b8004650 0x7fa9b8008e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 --2- 172.21.15.98:0/1403102693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8009350 0x7fa9b800b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d40c1900 con 0x7fa9b8009350 2024-05-27T00:21:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b800c000 con 0x7fa9b8004650 2024-05-27T00:21:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b800c200 con 0x7fa9b8003ce0 2024-05-27T00:21:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.922+0000 7fa9d2a00640 1 --2- 172.21.15.98:0/1403102693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9b8004650 0x7fa9b8008e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.926+0000 7fa9d2a00640 1 -- 172.21.15.98:0/1403102693 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9b8003ce0 msgr2=0x7fa9b80040c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T00:21:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.926+0000 7fa9d2a00640 1 --2- 172.21.15.98:0/1403102693 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9b8003ce0 0x7fa9b80040c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.926+0000 7fa9d2a00640 1 -- 172.21.15.98:0/1403102693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8009350 msgr2=0x7fa9b800b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.926+0000 7fa9d2a00640 1 --2- 172.21.15.98:0/1403102693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8009350 0x7fa9b800b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.926+0000 7fa9d2a00640 1 -- 172.21.15.98:0/1403102693 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d40c1b50 con 0x7fa9b8004650 2024-05-27T00:21:10.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.926+0000 7fa9d3e00640 1 -- 172.21.15.98:0/1403102693 reap_dead start 2024-05-27T00:21:10.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.938+0000 7fa9d2a00640 1 --2- 172.21.15.98:0/1403102693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9b8004650 0x7fa9b8008e10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40b2200 tx=0x7fa9c001c5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c42ba8598bc7d015 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:10.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.942+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0001570 con 0x7fa9b8004650 2024-05-27T00:21:10.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.942+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9c00af070 con 0x7fa9b8004650 2024-05-27T00:21:10.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.942+0000 7fa9d2000640 1 -- 172.21.15.98:0/1403102693 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c00aa8a0 con 0x7fa9b8004650 2024-05-27T00:21:10.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.942+0000 7fa9da800640 1 -- 172.21.15.98:0/1403102693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9b8004650 msgr2=0x7fa9b8008e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:10.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.942+0000 7fa9da800640 1 --2- 172.21.15.98:0/1403102693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9b8004650 0x7fa9b8008e10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40b2200 tx=0x7fa9c001c5e0 comp rx=0 tx=0).stop 2024-05-27T00:21:10.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.942+0000 7fa9da800640 1 -- 172.21.15.98:0/1403102693 shutdown_connections 2024-05-27T00:21:10.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.942+0000 7fa9da800640 1 --2- 172.21.15.98:0/1403102693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9b8004650 0x7fa9b8008e10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:10.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.942+0000 7fa9da800640 1 -- 172.21.15.98:0/1403102693 >> 172.21.15.98:0/1403102693 conn(0x7fa9d4067530 msgr2=0x7fa9d406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:10.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.946+0000 7fa9da800640 1 -- 172.21.15.98:0/1403102693 shutdown_connections 2024-05-27T00:21:10.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.946+0000 7fa9da800640 1 -- 172.21.15.98:0/1403102693 wait complete. 2024-05-27T00:21:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.958+0000 7fa9da800640 1 Processor -- start 2024-05-27T00:21:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.958+0000 7fa9da800640 1 -- start start 2024-05-27T00:21:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.958+0000 7fa9da800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8004650 0x7fa9d4072380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.958+0000 7fa9da800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9d4200310 0x7fa9d4160080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.958+0000 7fa9da800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9d41605c0 0x7fa9d4167c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9da800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d40c2630 con 0x7fa9b8004650 2024-05-27T00:21:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9da800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d40c2110 con 0x7fa9d41605c0 2024-05-27T00:21:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9da800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d40c27b0 con 0x7fa9d4200310 2024-05-27T00:21:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9d41605c0 0x7fa9d4167c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9d41605c0 0x7fa9d4167c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:52906/0 (socket says 172.21.15.98:52906) 2024-05-27T00:21:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d3e00640 1 -- 172.21.15.98:0/4092616686 learned_addr learned my addr 172.21.15.98:0/4092616686 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9d4200310 0x7fa9d4160080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d3400640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8004650 0x7fa9d4072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d2a00640 1 -- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9d41605c0 msgr2=0x7fa9d4167c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d2a00640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9d41605c0 0x7fa9d4167c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d2a00640 1 -- 172.21.15.98:0/4092616686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8004650 msgr2=0x7fa9d4072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d2a00640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8004650 0x7fa9d4072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:10.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d2a00640 1 -- 172.21.15.98:0/4092616686 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d4160a40 con 0x7fa9d4200310 2024-05-27T00:21:10.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.962+0000 7fa9d2a00640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9d4200310 0x7fa9d4160080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa9b800c1d0 tx=0x7fa9c0019380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.966+0000 7fa9cb400640 1 -- 172.21.15.98:0/4092616686 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0007220 con 0x7fa9d4200310 2024-05-27T00:21:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.966+0000 7fa9cb400640 1 -- 172.21.15.98:0/4092616686 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9c00073e0 con 0x7fa9d4200310 2024-05-27T00:21:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.966+0000 7fa9cb400640 1 -- 172.21.15.98:0/4092616686 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0001550 con 0x7fa9d4200310 2024-05-27T00:21:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.966+0000 7fa9da800640 1 -- 172.21.15.98:0/4092616686 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d41fd0b0 con 0x7fa9d4200310 2024-05-27T00:21:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.966+0000 7fa9da800640 1 -- 172.21.15.98:0/4092616686 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d41fd650 con 0x7fa9d4200310 2024-05-27T00:21:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.970+0000 7fa9da800640 1 -- 172.21.15.98:0/4092616686 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9ac0052a0 con 0x7fa9d4200310 2024-05-27T00:21:10.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.974+0000 7fa9cb400640 1 -- 172.21.15.98:0/4092616686 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9c00bc030 con 0x7fa9d4200310 2024-05-27T00:21:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.974+0000 7fa9cb400640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9b40678f0 0x7fa9b4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.974+0000 7fa9cb400640 1 -- 172.21.15.98:0/4092616686 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7fa9c0120230 con 0x7fa9d4200310 2024-05-27T00:21:10.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.978+0000 7fa9d3400640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9b40678f0 0x7fa9b4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.982+0000 7fa9d3400640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9b40678f0 0x7fa9b4069db0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4069d30 tx=0x7fa9c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:10.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:10.990+0000 7fa9cb400640 1 -- 172.21.15.98:0/4092616686 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa9c00c3030 con 0x7fa9d4200310 2024-05-27T00:21:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.250+0000 7fa9da800640 1 -- 172.21.15.98:0/4092616686 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9ac005590 con 0x7fa9d4200310 2024-05-27T00:21:11.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9cb400640 1 -- 172.21.15.98:0/4092616686 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6839 (secure 0 0 0) 0x7fa9c00162f0 con 0x7fa9d4200310 2024-05-27T00:21:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:11.258 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5202,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:54.483605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":17,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5073,"qdb_cluster":[5073]},"id":1}]} 2024-05-27T00:21:11.259 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-05-27T00:21:11.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 -- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9b40678f0 msgr2=0x7fa9b4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:11.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9b40678f0 0x7fa9b4069db0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4069d30 tx=0x7fa9c4008040 comp rx=0 tx=0).stop 2024-05-27T00:21:11.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 -- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9d4200310 msgr2=0x7fa9d4160080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:11.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9d4200310 0x7fa9d4160080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa9b800c1d0 tx=0x7fa9c0019380 comp rx=0 tx=0).stop 2024-05-27T00:21:11.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 -- 172.21.15.98:0/4092616686 shutdown_connections 2024-05-27T00:21:11.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9b40678f0 0x7fa9b4069db0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:11.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9d41605c0 0x7fa9d4167c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:11.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9d4200310 0x7fa9d4160080 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:11.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 --2- 172.21.15.98:0/4092616686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8004650 0x7fa9d4072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:11.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 -- 172.21.15.98:0/4092616686 >> 172.21.15.98:0/4092616686 conn(0x7fa9d4067530 msgr2=0x7fa9d40c8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:11.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 -- 172.21.15.98:0/4092616686 shutdown_connections 2024-05-27T00:21:11.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:11.254+0000 7fa9da800640 1 -- 172.21.15.98:0/4092616686 wait complete. 2024-05-27T00:21:11.280 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:21:13.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.422+0000 7f48ad800640 1 Processor -- start 2024-05-27T00:21:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.422+0000 7f48ad800640 1 -- start start 2024-05-27T00:21:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.422+0000 7f48ad800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80cc1f0 0x7f48a81592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.422+0000 7f48ad800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f48a8073570 0x7f48a81559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.422+0000 7f48ad800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a80c1f60 con 0x7f48a80cbe10 2024-05-27T00:21:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.422+0000 7f48ad800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f48a8159810 con 0x7f48a8073570 2024-05-27T00:21:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.422+0000 7f48ad800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f48a80c6dd0 con 0x7f48a80cc1f0 2024-05-27T00:21:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.422+0000 7f48a6a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f48a80cbe10 0x7f48a80c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49834/0 (socket says 172.21.15.98:49834) 2024-05-27T00:21:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.422+0000 7f48a6a00640 1 -- 172.21.15.98:0/2754330689 learned_addr learned my addr 172.21.15.98:0/2754330689 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.422+0000 7f489fe00640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80cc1f0 0x7f48a81592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a7400640 1 --2- 172.21.15.98:0/2754330689 >> v2:172.21.15.204:3300/0 conn(0x7f48a8073570 0x7f48a81559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455991591 0 0) 0x7f48a80c1f60 con 0x7f48a80cbe10 2024-05-27T00:21:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f488c003530 con 0x7f48a80cbe10 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 625516440 0 0) 0x7f488c003530 con 0x7f48a80cbe10 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80cc1f0 msgr2=0x7f48a81592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80cc1f0 0x7f48a81592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 >> v2:172.21.15.204:3300/0 conn(0x7f48a8073570 msgr2=0x7f48a81559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 --2- 172.21.15.98:0/2754330689 >> v2:172.21.15.204:3300/0 conn(0x7f48a8073570 0x7f48a81559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a80c70b0 con 0x7f48a80cbe10 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f4898001680 con 0x7f48a80cbe10 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 >> v1:172.21.15.98:6789/0 conn(0x7f48a80cbe10 legacy=0x7f48a80c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 0x7f488c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80cc1f0 0x7f488c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c004d50 0x7f488c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48a8159810 con 0x7f488c0039c0 2024-05-27T00:21:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f48a80c6dd0 con 0x7f488c004d50 2024-05-27T00:21:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f488c00b9d0 con 0x7f48a80cc1f0 2024-05-27T00:21:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6a00640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 0x7f488c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f489fe00640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80cc1f0 0x7f488c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a7400640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c004d50 0x7f488c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6a00640 1 -- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80cc1f0 msgr2=0x7f488c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6a00640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a80cc1f0 0x7f488c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6a00640 1 -- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c004d50 msgr2=0x7f488c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6a00640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f488c004d50 0x7f488c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.426+0000 7f48a6a00640 1 -- 172.21.15.98:0/2754330689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c003530 con 0x7f488c0039c0 2024-05-27T00:21:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48a6a00640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 0x7f488c003de0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f489800c2c0 tx=0x7f489801f4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a62d6a0b481f90 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4898007d60 con 0x7f488c0039c0 2024-05-27T00:21:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4898005b00 con 0x7f488c0039c0 2024-05-27T00:21:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48a6000640 1 -- 172.21.15.98:0/2754330689 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4898007070 con 0x7f488c0039c0 2024-05-27T00:21:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 -- 172.21.15.98:0/2754330689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 msgr2=0x7f488c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 --2- 172.21.15.98:0/2754330689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 0x7f488c003de0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f489800c2c0 tx=0x7f489801f4b0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48a7400640 1 -- 172.21.15.98:0/2754330689 reap_dead start 2024-05-27T00:21:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 -- 172.21.15.98:0/2754330689 shutdown_connections 2024-05-27T00:21:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 -- 172.21.15.98:0/2754330689 >> 172.21.15.98:0/2754330689 conn(0x7f48a8067530 msgr2=0x7f48a806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 -- 172.21.15.98:0/2754330689 shutdown_connections 2024-05-27T00:21:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 -- 172.21.15.98:0/2754330689 wait complete. 2024-05-27T00:21:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 Processor -- start 2024-05-27T00:21:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 -- start start 2024-05-27T00:21:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 0x7f48a8153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48a815c840 0x7f48a8157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a814fa70 0x7f48a8157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48a80c1330 con 0x7f488c0039c0 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f48a80c1050 con 0x7f48a815c840 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48ad800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f48a80c79d0 con 0x7f48a814fa70 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.430+0000 7f48a6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 0x7f48a8153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48a6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 0x7f48a8153f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60424/0 (socket says 172.21.15.98:60424) 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48a6a00640 1 -- 172.21.15.98:0/868175265 learned_addr learned my addr 172.21.15.98:0/868175265 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48a7400640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a814fa70 0x7f48a8157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f489fe00640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48a815c840 0x7f48a8157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48a6a00640 1 -- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a814fa70 msgr2=0x7f48a8157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48a6a00640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a814fa70 0x7f48a8157db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48a6a00640 1 -- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48a815c840 msgr2=0x7f48a8157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48a6a00640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48a815c840 0x7f48a8157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48a6a00640 1 -- 172.21.15.98:0/868175265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a81568a0 con 0x7f488c0039c0 2024-05-27T00:21:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48a7400640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a814fa70 0x7f48a8157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48a6a00640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 0x7f48a8153f00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f489800c2c0 tx=0x7f488c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f489ea00640 1 -- 172.21.15.98:0/868175265 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4898005cb0 con 0x7f488c0039c0 2024-05-27T00:21:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f489ea00640 1 -- 172.21.15.98:0/868175265 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4898005e50 con 0x7f488c0039c0 2024-05-27T00:21:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48ad800640 1 -- 172.21.15.98:0/868175265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a81ee080 con 0x7f488c0039c0 2024-05-27T00:21:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f489ea00640 1 -- 172.21.15.98:0/868175265 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4898004a30 con 0x7f488c0039c0 2024-05-27T00:21:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48ad800640 1 -- 172.21.15.98:0/868175265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a81ee640 con 0x7f488c0039c0 2024-05-27T00:21:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f48ad800640 1 -- 172.21.15.98:0/868175265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a8075cb0 con 0x7f488c0039c0 2024-05-27T00:21:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f489ea00640 1 -- 172.21.15.98:0/868175265 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4898004bd0 con 0x7f488c0039c0 2024-05-27T00:21:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f489ea00640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48840678f0 0x7f4884069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.434+0000 7f489ea00640 1 -- 172.21.15.98:0/868175265 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7f4898122c20 con 0x7f488c0039c0 2024-05-27T00:21:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.438+0000 7f489fe00640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48840678f0 0x7f4884069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.438+0000 7f489fe00640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48840678f0 0x7f4884069db0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f48a80c8720 tx=0x7f4890008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.442+0000 7f489ea00640 1 -- 172.21.15.98:0/868175265 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f48980c3030 con 0x7f488c0039c0 2024-05-27T00:21:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 -- 172.21.15.98:0/868175265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48a81574d0 con 0x7f488c0039c0 2024-05-27T00:21:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f489ea00640 1 -- 172.21.15.98:0/868175265 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6839 (secure 0 0 0) 0x7f48980191c0 con 0x7f488c0039c0 2024-05-27T00:21:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:13.638 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5202,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:54.483605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":17,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5073,"qdb_cluster":[5073]},"id":1}]} 2024-05-27T00:21:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-05-27T00:21:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 -- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48840678f0 msgr2=0x7f4884069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48840678f0 0x7f4884069db0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f48a80c8720 tx=0x7f4890008040 comp rx=0 tx=0).stop 2024-05-27T00:21:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 -- 172.21.15.98:0/868175265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 msgr2=0x7f48a8153f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 0x7f48a8153f00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f489800c2c0 tx=0x7f488c005840 comp rx=0 tx=0).stop 2024-05-27T00:21:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 -- 172.21.15.98:0/868175265 shutdown_connections 2024-05-27T00:21:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48a814fa70 0x7f48a8157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48840678f0 0x7f4884069db0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48a815c840 0x7f48a8157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 --2- 172.21.15.98:0/868175265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488c0039c0 0x7f48a8153f00 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 -- 172.21.15.98:0/868175265 >> 172.21.15.98:0/868175265 conn(0x7f48a8067530 msgr2=0x7f48a80c5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 -- 172.21.15.98:0/868175265 shutdown_connections 2024-05-27T00:21:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:13.634+0000 7f48ad800640 1 -- 172.21.15.98:0/868175265 wait complete. 2024-05-27T00:21:13.659 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:21:15.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:21:15.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:21:15.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:15.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa511200640 1 Processor -- start 2024-05-27T00:21:15.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa511200640 1 -- start start 2024-05-27T00:21:15.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa511200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa50c0cbe10 0x7fa50c1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:15.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa511200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa50c0cc1f0 0x7fa50c0c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:15.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa511200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa50c0c6dd0 con 0x7fa50c073570 2024-05-27T00:21:15.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa511200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa50c159810 con 0x7fa50c0cc1f0 2024-05-27T00:21:15.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa511200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa50c0c70b0 con 0x7fa50c0cbe10 2024-05-27T00:21:15.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa509400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa50c0cc1f0 0x7fa50c0c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:15.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa509e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa50c0cbe10 0x7fa50c1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:15.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa50a800640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa50c073570 0x7fa50c1559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:35726/0 (socket says 172.21.15.98:35726) 2024-05-27T00:21:15.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa50a800640 1 -- 172.21.15.98:0/4054337998 learned_addr learned my addr 172.21.15.98:0/4054337998 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:15.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa509400640 1 -- 172.21.15.98:0/4054337998 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa50c0cbe10 msgr2=0x7fa50c1592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:15.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa509400640 1 --2- 172.21.15.98:0/4054337998 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa50c0cbe10 0x7fa50c1592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:15.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa509400640 1 -- 172.21.15.98:0/4054337998 >> v1:172.21.15.204:6789/0 conn(0x7fa50c073570 legacy=0x7fa50c1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:15.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.818+0000 7fa509400640 1 -- 172.21.15.98:0/4054337998 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50c0c7390 con 0x7fa50c0cc1f0 2024-05-27T00:21:15.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa509400640 1 --2- 172.21.15.98:0/4054337998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa50c0cc1f0 0x7fa50c0c1850 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80097c0 tx=0x7fa4f801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6e1e6c9d6688b1d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:15.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa508a00640 1 -- 172.21.15.98:0/4054337998 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f80b5070 con 0x7fa50c0cc1f0 2024-05-27T00:21:15.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa508a00640 1 -- 172.21.15.98:0/4054337998 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f80a8990 con 0x7fa50c0cc1f0 2024-05-27T00:21:15.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa508a00640 1 -- 172.21.15.98:0/4054337998 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f80a8d60 con 0x7fa50c0cc1f0 2024-05-27T00:21:15.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa511200640 1 -- 172.21.15.98:0/4054337998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa50c0cc1f0 msgr2=0x7fa50c0c1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:15.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa511200640 1 --2- 172.21.15.98:0/4054337998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa50c0cc1f0 0x7fa50c0c1850 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa4f80097c0 tx=0x7fa4f801ec90 comp rx=0 tx=0).stop 2024-05-27T00:21:15.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa511200640 1 -- 172.21.15.98:0/4054337998 shutdown_connections 2024-05-27T00:21:15.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa511200640 1 --2- 172.21.15.98:0/4054337998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa50c0cc1f0 0x7fa50c0c1850 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:15.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa511200640 1 --2- 172.21.15.98:0/4054337998 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa50c0cbe10 0x7fa50c1592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:15.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa511200640 1 -- 172.21.15.98:0/4054337998 >> 172.21.15.98:0/4054337998 conn(0x7fa50c067530 msgr2=0x7fa50c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:15.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa511200640 1 -- 172.21.15.98:0/4054337998 shutdown_connections 2024-05-27T00:21:15.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.822+0000 7fa511200640 1 -- 172.21.15.98:0/4054337998 wait complete. 2024-05-27T00:21:15.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa511200640 1 Processor -- start 2024-05-27T00:21:15.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa511200640 1 -- start start 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa511200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa50c073570 0x7fa50c1570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa511200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c0cbe10 0x7fa50c070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa511200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa50c0cc1f0 0x7fa50c06b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa511200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c0c79d0 con 0x7fa50c0cbe10 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa511200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa50c0c1330 con 0x7fa50c0cc1f0 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa511200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa50c0c6990 con 0x7fa50c073570 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa509400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c0cbe10 0x7fa50c070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa509400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c0cbe10 0x7fa50c070e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60430/0 (socket says 172.21.15.98:60430) 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa509400640 1 -- 172.21.15.98:0/379397709 learned_addr learned my addr 172.21.15.98:0/379397709 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa509e00640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa50c073570 0x7fa50c1570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:15.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa50a800640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa50c0cc1f0 0x7fa50c06b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:15.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa509400640 1 -- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa50c073570 msgr2=0x7fa50c1570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:15.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa509400640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa50c073570 0x7fa50c1570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:15.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa509400640 1 -- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa50c0cc1f0 msgr2=0x7fa50c06b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:15.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa509400640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa50c0cc1f0 0x7fa50c06b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:15.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa509400640 1 -- 172.21.15.98:0/379397709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50c0c3710 con 0x7fa50c0cbe10 2024-05-27T00:21:15.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa509400640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c0cbe10 0x7fa50c070e10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa4f801d340 tx=0x7fa4f801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:15.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa502000640 1 -- 172.21.15.98:0/379397709 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f80c1070 con 0x7fa50c0cbe10 2024-05-27T00:21:15.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa502000640 1 -- 172.21.15.98:0/379397709 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f80b0700 con 0x7fa50c0cbe10 2024-05-27T00:21:15.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa502000640 1 -- 172.21.15.98:0/379397709 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f80b5040 con 0x7fa50c0cbe10 2024-05-27T00:21:15.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa511200640 1 -- 172.21.15.98:0/379397709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50c0c3cb0 con 0x7fa50c0cbe10 2024-05-27T00:21:15.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.826+0000 7fa511200640 1 -- 172.21.15.98:0/379397709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50c0c4930 con 0x7fa50c0cbe10 2024-05-27T00:21:15.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.830+0000 7fa511200640 1 -- 172.21.15.98:0/379397709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa50c075cb0 con 0x7fa50c0cbe10 2024-05-27T00:21:15.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.830+0000 7fa502000640 1 -- 172.21.15.98:0/379397709 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa4f80c4070 con 0x7fa50c0cbe10 2024-05-27T00:21:15.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.830+0000 7fa502000640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4ec0678c0 0x7fa4ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:15.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.830+0000 7fa502000640 1 -- 172.21.15.98:0/379397709 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7fa4f8122c30 con 0x7fa50c0cbe10 2024-05-27T00:21:15.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.830+0000 7fa509e00640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4ec0678c0 0x7fa4ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:15.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.830+0000 7fa509e00640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4ec0678c0 0x7fa4ec069d80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009b60 tx=0x7fa4fc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:15.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:15.834+0000 7fa502000640 1 -- 172.21.15.98:0/379397709 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa4f80187f0 con 0x7fa50c0cbe10 2024-05-27T00:21:16.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 -- 172.21.15.98:0/379397709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa50c071350 con 0x7fa50c0cbe10 2024-05-27T00:21:16.035 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:16.035 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5202,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:54.483605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":17,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5073,"qdb_cluster":[5073]},"id":1}]} 2024-05-27T00:21:16.035 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-05-27T00:21:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa502000640 1 -- 172.21.15.98:0/379397709 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6839 (secure 0 0 0) 0x7fa4f80f2e70 con 0x7fa50c0cbe10 2024-05-27T00:21:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 -- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4ec0678c0 msgr2=0x7fa4ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4ec0678c0 0x7fa4ec069d80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009b60 tx=0x7fa4fc009190 comp rx=0 tx=0).stop 2024-05-27T00:21:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 -- 172.21.15.98:0/379397709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c0cbe10 msgr2=0x7fa50c070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c0cbe10 0x7fa50c070e10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa4f801d340 tx=0x7fa4f801ba30 comp rx=0 tx=0).stop 2024-05-27T00:21:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 -- 172.21.15.98:0/379397709 shutdown_connections 2024-05-27T00:21:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa4ec0678c0 0x7fa4ec069d80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa50c0cc1f0 0x7fa50c06b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50c0cbe10 0x7fa50c070e10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:16.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 --2- 172.21.15.98:0/379397709 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa50c073570 0x7fa50c1570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:16.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 -- 172.21.15.98:0/379397709 >> 172.21.15.98:0/379397709 conn(0x7fa50c067530 msgr2=0x7fa50c0ca370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:16.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 -- 172.21.15.98:0/379397709 shutdown_connections 2024-05-27T00:21:16.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:16.030+0000 7fa511200640 1 -- 172.21.15.98:0/379397709 wait complete. 2024-05-27T00:21:16.344 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:21:18.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:18.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.490+0000 7f8a96200640 1 Processor -- start 2024-05-27T00:21:18.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.490+0000 7f8a96200640 1 -- start start 2024-05-27T00:21:18.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a96200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8a900c79f0 0x7f8a90157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:18.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a96200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a900c17d0 con 0x7f8a900731f0 2024-05-27T00:21:18.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a96200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a90070e10 con 0x7f8a900735d0 2024-05-27T00:21:18.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a96200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f8a90071990 con 0x7f8a900c79f0 2024-05-27T00:21:18.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8f400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8a900731f0 0x7f8a900c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49838/0 (socket says 172.21.15.98:49838) 2024-05-27T00:21:18.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8f400640 1 -- 172.21.15.98:0/357492243 learned_addr learned my addr 172.21.15.98:0/357492243 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:18.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8fe00640 1 --2- 172.21.15.98:0/357492243 >> v2:172.21.15.204:3300/0 conn(0x7f8a900c79f0 0x7f8a90157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:18.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3730674639 0 0) 0x7f8a900c17d0 con 0x7f8a900731f0 2024-05-27T00:21:18.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a74005740 con 0x7f8a900731f0 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1283211374 0 0) 0x7f8a74005740 con 0x7f8a900731f0 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 >> v2:172.21.15.204:3300/0 conn(0x7f8a900c79f0 msgr2=0x7f8a90157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 --2- 172.21.15.98:0/357492243 >> v2:172.21.15.204:3300/0 conn(0x7f8a900c79f0 0x7f8a90157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 >> v1:172.21.15.204:6789/0 conn(0x7f8a900735d0 legacy=0x7f8a90151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a900710f0 con 0x7f8a900731f0 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f8a80001680 con 0x7f8a900731f0 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 >> v1:172.21.15.98:6789/0 conn(0x7f8a900731f0 legacy=0x7f8a900c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74005bd0 0x7f8a74006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a74006a20 0x7f8a74006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a740073e0 0x7f8a7400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a90071990 con 0x7f8a74005bd0 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7400e2b0 con 0x7f8a74006a20 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8a7400e4b0 con 0x7f8a740073e0 2024-05-27T00:21:18.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8f400640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74005bd0 0x7f8a74006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:18.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a87e00640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a74006a20 0x7f8a74006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:18.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8fe00640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a740073e0 0x7f8a7400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:18.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8f400640 1 -- 172.21.15.98:0/357492243 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a740073e0 msgr2=0x7f8a7400dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:18.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8f400640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a740073e0 0x7f8a7400dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8f400640 1 -- 172.21.15.98:0/357492243 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a74006a20 msgr2=0x7f8a74006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:18.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8f400640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a74006a20 0x7f8a74006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8f400640 1 -- 172.21.15.98:0/357492243 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a74005740 con 0x7f8a74005bd0 2024-05-27T00:21:18.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.494+0000 7f8a8fe00640 1 -- 172.21.15.98:0/357492243 reap_dead start 2024-05-27T00:21:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a8f400640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74005bd0 0x7f8a74006010 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000c240 tx=0x7f8a8001cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6de1584816ef10 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a80007d20 con 0x7f8a74005bd0 2024-05-27T00:21:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a800a9020 con 0x7f8a74005bd0 2024-05-27T00:21:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a8ea00640 1 -- 172.21.15.98:0/357492243 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a80005ae0 con 0x7f8a74005bd0 2024-05-27T00:21:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a96200640 1 -- 172.21.15.98:0/357492243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74005bd0 msgr2=0x7f8a74006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a96200640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74005bd0 0x7f8a74006010 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000c240 tx=0x7f8a8001cbd0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a96200640 1 -- 172.21.15.98:0/357492243 shutdown_connections 2024-05-27T00:21:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a96200640 1 --2- 172.21.15.98:0/357492243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a74005bd0 0x7f8a74006010 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a96200640 1 -- 172.21.15.98:0/357492243 >> 172.21.15.98:0/357492243 conn(0x7f8a90067530 msgr2=0x7f8a9006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a96200640 1 -- 172.21.15.98:0/357492243 shutdown_connections 2024-05-27T00:21:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.498+0000 7f8a96200640 1 -- 172.21.15.98:0/357492243 wait complete. 2024-05-27T00:21:18.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 Processor -- start 2024-05-27T00:21:18.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 -- start start 2024-05-27T00:21:18.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a74005bd0 0x7f8a9014cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:18.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a90152650 0x7f8a90154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:18.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a901f71a0 0x7f8a90146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:18.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a901515b0 con 0x7f8a901f71a0 2024-05-27T00:21:18.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8a90151430 con 0x7f8a90152650 2024-05-27T00:21:18.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8a90151730 con 0x7f8a74005bd0 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a901f71a0 0x7f8a90146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a901f71a0 0x7f8a90146ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60444/0 (socket says 172.21.15.98:60444) 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8fe00640 1 -- 172.21.15.98:0/3599641097 learned_addr learned my addr 172.21.15.98:0/3599641097 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8f400640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a74005bd0 0x7f8a9014cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a87e00640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a90152650 0x7f8a90154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8fe00640 1 -- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a74005bd0 msgr2=0x7f8a9014cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8fe00640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a74005bd0 0x7f8a9014cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8fe00640 1 -- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a90152650 msgr2=0x7f8a90154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8fe00640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a90152650 0x7f8a90154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8fe00640 1 -- 172.21.15.98:0/3599641097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a901fa7b0 con 0x7f8a901f71a0 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8f400640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a74005bd0 0x7f8a9014cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a87e00640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a90152650 0x7f8a90154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:18.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a8fe00640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a901f71a0 0x7f8a90146ef0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8a90070bf0 tx=0x7f8a880115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:18.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a87400640 1 -- 172.21.15.98:0/3599641097 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88011de0 con 0x7f8a901f71a0 2024-05-27T00:21:18.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a87400640 1 -- 172.21.15.98:0/3599641097 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a8801b070 con 0x7f8a901f71a0 2024-05-27T00:21:18.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 -- 172.21.15.98:0/3599641097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a90152bb0 con 0x7f8a901f71a0 2024-05-27T00:21:18.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a87400640 1 -- 172.21.15.98:0/3599641097 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a880039a0 con 0x7f8a901f71a0 2024-05-27T00:21:18.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 -- 172.21.15.98:0/3599641097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a901fad30 con 0x7f8a901f71a0 2024-05-27T00:21:18.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.502+0000 7f8a96200640 1 -- 172.21.15.98:0/3599641097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a90075cb0 con 0x7f8a901f71a0 2024-05-27T00:21:18.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.506+0000 7f8a87400640 1 -- 172.21.15.98:0/3599641097 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8a88003b40 con 0x7f8a901f71a0 2024-05-27T00:21:18.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.506+0000 7f8a87400640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8a6c0678c0 0x7f8a6c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:18.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.506+0000 7f8a87400640 1 -- 172.21.15.98:0/3599641097 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7f8a880866a0 con 0x7f8a901f71a0 2024-05-27T00:21:18.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.506+0000 7f8a8f400640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8a6c0678c0 0x7f8a6c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:18.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.506+0000 7f8a8f400640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8a6c0678c0 0x7f8a6c069d80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000c240 tx=0x7f8a900c19a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:18.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.510+0000 7f8a87400640 1 -- 172.21.15.98:0/3599641097 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8a88056fc0 con 0x7f8a901f71a0 2024-05-27T00:21:18.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 -- 172.21.15.98:0/3599641097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a90158510 con 0x7f8a901f71a0 2024-05-27T00:21:18.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a87400640 1 -- 172.21.15.98:0/3599641097 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6839 (secure 0 0 0) 0x7f8a880568c0 con 0x7f8a901f71a0 2024-05-27T00:21:18.703 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:18.703 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5202,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:54.483605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":17,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5073,"qdb_cluster":[5073]},"id":1}]} 2024-05-27T00:21:18.703 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-05-27T00:21:18.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 -- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8a6c0678c0 msgr2=0x7f8a6c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:18.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8a6c0678c0 0x7f8a6c069d80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000c240 tx=0x7f8a900c19a0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 -- 172.21.15.98:0/3599641097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a901f71a0 msgr2=0x7f8a90146ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:18.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a901f71a0 0x7f8a90146ef0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8a90070bf0 tx=0x7f8a880115a0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 -- 172.21.15.98:0/3599641097 shutdown_connections 2024-05-27T00:21:18.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a901f71a0 0x7f8a90146ef0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8a6c0678c0 0x7f8a6c069d80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a90152650 0x7f8a90154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 --2- 172.21.15.98:0/3599641097 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a74005bd0 0x7f8a9014cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 -- 172.21.15.98:0/3599641097 >> 172.21.15.98:0/3599641097 conn(0x7f8a90067530 msgr2=0x7f8a900c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 -- 172.21.15.98:0/3599641097 shutdown_connections 2024-05-27T00:21:18.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:18.702+0000 7f8a96200640 1 -- 172.21.15.98:0/3599641097 wait complete. 2024-05-27T00:21:18.724 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:21:20.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:20.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.870+0000 7f7220400640 1 Processor -- start 2024-05-27T00:21:20.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.870+0000 7f7220400640 1 -- start start 2024-05-27T00:21:20.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f7220400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f72180cbe10 0x7f72181592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:20.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f7220400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7218073570 0x7f72181559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:20.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f7220400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72180c1f60 con 0x7f72180cc1f0 2024-05-27T00:21:20.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f7220400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7218159810 con 0x7f7218073570 2024-05-27T00:21:20.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f7220400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f72180c6dd0 con 0x7f72180cbe10 2024-05-27T00:21:20.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f721e400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7218073570 0x7f72181559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:20.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f721e400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7218073570 0x7f72181559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60454/0 (socket says 172.21.15.98:60454) 2024-05-27T00:21:20.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f721e400640 1 -- 172.21.15.98:0/1663485866 learned_addr learned my addr 172.21.15.98:0/1663485866 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:20.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f721da00640 1 --2- 172.21.15.98:0/1663485866 >> v2:172.21.15.204:3300/0 conn(0x7f72180cbe10 0x7f72181592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:20.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f721e400640 1 -- 172.21.15.98:0/1663485866 >> v2:172.21.15.204:3300/0 conn(0x7f72180cbe10 msgr2=0x7f72181592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:20.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f721e400640 1 --2- 172.21.15.98:0/1663485866 >> v2:172.21.15.204:3300/0 conn(0x7f72180cbe10 0x7f72181592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:20.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f721e400640 1 -- 172.21.15.98:0/1663485866 >> v1:172.21.15.204:6789/0 conn(0x7f72180cc1f0 legacy=0x7f72180c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:20.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f721e400640 1 -- 172.21.15.98:0/1663485866 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72180c70b0 con 0x7f7218073570 2024-05-27T00:21:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f721e400640 1 --2- 172.21.15.98:0/1663485866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7218073570 0x7f72181559e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f72180cc760 tx=0x7f721001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9516cd69c3df45f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.874+0000 7f7217e00640 1 -- 172.21.15.98:0/1663485866 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72100b4070 con 0x7f7218073570 2024-05-27T00:21:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7217e00640 1 -- 172.21.15.98:0/1663485866 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72100af400 con 0x7f7218073570 2024-05-27T00:21:20.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7217e00640 1 -- 172.21.15.98:0/1663485866 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72100af7d0 con 0x7f7218073570 2024-05-27T00:21:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 -- 172.21.15.98:0/1663485866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7218073570 msgr2=0x7f72181559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:20.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 --2- 172.21.15.98:0/1663485866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7218073570 0x7f72181559e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f72180cc760 tx=0x7f721001f1e0 comp rx=0 tx=0).stop 2024-05-27T00:21:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 -- 172.21.15.98:0/1663485866 shutdown_connections 2024-05-27T00:21:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 --2- 172.21.15.98:0/1663485866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7218073570 0x7f72181559e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 --2- 172.21.15.98:0/1663485866 >> v2:172.21.15.204:3300/0 conn(0x7f72180cbe10 0x7f72181592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 -- 172.21.15.98:0/1663485866 >> 172.21.15.98:0/1663485866 conn(0x7f7218067530 msgr2=0x7f721806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 -- 172.21.15.98:0/1663485866 shutdown_connections 2024-05-27T00:21:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 -- 172.21.15.98:0/1663485866 wait complete. 2024-05-27T00:21:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 Processor -- start 2024-05-27T00:21:20.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 -- start start 2024-05-27T00:21:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7218073570 0x7f72180c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72180cbe10 0x7f72181534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72180cc1f0 0x7f72181f1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f72180c1330 con 0x7f72180cc1f0 2024-05-27T00:21:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f72180c1050 con 0x7f72180cbe10 2024-05-27T00:21:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f7220400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f72180c79d0 con 0x7f7218073570 2024-05-27T00:21:20.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721e400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72180cc1f0 0x7f72181f1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721e400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72180cc1f0 0x7f72181f1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60456/0 (socket says 172.21.15.98:60456) 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721e400640 1 -- 172.21.15.98:0/1028640168 learned_addr learned my addr 172.21.15.98:0/1028640168 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721da00640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7218073570 0x7f72180c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721d000640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72180cbe10 0x7f72181534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721e400640 1 -- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7218073570 msgr2=0x7f72180c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721e400640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7218073570 0x7f72180c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721e400640 1 -- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72180cbe10 msgr2=0x7f72181534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721e400640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72180cbe10 0x7f72181534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721e400640 1 -- 172.21.15.98:0/1028640168 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7218154080 con 0x7f72180cc1f0 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.878+0000 7f721d000640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72180cbe10 0x7f72181534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:21:20.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f721e400640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72180cc1f0 0x7f72181f1540 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f72100b2040 tx=0x7f721001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f7215600640 1 -- 172.21.15.98:0/1028640168 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72100c0070 con 0x7f72180cc1f0 2024-05-27T00:21:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f7215600640 1 -- 172.21.15.98:0/1028640168 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72100090d0 con 0x7f72180cc1f0 2024-05-27T00:21:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f7215600640 1 -- 172.21.15.98:0/1028640168 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72100b4040 con 0x7f72180cc1f0 2024-05-27T00:21:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f7220400640 1 -- 172.21.15.98:0/1028640168 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7218153da0 con 0x7f72180cc1f0 2024-05-27T00:21:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f7220400640 1 -- 172.21.15.98:0/1028640168 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72180c5cf0 con 0x7f72180cc1f0 2024-05-27T00:21:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f7220400640 1 -- 172.21.15.98:0/1028640168 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7218075cb0 con 0x7f72180cc1f0 2024-05-27T00:21:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f7215600640 1 -- 172.21.15.98:0/1028640168 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f72100b8670 con 0x7f72180cc1f0 2024-05-27T00:21:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f7215600640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71f80678c0 0x7f71f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:20.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f7215600640 1 -- 172.21.15.98:0/1028640168 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7f7210123470 con 0x7f72180cc1f0 2024-05-27T00:21:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.882+0000 7f721da00640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71f80678c0 0x7f71f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:20.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.886+0000 7f721da00640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71f80678c0 0x7f71f8069d80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f720c00b9d0 tx=0x7f720c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:20.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:20.886+0000 7f7215600640 1 -- 172.21.15.98:0/1028640168 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7210018d40 con 0x7f72180cc1f0 2024-05-27T00:21:21.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.074+0000 7f7220400640 1 -- 172.21.15.98:0/1028640168 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f721806eb30 con 0x7f72180cc1f0 2024-05-27T00:21:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.074+0000 7f7215600640 1 -- 172.21.15.98:0/1028640168 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6839 (secure 0 0 0) 0x7f72100f3d90 con 0x7f72180cc1f0 2024-05-27T00:21:21.077 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:21.077 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5202,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:54.483605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":17,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5073,"qdb_cluster":[5073]},"id":1}]} 2024-05-27T00:21:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-05-27T00:21:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 -- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71f80678c0 msgr2=0x7f71f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71f80678c0 0x7f71f8069d80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f720c00b9d0 tx=0x7f720c008040 comp rx=0 tx=0).stop 2024-05-27T00:21:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 -- 172.21.15.98:0/1028640168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72180cc1f0 msgr2=0x7f72181f1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72180cc1f0 0x7f72181f1540 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f72100b2040 tx=0x7f721001bf80 comp rx=0 tx=0).stop 2024-05-27T00:21:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 -- 172.21.15.98:0/1028640168 shutdown_connections 2024-05-27T00:21:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72180cc1f0 0x7f72181f1540 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71f80678c0 0x7f71f8069d80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72180cbe10 0x7f72181534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 --2- 172.21.15.98:0/1028640168 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7218073570 0x7f72180c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:21.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 -- 172.21.15.98:0/1028640168 >> 172.21.15.98:0/1028640168 conn(0x7f7218067530 msgr2=0x7f72180c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:21.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 -- 172.21.15.98:0/1028640168 shutdown_connections 2024-05-27T00:21:21.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:21.078+0000 7f7220400640 1 -- 172.21.15.98:0/1028640168 wait complete. 2024-05-27T00:21:21.099 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:21:23.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:23.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f9600640 1 Processor -- start 2024-05-27T00:21:23.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f9600640 1 -- start start 2024-05-27T00:21:23.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f9600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7ff3f4070c00 0x7ff3f40c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:23.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f9600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4154e90 0x7ff3f415ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:23.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f9600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3f415a950 con 0x7ff3f4070fe0 2024-05-27T00:21:23.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f9600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7ff3f415e060 con 0x7ff3f4070c00 2024-05-27T00:21:23.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f9600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff3f415e340 con 0x7ff3f4154e90 2024-05-27T00:21:23.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3ea000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff3f4070fe0 0x7ff3f4154780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60280/0 (socket says 172.21.15.98:60280) 2024-05-27T00:21:23.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3ea000640 1 -- 172.21.15.98:0/2548612287 learned_addr learned my addr 172.21.15.98:0/2548612287 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:23.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f3400640 1 --2- 172.21.15.98:0/2548612287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4154e90 0x7ff3f415ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:23.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f2a00640 1 --2- 172.21.15.98:0/2548612287 >> v2:172.21.15.204:3300/0 conn(0x7ff3f4070c00 0x7ff3f40c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:23.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f2000640 1 -- 172.21.15.98:0/2548612287 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3263694946 0 0) 0x7ff3f415a950 con 0x7ff3f4070fe0 2024-05-27T00:21:23.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f2000640 1 -- 172.21.15.98:0/2548612287 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3d8003530 con 0x7ff3f4070fe0 2024-05-27T00:21:23.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f3400640 1 -- 172.21.15.98:0/2548612287 >> v2:172.21.15.204:3300/0 conn(0x7ff3f4070c00 msgr2=0x7ff3f40c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:23.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f3400640 1 --2- 172.21.15.98:0/2548612287 >> v2:172.21.15.204:3300/0 conn(0x7ff3f4070c00 0x7ff3f40c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:23.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f3400640 1 -- 172.21.15.98:0/2548612287 >> v1:172.21.15.98:6789/0 conn(0x7ff3f4070fe0 legacy=0x7ff3f4154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:23.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.246+0000 7ff3f3400640 1 -- 172.21.15.98:0/2548612287 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f415e620 con 0x7ff3f4154e90 2024-05-27T00:21:23.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f3400640 1 --2- 172.21.15.98:0/2548612287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4154e90 0x7ff3f415ae50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009d90 tx=0x7ff3ec01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d77088e05c95dfdf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:23.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f2000640 1 -- 172.21.15.98:0/2548612287 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0b4070 con 0x7ff3f4154e90 2024-05-27T00:21:23.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f2000640 1 -- 172.21.15.98:0/2548612287 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3ec0af600 con 0x7ff3f4154e90 2024-05-27T00:21:23.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f2000640 1 -- 172.21.15.98:0/2548612287 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0af9b0 con 0x7ff3f4154e90 2024-05-27T00:21:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f9600640 1 -- 172.21.15.98:0/2548612287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4154e90 msgr2=0x7ff3f415ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f9600640 1 --2- 172.21.15.98:0/2548612287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4154e90 0x7ff3f415ae50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009d90 tx=0x7ff3ec01f150 comp rx=0 tx=0).stop 2024-05-27T00:21:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f9600640 1 -- 172.21.15.98:0/2548612287 shutdown_connections 2024-05-27T00:21:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f9600640 1 --2- 172.21.15.98:0/2548612287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4154e90 0x7ff3f415ae50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f9600640 1 --2- 172.21.15.98:0/2548612287 >> v2:172.21.15.204:3300/0 conn(0x7ff3f4070c00 0x7ff3f40c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f9600640 1 -- 172.21.15.98:0/2548612287 >> 172.21.15.98:0/2548612287 conn(0x7ff3f4072f90 msgr2=0x7ff3f40733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f9600640 1 -- 172.21.15.98:0/2548612287 shutdown_connections 2024-05-27T00:21:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.250+0000 7ff3f9600640 1 -- 172.21.15.98:0/2548612287 wait complete. 2024-05-27T00:21:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f9600640 1 Processor -- start 2024-05-27T00:21:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f9600640 1 -- start start 2024-05-27T00:21:23.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f9600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3f4070c00 0x7ff3f41f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:23.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f9600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4070fe0 0x7ff3f414ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:23.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f9600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f4154e90 0x7ff3f414f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:23.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f9600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f40c8f40 con 0x7ff3f4154e90 2024-05-27T00:21:23.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f9600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f415ebe0 con 0x7ff3f4070c00 2024-05-27T00:21:23.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f9600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff3f40c90c0 con 0x7ff3f4070fe0 2024-05-27T00:21:23.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f4154e90 0x7ff3f414f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f4154e90 0x7ff3f414f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37286/0 (socket says 172.21.15.98:37286) 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f3400640 1 -- 172.21.15.98:0/2981119625 learned_addr learned my addr 172.21.15.98:0/2981119625 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3ea000640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4070fe0 0x7ff3f414ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f2a00640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3f4070c00 0x7ff3f41f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f3400640 1 -- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4070fe0 msgr2=0x7ff3f414ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f3400640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4070fe0 0x7ff3f414ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f3400640 1 -- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3f4070c00 msgr2=0x7ff3f41f3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f3400640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3f4070c00 0x7ff3f41f3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f3400640 1 -- 172.21.15.98:0/2981119625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f41f42f0 con 0x7ff3f4154e90 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3ea000640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4070fe0 0x7ff3f414ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:23.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f3400640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f4154e90 0x7ff3f414f4d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec0b2040 tx=0x7ff3ec01bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:23.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3eb400640 1 -- 172.21.15.98:0/2981119625 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0b4040 con 0x7ff3f4154e90 2024-05-27T00:21:23.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3eb400640 1 -- 172.21.15.98:0/2981119625 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3ec0c1070 con 0x7ff3f4154e90 2024-05-27T00:21:23.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f9600640 1 -- 172.21.15.98:0/2981119625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f41f45d0 con 0x7ff3f4154e90 2024-05-27T00:21:23.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3eb400640 1 -- 172.21.15.98:0/2981119625 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec009450 con 0x7ff3f4154e90 2024-05-27T00:21:23.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.254+0000 7ff3f9600640 1 -- 172.21.15.98:0/2981119625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f415ca80 con 0x7ff3f4154e90 2024-05-27T00:21:23.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.258+0000 7ff3f9600640 1 -- 172.21.15.98:0/2981119625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3f4075cb0 con 0x7ff3f4154e90 2024-05-27T00:21:23.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.258+0000 7ff3eb400640 1 -- 172.21.15.98:0/2981119625 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff3ec0ba690 con 0x7ff3f4154e90 2024-05-27T00:21:23.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.258+0000 7ff3eb400640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff3d00678c0 0x7ff3d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:23.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.258+0000 7ff3eb400640 1 -- 172.21.15.98:0/2981119625 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6083+0+0 (secure 0 0 0) 0x7ff3ec1247e0 con 0x7ff3f4154e90 2024-05-27T00:21:23.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.258+0000 7ff3f2a00640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff3d00678c0 0x7ff3d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:23.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.258+0000 7ff3f2a00640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff3d00678c0 0x7ff3d0069d80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009af0 tx=0x7ff3e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.262+0000 7ff3eb400640 1 -- 172.21.15.98:0/2981119625 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7ff3ec018cb0 con 0x7ff3f4154e90 2024-05-27T00:21:23.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 -- 172.21.15.98:0/2981119625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3f40c9720 con 0x7ff3f4154e90 2024-05-27T00:21:23.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3eb400640 1 -- 172.21.15.98:0/2981119625 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6839 (secure 0 0 0) 0x7ff3ec0f5080 con 0x7ff3f4154e90 2024-05-27T00:21:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:23.456 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5202,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:19:54.483605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5073},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5073":{"gid":5073,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":17,"addr":"172.21.15.98:6834/3247908011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3247908011},{"type":"v1","addr":"172.21.15.98:6834","nonce":3247908011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5073,"qdb_cluster":[5073]},"id":1}]} 2024-05-27T00:21:23.456 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2024-05-27T00:21:23.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 -- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff3d00678c0 msgr2=0x7ff3d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:23.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff3d00678c0 0x7ff3d0069d80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009af0 tx=0x7ff3e4008040 comp rx=0 tx=0).stop 2024-05-27T00:21:23.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 -- 172.21.15.98:0/2981119625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f4154e90 msgr2=0x7ff3f414f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:23.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f4154e90 0x7ff3f414f4d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec0b2040 tx=0x7ff3ec01bef0 comp rx=0 tx=0).stop 2024-05-27T00:21:23.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 -- 172.21.15.98:0/2981119625 shutdown_connections 2024-05-27T00:21:23.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3f4154e90 0x7ff3f414f4d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:23.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3f4070fe0 0x7ff3f414ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:23.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff3d00678c0 0x7ff3d0069d80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:23.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 --2- 172.21.15.98:0/2981119625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3f4070c00 0x7ff3f41f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:23.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 -- 172.21.15.98:0/2981119625 >> 172.21.15.98:0/2981119625 conn(0x7ff3f4072f90 msgr2=0x7ff3f4159c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:23.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 -- 172.21.15.98:0/2981119625 shutdown_connections 2024-05-27T00:21:23.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:23.454+0000 7ff3f9600640 1 -- 172.21.15.98:0/2981119625 wait complete. 2024-05-27T00:21:23.477 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:21:25.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:25.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.626+0000 7fe1c1a00640 1 Processor -- start 2024-05-27T00:21:25.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.626+0000 7fe1c1a00640 1 -- start start 2024-05-27T00:21:25.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1c1a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe1bc154690 0x7fe1bc15b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:25.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1c1a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0ccda0 0x7fe1bc0cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:25.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1c1a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1bc0cb190 con 0x7fe1bc0cc290 2024-05-27T00:21:25.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1c1a00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fe1bc15b650 con 0x7fe1bc154690 2024-05-27T00:21:25.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1c1a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe1bc0caf40 con 0x7fe1bc0ccda0 2024-05-27T00:21:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1c1000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0ccda0 0x7fe1bc0cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1bbe00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe1bc154690 0x7fe1bc15b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1bbe00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe1bc154690 0x7fe1bc15b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:39504/0 (socket says 172.21.15.98:39504) 2024-05-27T00:21:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1c1000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0ccda0 0x7fe1bc0cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:36626/0 (socket says 172.21.15.98:36626) 2024-05-27T00:21:25.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1bbe00640 1 -- 172.21.15.98:0/659552401 learned_addr learned my addr 172.21.15.98:0/659552401 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:25.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1bbe00640 1 -- 172.21.15.98:0/659552401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0ccda0 msgr2=0x7fe1bc0cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:25.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1bbe00640 1 --2- 172.21.15.98:0/659552401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0ccda0 0x7fe1bc0cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:25.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1bbe00640 1 -- 172.21.15.98:0/659552401 >> v1:172.21.15.204:6789/0 conn(0x7fe1bc0cc290 legacy=0x7fe1bc0c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:25.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1bbe00640 1 -- 172.21.15.98:0/659552401 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1bc15dd20 con 0x7fe1bc154690 2024-05-27T00:21:25.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.630+0000 7fe1c1000640 1 --2- 172.21.15.98:0/659552401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0ccda0 0x7fe1bc0cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:21:25.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1bbe00640 1 --2- 172.21.15.98:0/659552401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1bc154690 0x7fe1bc15b110 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009870 tx=0x7fe1ac01ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=798915f647df41aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1bb400640 1 -- 172.21.15.98:0/659552401 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0b5070 con 0x7fe1bc154690 2024-05-27T00:21:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1bb400640 1 -- 172.21.15.98:0/659552401 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1ac0a8a50 con 0x7fe1bc154690 2024-05-27T00:21:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1bb400640 1 -- 172.21.15.98:0/659552401 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0a8e20 con 0x7fe1bc154690 2024-05-27T00:21:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1c1a00640 1 -- 172.21.15.98:0/659552401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1bc154690 msgr2=0x7fe1bc15b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1c1a00640 1 --2- 172.21.15.98:0/659552401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1bc154690 0x7fe1bc15b110 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009870 tx=0x7fe1ac01ed50 comp rx=0 tx=0).stop 2024-05-27T00:21:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1c1a00640 1 -- 172.21.15.98:0/659552401 shutdown_connections 2024-05-27T00:21:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1c1a00640 1 --2- 172.21.15.98:0/659552401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0ccda0 0x7fe1bc0cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1c1a00640 1 --2- 172.21.15.98:0/659552401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1bc154690 0x7fe1bc15b110 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1c1a00640 1 -- 172.21.15.98:0/659552401 >> 172.21.15.98:0/659552401 conn(0x7fe1bc06c740 msgr2=0x7fe1bc06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1c1a00640 1 -- 172.21.15.98:0/659552401 shutdown_connections 2024-05-27T00:21:25.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1c1a00640 1 -- 172.21.15.98:0/659552401 wait complete. 2024-05-27T00:21:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1c1a00640 1 Processor -- start 2024-05-27T00:21:25.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.634+0000 7fe1c1a00640 1 -- start start 2024-05-27T00:21:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0cc290 0x7fe1bc1e5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1bc0ccda0 0x7fe1bc1f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc154690 0x7fe1bc1e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc15e5c0 con 0x7fe1bc154690 2024-05-27T00:21:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc15e2e0 con 0x7fe1bc0ccda0 2024-05-27T00:21:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe1bc0cb7e0 con 0x7fe1bc0cc290 2024-05-27T00:21:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc154690 0x7fe1bc1e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc154690 0x7fe1bc1e5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37290/0 (socket says 172.21.15.98:37290) 2024-05-27T00:21:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1000640 1 -- 172.21.15.98:0/2422432256 learned_addr learned my addr 172.21.15.98:0/2422432256 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1b3e00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1bc0ccda0 0x7fe1bc1f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1bbe00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0cc290 0x7fe1bc1e5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1000640 1 -- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0cc290 msgr2=0x7fe1bc1e5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1000640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0cc290 0x7fe1bc1e5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1000640 1 -- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1bc0ccda0 msgr2=0x7fe1bc1f0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1000640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1bc0ccda0 0x7fe1bc1f0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1000640 1 -- 172.21.15.98:0/2422432256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1bc0b62e0 con 0x7fe1bc154690 2024-05-27T00:21:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1000640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc154690 0x7fe1bc1e5ca0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009f00 tx=0x7fe1b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1b8c00640 1 -- 172.21.15.98:0/2422432256 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4018030 con 0x7fe1bc154690 2024-05-27T00:21:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1b8c00640 1 -- 172.21.15.98:0/2422432256 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b400ce50 con 0x7fe1bc154690 2024-05-27T00:21:25.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1b8c00640 1 -- 172.21.15.98:0/2422432256 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4003bc0 con 0x7fe1bc154690 2024-05-27T00:21:25.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1a00640 1 -- 172.21.15.98:0/2422432256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1bc159d90 con 0x7fe1bc154690 2024-05-27T00:21:25.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1a00640 1 -- 172.21.15.98:0/2422432256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1bc15a330 con 0x7fe1bc154690 2024-05-27T00:21:25.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.638+0000 7fe1c1a00640 1 -- 172.21.15.98:0/2422432256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1bc075c40 con 0x7fe1bc154690 2024-05-27T00:21:25.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.642+0000 7fe1b8c00640 1 -- 172.21.15.98:0/2422432256 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe1b401b070 con 0x7fe1bc154690 2024-05-27T00:21:25.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.642+0000 7fe1b8c00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1980678c0 0x7fe198069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:25.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.642+0000 7fe1b8c00640 1 -- 172.21.15.98:0/2422432256 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7fe1b40867b0 con 0x7fe1bc154690 2024-05-27T00:21:25.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.642+0000 7fe1bbe00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1980678c0 0x7fe198069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:25.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.642+0000 7fe1bbe00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1980678c0 0x7fe198069d80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0b2000 tx=0x7fe1ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:25.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.646+0000 7fe1b8c00640 1 -- 172.21.15.98:0/2422432256 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fe1b4022070 con 0x7fe1bc154690 2024-05-27T00:21:25.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.834+0000 7fe1c1a00640 1 -- 172.21.15.98:0/2422432256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1bc0cba80 con 0x7fe1bc154690 2024-05-27T00:21:25.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1b8c00640 1 -- 172.21.15.98:0/2422432256 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6006 (secure 0 0 0) 0x7fe1b4010040 con 0x7fe1bc154690 2024-05-27T00:21:25.838 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:25.838 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:25.205553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:25.838 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 -- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1980678c0 msgr2=0x7fe198069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1980678c0 0x7fe198069d80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0b2000 tx=0x7fe1ac009040 comp rx=0 tx=0).stop 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 -- 172.21.15.98:0/2422432256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc154690 msgr2=0x7fe1bc1e5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc154690 0x7fe1bc1e5ca0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009f00 tx=0x7fe1b400c490 comp rx=0 tx=0).stop 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 -- 172.21.15.98:0/2422432256 shutdown_connections 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc154690 0x7fe1bc1e5ca0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1980678c0 0x7fe198069d80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1bc0ccda0 0x7fe1bc1f0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 --2- 172.21.15.98:0/2422432256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1bc0cc290 0x7fe1bc1e5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 -- 172.21.15.98:0/2422432256 >> 172.21.15.98:0/2422432256 conn(0x7fe1bc06c740 msgr2=0x7fe1bc0cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:25.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 -- 172.21.15.98:0/2422432256 shutdown_connections 2024-05-27T00:21:25.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.838+0000 7fe1c1a00640 1 -- 172.21.15.98:0/2422432256 wait complete. 2024-05-27T00:21:25.859 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-05-27T00:21:25.859 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:21:25.859 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:25.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f7062000640 1 Processor -- start 2024-05-27T00:21:25.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f7062000640 1 -- start start 2024-05-27T00:21:25.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f7062000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c152520 0x7f705c0c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:25.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f7062000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f705c0c70e0 0x7f705c06b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f7062000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f705c06cc70 con 0x7f705c073760 2024-05-27T00:21:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f7062000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f705c072fa0 con 0x7f705c0c70e0 2024-05-27T00:21:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f7062000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f705c152900 con 0x7f705c152520 2024-05-27T00:21:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f705be00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f705c0c70e0 0x7f705c06b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f705be00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f705c0c70e0 0x7f705c06b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37298/0 (socket says 172.21.15.98:37298) 2024-05-27T00:21:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f705be00640 1 -- 172.21.15.98:0/2148569234 learned_addr learned my addr 172.21.15.98:0/2148569234 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f705b400640 1 --2- 172.21.15.98:0/2148569234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c152520 0x7f705c0c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:25.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f705be00640 1 -- 172.21.15.98:0/2148569234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c152520 msgr2=0x7f705c0c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:25.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f705be00640 1 --2- 172.21.15.98:0/2148569234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c152520 0x7f705c0c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:25.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f705be00640 1 -- 172.21.15.98:0/2148569234 >> v1:172.21.15.204:6789/0 conn(0x7f705c073760 legacy=0x7f705c071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:25.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f705be00640 1 -- 172.21.15.98:0/2148569234 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705c06c6b0 con 0x7f705c0c70e0 2024-05-27T00:21:25.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.994+0000 7f705b400640 1 --2- 172.21.15.98:0/2148569234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c152520 0x7f705c0c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:25.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f705be00640 1 --2- 172.21.15.98:0/2148569234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 0x7f705c06b720 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7054009870 tx=0x7f705401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad1f572f14b8be0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f705a000640 1 -- 172.21.15.98:0/2148569234 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70540ab650 con 0x7f705c0c70e0 2024-05-27T00:21:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f705a000640 1 -- 172.21.15.98:0/2148569234 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70540b0070 con 0x7f705c0c70e0 2024-05-27T00:21:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f705a000640 1 -- 172.21.15.98:0/2148569234 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70540aba10 con 0x7f705c0c70e0 2024-05-27T00:21:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f7062000640 1 -- 172.21.15.98:0/2148569234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 msgr2=0x7f705c06b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f7062000640 1 --2- 172.21.15.98:0/2148569234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 0x7f705c06b720 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7054009870 tx=0x7f705401c560 comp rx=0 tx=0).stop 2024-05-27T00:21:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f7062000640 1 -- 172.21.15.98:0/2148569234 shutdown_connections 2024-05-27T00:21:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f7062000640 1 --2- 172.21.15.98:0/2148569234 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 0x7f705c06b720 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f7062000640 1 --2- 172.21.15.98:0/2148569234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c152520 0x7f705c0c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f7062000640 1 -- 172.21.15.98:0/2148569234 >> 172.21.15.98:0/2148569234 conn(0x7f705c0c9b70 msgr2=0x7f705c0c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f7062000640 1 -- 172.21.15.98:0/2148569234 shutdown_connections 2024-05-27T00:21:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:25.998+0000 7f7062000640 1 -- 172.21.15.98:0/2148569234 wait complete. 2024-05-27T00:21:26.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 Processor -- start 2024-05-27T00:21:26.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 -- start start 2024-05-27T00:21:26.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c073760 0x7f705c06de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:26.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 0x7f705c1f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:26.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f705c152520 0x7f705c145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:26.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f705c165a30 con 0x7f705c0c70e0 2024-05-27T00:21:26.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f705c1658b0 con 0x7f705c152520 2024-05-27T00:21:26.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f705c165bb0 con 0x7f705c073760 2024-05-27T00:21:26.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 0x7f705c1f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 0x7f705c1f3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37310/0 (socket says 172.21.15.98:37310) 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705aa00640 1 -- 172.21.15.98:0/2408244195 learned_addr learned my addr 172.21.15.98:0/2408244195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705b400640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c073760 0x7f705c06de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705be00640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f705c152520 0x7f705c145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705aa00640 1 -- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c073760 msgr2=0x7f705c06de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705aa00640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c073760 0x7f705c06de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705aa00640 1 -- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f705c152520 msgr2=0x7f705c145390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705aa00640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f705c152520 0x7f705c145390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705aa00640 1 -- 172.21.15.98:0/2408244195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705c0c6300 con 0x7f705c0c70e0 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705b400640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c073760 0x7f705c06de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f705aa00640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 0x7f705c1f3c40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f704800c3a0 tx=0x7f704800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:26.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7053400640 1 -- 172.21.15.98:0/2408244195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70480052d0 con 0x7f705c0c70e0 2024-05-27T00:21:26.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7053400640 1 -- 172.21.15.98:0/2408244195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7048005470 con 0x7f705c0c70e0 2024-05-27T00:21:26.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 -- 172.21.15.98:0/2408244195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f705c1f7400 con 0x7f705c0c70e0 2024-05-27T00:21:26.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7053400640 1 -- 172.21.15.98:0/2408244195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7048007260 con 0x7f705c0c70e0 2024-05-27T00:21:26.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 -- 172.21.15.98:0/2408244195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f705c1f79c0 con 0x7f705c0c70e0 2024-05-27T00:21:26.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.002+0000 7f7062000640 1 -- 172.21.15.98:0/2408244195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f705c075cb0 con 0x7f705c0c70e0 2024-05-27T00:21:26.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.006+0000 7f7053400640 1 -- 172.21.15.98:0/2408244195 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7048007400 con 0x7f705c0c70e0 2024-05-27T00:21:26.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.006+0000 7f7053400640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f703c0678c0 0x7f703c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:26.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.006+0000 7f7053400640 1 -- 172.21.15.98:0/2408244195 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f7048085350 con 0x7f705c0c70e0 2024-05-27T00:21:26.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.006+0000 7f705b400640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f703c0678c0 0x7f703c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:26.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.006+0000 7f705b400640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f703c0678c0 0x7f703c069d80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f704c009b40 tx=0x7f704c0039a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:26.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.010+0000 7f7053400640 1 -- 172.21.15.98:0/2408244195 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7048055540 con 0x7f705c0c70e0 2024-05-27T00:21:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.198+0000 7f7062000640 1 -- 172.21.15.98:0/2408244195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f705c072820 con 0x7f705c0c70e0 2024-05-27T00:21:26.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7053400640 1 -- 172.21.15.98:0/2408244195 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6006 (secure 0 0 0) 0x7f7048059840 con 0x7f705c0c70e0 2024-05-27T00:21:26.202 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:26.202 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:25.205553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:26.202 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-05-27T00:21:26.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 -- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f703c0678c0 msgr2=0x7f703c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:26.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f703c0678c0 0x7f703c069d80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f704c009b40 tx=0x7f704c0039a0 comp rx=0 tx=0).stop 2024-05-27T00:21:26.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 -- 172.21.15.98:0/2408244195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 msgr2=0x7f705c1f3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:26.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 0x7f705c1f3c40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f704800c3a0 tx=0x7f704800c860 comp rx=0 tx=0).stop 2024-05-27T00:21:26.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 -- 172.21.15.98:0/2408244195 shutdown_connections 2024-05-27T00:21:26.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f703c0678c0 0x7f703c069d80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:26.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f705c152520 0x7f705c145390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:26.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f705c0c70e0 0x7f705c1f3c40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:26.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 --2- 172.21.15.98:0/2408244195 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f705c073760 0x7f705c06de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:26.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 -- 172.21.15.98:0/2408244195 >> 172.21.15.98:0/2408244195 conn(0x7f705c0c9b70 msgr2=0x7f705c0c9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:26.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 -- 172.21.15.98:0/2408244195 shutdown_connections 2024-05-27T00:21:26.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:26.202+0000 7f7062000640 1 -- 172.21.15.98:0/2408244195 wait complete. 2024-05-27T00:21:26.225 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5202": { "addr": "172.21.15.98:6837/1698253956", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 1698253956, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 1698253956, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5202, "incarnation": 24, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:21:25.205553+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5202 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:21:28.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:28.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.370+0000 7f74bc200640 1 Processor -- start 2024-05-27T00:21:28.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.370+0000 7f74bc200640 1 -- start start 2024-05-27T00:21:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.370+0000 7f74bc200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cc1f0 0x7f74b41592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.370+0000 7f74bc200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f74b4073570 0x7f74b41559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.370+0000 7f74bc200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74b40c1f60 con 0x7f74b40cbe10 2024-05-27T00:21:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.370+0000 7f74bc200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f74b4159810 con 0x7f74b4073570 2024-05-27T00:21:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.370+0000 7f74bc200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f74b40c6dd0 con 0x7f74b40cc1f0 2024-05-27T00:21:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.374+0000 7f74ba200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f74b4073570 0x7f74b41559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.374+0000 7f74b8e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cc1f0 0x7f74b41592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.374+0000 7f74b9800640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f74b40cbe10 0x7f74b40c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:50362/0 (socket says 172.21.15.98:50362) 2024-05-27T00:21:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.374+0000 7f74b9800640 1 -- 172.21.15.98:0/619422139 learned_addr learned my addr 172.21.15.98:0/619422139 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.374+0000 7f74ba200640 1 -- 172.21.15.98:0/619422139 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cc1f0 msgr2=0x7f74b41592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:28.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.374+0000 7f74ba200640 1 --2- 172.21.15.98:0/619422139 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cc1f0 0x7f74b41592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:28.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.374+0000 7f74ba200640 1 -- 172.21.15.98:0/619422139 >> v1:172.21.15.204:6789/0 conn(0x7f74b40cbe10 legacy=0x7f74b40c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:28.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.374+0000 7f74ba200640 1 -- 172.21.15.98:0/619422139 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b40c70b0 con 0x7f74b4073570 2024-05-27T00:21:28.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.374+0000 7f74b8e00640 1 --2- 172.21.15.98:0/619422139 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cc1f0 0x7f74b41592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:21:28.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.374+0000 7f74ba200640 1 --2- 172.21.15.98:0/619422139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b4073570 0x7f74b41559e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0097c0 tx=0x7f74ac01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=880a2daa0851883a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:28.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74b3e00640 1 -- 172.21.15.98:0/619422139 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac0b5070 con 0x7f74b4073570 2024-05-27T00:21:28.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74b3e00640 1 -- 172.21.15.98:0/619422139 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74ac0a8990 con 0x7f74b4073570 2024-05-27T00:21:28.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74b3e00640 1 -- 172.21.15.98:0/619422139 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac0a8d60 con 0x7f74b4073570 2024-05-27T00:21:28.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 -- 172.21.15.98:0/619422139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b4073570 msgr2=0x7f74b41559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:28.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 --2- 172.21.15.98:0/619422139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b4073570 0x7f74b41559e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0097c0 tx=0x7f74ac01ec90 comp rx=0 tx=0).stop 2024-05-27T00:21:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 -- 172.21.15.98:0/619422139 shutdown_connections 2024-05-27T00:21:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 --2- 172.21.15.98:0/619422139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b4073570 0x7f74b41559e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 --2- 172.21.15.98:0/619422139 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cc1f0 0x7f74b41592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 -- 172.21.15.98:0/619422139 >> 172.21.15.98:0/619422139 conn(0x7f74b4067530 msgr2=0x7f74b406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 -- 172.21.15.98:0/619422139 shutdown_connections 2024-05-27T00:21:28.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 -- 172.21.15.98:0/619422139 wait complete. 2024-05-27T00:21:28.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 Processor -- start 2024-05-27T00:21:28.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 -- start start 2024-05-27T00:21:28.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74b4073570 0x7f74b4071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cbe10 0x7f74b41f1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b40cc1f0 0x7f74b41585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f74b40c1330 con 0x7f74b4073570 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f74b40c1050 con 0x7f74b40cc1f0 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74bc200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f74b40c79d0 con 0x7f74b40cbe10 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74b9800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74b4073570 0x7f74b4071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74b9800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74b4073570 0x7f74b4071ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37320/0 (socket says 172.21.15.98:37320) 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74b9800640 1 -- 172.21.15.98:0/2439122768 learned_addr learned my addr 172.21.15.98:0/2439122768 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.378+0000 7f74b8e00640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cbe10 0x7f74b41f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74ba200640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b40cc1f0 0x7f74b41585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b9800640 1 -- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cbe10 msgr2=0x7f74b41f1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b9800640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cbe10 0x7f74b41f1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b9800640 1 -- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b40cc1f0 msgr2=0x7f74b41585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b9800640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b40cc1f0 0x7f74b41585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b9800640 1 -- 172.21.15.98:0/2439122768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b41f06b0 con 0x7f74b4073570 2024-05-27T00:21:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74ba200640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b40cc1f0 0x7f74b41585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b9800640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74b4073570 0x7f74b4071ec0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f74a800cb40 tx=0x7f74a800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b1600640 1 -- 172.21.15.98:0/2439122768 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a8017070 con 0x7f74b4073570 2024-05-27T00:21:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b1600640 1 -- 172.21.15.98:0/2439122768 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74a800aca0 con 0x7f74b4073570 2024-05-27T00:21:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b1600640 1 -- 172.21.15.98:0/2439122768 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a8012620 con 0x7f74b4073570 2024-05-27T00:21:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74bc200640 1 -- 172.21.15.98:0/2439122768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b4072590 con 0x7f74b4073570 2024-05-27T00:21:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74bc200640 1 -- 172.21.15.98:0/2439122768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b4072b50 con 0x7f74b4073570 2024-05-27T00:21:28.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74bc200640 1 -- 172.21.15.98:0/2439122768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74b4075cb0 con 0x7f74b4073570 2024-05-27T00:21:28.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b1600640 1 -- 172.21.15.98:0/2439122768 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f74a801a070 con 0x7f74b4073570 2024-05-27T00:21:28.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b1600640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74940678c0 0x7f7494069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:28.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.382+0000 7f74b1600640 1 -- 172.21.15.98:0/2439122768 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f74a8085c80 con 0x7f74b4073570 2024-05-27T00:21:28.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.386+0000 7f74b8e00640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74940678c0 0x7f7494069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:28.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.386+0000 7f74b8e00640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74940678c0 0x7f7494069d80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74b40b1680 tx=0x7f74a0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:28.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.386+0000 7f74b1600640 1 -- 172.21.15.98:0/2439122768 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f74a8055e70 con 0x7f74b4073570 2024-05-27T00:21:28.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.578+0000 7f74bc200640 1 -- 172.21.15.98:0/2439122768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74b40caab0 con 0x7f74b4073570 2024-05-27T00:21:28.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.578+0000 7f74b1600640 1 -- 172.21.15.98:0/2439122768 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6006 (secure 0 0 0) 0x7f74a805a170 con 0x7f74b4073570 2024-05-27T00:21:28.580 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:28.581 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:25.205553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:28.581 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-05-27T00:21:28.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 -- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74940678c0 msgr2=0x7f7494069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:28.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74940678c0 0x7f7494069d80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74b40b1680 tx=0x7f74a0009190 comp rx=0 tx=0).stop 2024-05-27T00:21:28.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 -- 172.21.15.98:0/2439122768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74b4073570 msgr2=0x7f74b4071ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:28.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74b4073570 0x7f74b4071ec0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f74a800cb40 tx=0x7f74a800a060 comp rx=0 tx=0).stop 2024-05-27T00:21:28.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 -- 172.21.15.98:0/2439122768 shutdown_connections 2024-05-27T00:21:28.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f74940678c0 0x7f7494069d80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:28.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b40cc1f0 0x7f74b41585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:28.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b40cbe10 0x7f74b41f1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:28.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 --2- 172.21.15.98:0/2439122768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74b4073570 0x7f74b4071ec0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:28.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 -- 172.21.15.98:0/2439122768 >> 172.21.15.98:0/2439122768 conn(0x7f74b4067530 msgr2=0x7f74b40c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:28.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 -- 172.21.15.98:0/2439122768 shutdown_connections 2024-05-27T00:21:28.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:28.582+0000 7f74bc200640 1 -- 172.21.15.98:0/2439122768 wait complete. 2024-05-27T00:21:28.602 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:21:30.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d88c00640 1 Processor -- start 2024-05-27T00:21:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d88c00640 1 -- start start 2024-05-27T00:21:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d88c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d840cdf90 0x7f4d840c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d88c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d840c8240 0x7f4d8415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d88c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d840c8bb0 con 0x7f4d8406f760 2024-05-27T00:21:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d88c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4d84158b90 con 0x7f4d840cdf90 2024-05-27T00:21:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d88c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4d840c3640 con 0x7f4d840c8240 2024-05-27T00:21:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d81e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d840cdf90 0x7f4d840c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d81e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4d840cdf90 0x7f4d840c3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37326/0 (socket says 172.21.15.98:37326) 2024-05-27T00:21:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d81e00640 1 -- 172.21.15.98:0/3816932872 learned_addr learned my addr 172.21.15.98:0/3816932872 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d81400640 1 --2- 172.21.15.98:0/3816932872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d840c8240 0x7f4d8415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d81e00640 1 -- 172.21.15.98:0/3816932872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d840c8240 msgr2=0x7f4d8415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d81e00640 1 --2- 172.21.15.98:0/3816932872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d840c8240 0x7f4d8415bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d81e00640 1 -- 172.21.15.98:0/3816932872 >> v1:172.21.15.204:6789/0 conn(0x7f4d8406f760 legacy=0x7f4d840caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.746+0000 7f4d81e00640 1 -- 172.21.15.98:0/3816932872 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d84158560 con 0x7f4d840cdf90 2024-05-27T00:21:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d81e00640 1 --2- 172.21.15.98:0/3816932872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d840cdf90 0x7f4d840c3020 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4d74009870 tx=0x7f4d7401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbea86215497f46 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d80a00640 1 -- 172.21.15.98:0/3816932872 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d740b5070 con 0x7f4d840cdf90 2024-05-27T00:21:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d80a00640 1 -- 172.21.15.98:0/3816932872 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d740a89d0 con 0x7f4d840cdf90 2024-05-27T00:21:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d80a00640 1 -- 172.21.15.98:0/3816932872 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d740a8da0 con 0x7f4d840cdf90 2024-05-27T00:21:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d88c00640 1 -- 172.21.15.98:0/3816932872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d840cdf90 msgr2=0x7f4d840c3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d88c00640 1 --2- 172.21.15.98:0/3816932872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d840cdf90 0x7f4d840c3020 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4d74009870 tx=0x7f4d7401ed10 comp rx=0 tx=0).stop 2024-05-27T00:21:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d88c00640 1 -- 172.21.15.98:0/3816932872 shutdown_connections 2024-05-27T00:21:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d88c00640 1 --2- 172.21.15.98:0/3816932872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d840c8240 0x7f4d8415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d88c00640 1 --2- 172.21.15.98:0/3816932872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d840cdf90 0x7f4d840c3020 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d88c00640 1 -- 172.21.15.98:0/3816932872 >> 172.21.15.98:0/3816932872 conn(0x7f4d8406d680 msgr2=0x7f4d8406da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d88c00640 1 -- 172.21.15.98:0/3816932872 shutdown_connections 2024-05-27T00:21:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.750+0000 7f4d88c00640 1 -- 172.21.15.98:0/3816932872 wait complete. 2024-05-27T00:21:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d88c00640 1 Processor -- start 2024-05-27T00:21:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d88c00640 1 -- start start 2024-05-27T00:21:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d88c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d8406f760 0x7f4d8415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d88c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d840c8240 0x7f4d840c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d88c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d840cdf90 0x7f4d8414e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d88c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d84158820 con 0x7f4d8406f760 2024-05-27T00:21:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d88c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4d840c4fc0 con 0x7f4d840c8240 2024-05-27T00:21:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d88c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4d84152be0 con 0x7f4d840cdf90 2024-05-27T00:21:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d81e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d8406f760 0x7f4d8415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d81e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d8406f760 0x7f4d8415ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37332/0 (socket says 172.21.15.98:37332) 2024-05-27T00:21:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d81e00640 1 -- 172.21.15.98:0/10383523 learned_addr learned my addr 172.21.15.98:0/10383523 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d82800640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d840cdf90 0x7f4d8414e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d81400640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d840c8240 0x7f4d840c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d81e00640 1 -- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d840cdf90 msgr2=0x7f4d8414e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d81e00640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d840cdf90 0x7f4d8414e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d81e00640 1 -- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d840c8240 msgr2=0x7f4d840c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d81e00640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d840c8240 0x7f4d840c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d81e00640 1 -- 172.21.15.98:0/10383523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d840cd7d0 con 0x7f4d8406f760 2024-05-27T00:21:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d81e00640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d8406f760 0x7f4d8415ae30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4d740b3040 tx=0x7f4d7401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d7a000640 1 -- 172.21.15.98:0/10383523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d740c0070 con 0x7f4d8406f760 2024-05-27T00:21:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d7a000640 1 -- 172.21.15.98:0/10383523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d740b08f0 con 0x7f4d8406f760 2024-05-27T00:21:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d7a000640 1 -- 172.21.15.98:0/10383523 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d740b5040 con 0x7f4d8406f760 2024-05-27T00:21:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d88c00640 1 -- 172.21.15.98:0/10383523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d840cda90 con 0x7f4d8406f760 2024-05-27T00:21:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.754+0000 7f4d88c00640 1 -- 172.21.15.98:0/10383523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d84159c50 con 0x7f4d8406f760 2024-05-27T00:21:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.758+0000 7f4d88c00640 1 -- 172.21.15.98:0/10383523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d84075cb0 con 0x7f4d8406f760 2024-05-27T00:21:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.758+0000 7f4d7a000640 1 -- 172.21.15.98:0/10383523 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4d740c3070 con 0x7f4d8406f760 2024-05-27T00:21:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.758+0000 7f4d7a000640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d640678c0 0x7f4d64069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.758+0000 7f4d7a000640 1 -- 172.21.15.98:0/10383523 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f4d74122d50 con 0x7f4d8406f760 2024-05-27T00:21:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.758+0000 7f4d81400640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d640678c0 0x7f4d64069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.758+0000 7f4d81400640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d640678c0 0x7f4d64069d80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4d840c7620 tx=0x7f4d70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.762+0000 7f4d7a000640 1 -- 172.21.15.98:0/10383523 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4d74018870 con 0x7f4d8406f760 2024-05-27T00:21:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.978+0000 7f4d88c00640 1 -- 172.21.15.98:0/10383523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d840c8db0 con 0x7f4d8406f760 2024-05-27T00:21:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.978+0000 7f4d7a000640 1 -- 172.21.15.98:0/10383523 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6006 (secure 0 0 0) 0x7f4d740f36b0 con 0x7f4d8406f760 2024-05-27T00:21:30.981 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:30.981 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:25.205553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 -- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d640678c0 msgr2=0x7f4d64069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d640678c0 0x7f4d64069d80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4d840c7620 tx=0x7f4d70008040 comp rx=0 tx=0).stop 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 -- 172.21.15.98:0/10383523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d8406f760 msgr2=0x7f4d8415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d8406f760 0x7f4d8415ae30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4d740b3040 tx=0x7f4d7401bab0 comp rx=0 tx=0).stop 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 -- 172.21.15.98:0/10383523 shutdown_connections 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d840cdf90 0x7f4d8414e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4d640678c0 0x7f4d64069d80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d840c8240 0x7f4d840c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 --2- 172.21.15.98:0/10383523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d8406f760 0x7f4d8415ae30 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 -- 172.21.15.98:0/10383523 >> 172.21.15.98:0/10383523 conn(0x7f4d8406d680 msgr2=0x7f4d8415c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 -- 172.21.15.98:0/10383523 shutdown_connections 2024-05-27T00:21:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:30.982+0000 7f4d88c00640 1 -- 172.21.15.98:0/10383523 wait complete. 2024-05-27T00:21:31.003 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:21:33.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:33.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.146+0000 7f6d78400640 1 Processor -- start 2024-05-27T00:21:33.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.146+0000 7f6d78400640 1 -- start start 2024-05-27T00:21:33.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d78400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f6d70154830 0x7f6d7015b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:33.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d78400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d700cbbb0 0x7f6d700cc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d78400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700c68c0 0x7f6d700c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d78400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6d700c2b30 con 0x7f6d700cbbb0 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d78400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f6d700c26a0 con 0x7f6d70154830 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d78400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6d700c1a90 con 0x7f6d700c68c0 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d75000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d700cbbb0 0x7f6d700cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d75000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6d700cbbb0 0x7f6d700cc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37348/0 (socket says 172.21.15.98:37348) 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d75000640 1 -- 172.21.15.98:0/2776499225 learned_addr learned my addr 172.21.15.98:0/2776499225 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d75a00640 1 --2- 172.21.15.98:0/2776499225 >> v2:172.21.15.204:3300/0 conn(0x7f6d70154830 0x7f6d7015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d76400640 1 --2- 172.21.15.98:0/2776499225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700c68c0 0x7f6d700c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d75000640 1 -- 172.21.15.98:0/2776499225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700c68c0 msgr2=0x7f6d700c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d75000640 1 --2- 172.21.15.98:0/2776499225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700c68c0 0x7f6d700c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d75000640 1 -- 172.21.15.98:0/2776499225 >> v2:172.21.15.204:3300/0 conn(0x7f6d70154830 msgr2=0x7f6d7015b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:33.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d75000640 1 --2- 172.21.15.98:0/2776499225 >> v2:172.21.15.204:3300/0 conn(0x7f6d70154830 0x7f6d7015b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d75000640 1 -- 172.21.15.98:0/2776499225 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d700c1d70 con 0x7f6d700cbbb0 2024-05-27T00:21:33.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d75a00640 1 --2- 172.21.15.98:0/2776499225 >> v2:172.21.15.204:3300/0 conn(0x7f6d70154830 0x7f6d7015b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:33.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.150+0000 7f6d76400640 1 --2- 172.21.15.98:0/2776499225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700c68c0 0x7f6d700c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:33.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d75000640 1 --2- 172.21.15.98:0/2776499225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700cbbb0 0x7f6d700cc010 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6d700cc760 tx=0x7f6d5c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f85900ff554369c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d6fe00640 1 -- 172.21.15.98:0/2776499225 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c0b1070 con 0x7f6d700cbbb0 2024-05-27T00:21:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d6fe00640 1 -- 172.21.15.98:0/2776499225 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d5c0ac630 con 0x7f6d700cbbb0 2024-05-27T00:21:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d6fe00640 1 -- 172.21.15.98:0/2776499225 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c0a8050 con 0x7f6d700cbbb0 2024-05-27T00:21:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d78400640 1 -- 172.21.15.98:0/2776499225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700cbbb0 msgr2=0x7f6d700cc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d78400640 1 --2- 172.21.15.98:0/2776499225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700cbbb0 0x7f6d700cc010 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6d700cc760 tx=0x7f6d5c01c520 comp rx=0 tx=0).stop 2024-05-27T00:21:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d78400640 1 -- 172.21.15.98:0/2776499225 shutdown_connections 2024-05-27T00:21:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d78400640 1 --2- 172.21.15.98:0/2776499225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700c68c0 0x7f6d700c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d78400640 1 --2- 172.21.15.98:0/2776499225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700cbbb0 0x7f6d700cc010 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d78400640 1 --2- 172.21.15.98:0/2776499225 >> v2:172.21.15.204:3300/0 conn(0x7f6d70154830 0x7f6d7015b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d78400640 1 -- 172.21.15.98:0/2776499225 >> 172.21.15.98:0/2776499225 conn(0x7f6d70067530 msgr2=0x7f6d7006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d78400640 1 -- 172.21.15.98:0/2776499225 shutdown_connections 2024-05-27T00:21:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.154+0000 7f6d78400640 1 -- 172.21.15.98:0/2776499225 wait complete. 2024-05-27T00:21:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d78400640 1 Processor -- start 2024-05-27T00:21:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d78400640 1 -- start start 2024-05-27T00:21:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d78400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700c68c0 0x7f6d7015d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d78400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700cbbb0 0x7f6d70156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d78400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d70154830 0x7f6d701571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d78400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7015e040 con 0x7f6d700c68c0 2024-05-27T00:21:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d78400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7015dec0 con 0x7f6d70154830 2024-05-27T00:21:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d78400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6d7015e1c0 con 0x7f6d700cbbb0 2024-05-27T00:21:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d75a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700c68c0 0x7f6d7015d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d75a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700c68c0 0x7f6d7015d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37360/0 (socket says 172.21.15.98:37360) 2024-05-27T00:21:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d75a00640 1 -- 172.21.15.98:0/971965005 learned_addr learned my addr 172.21.15.98:0/971965005 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d75000640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700cbbb0 0x7f6d70156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d76400640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d70154830 0x7f6d701571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d75a00640 1 -- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700cbbb0 msgr2=0x7f6d70156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d75a00640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700cbbb0 0x7f6d70156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d75a00640 1 -- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d70154830 msgr2=0x7f6d701571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d75a00640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d70154830 0x7f6d701571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d75a00640 1 -- 172.21.15.98:0/971965005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7014c3a0 con 0x7f6d700c68c0 2024-05-27T00:21:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d75a00640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700c68c0 0x7f6d7015d630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400d830 tx=0x7f6d6400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d6d600640 1 -- 172.21.15.98:0/971965005 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64018020 con 0x7f6d700c68c0 2024-05-27T00:21:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d6d600640 1 -- 172.21.15.98:0/971965005 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d640096c0 con 0x7f6d700c68c0 2024-05-27T00:21:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d6d600640 1 -- 172.21.15.98:0/971965005 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64009ab0 con 0x7f6d700c68c0 2024-05-27T00:21:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d78400640 1 -- 172.21.15.98:0/971965005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d7014c660 con 0x7f6d700c68c0 2024-05-27T00:21:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.158+0000 7f6d78400640 1 -- 172.21.15.98:0/971965005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d70156460 con 0x7f6d700c68c0 2024-05-27T00:21:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.162+0000 7f6d78400640 1 -- 172.21.15.98:0/971965005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d70075cb0 con 0x7f6d700c68c0 2024-05-27T00:21:33.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.162+0000 7f6d6d600640 1 -- 172.21.15.98:0/971965005 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6d6401b020 con 0x7f6d700c68c0 2024-05-27T00:21:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.162+0000 7f6d6d600640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6d500678c0 0x7f6d50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.162+0000 7f6d6d600640 1 -- 172.21.15.98:0/971965005 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f6d640864d0 con 0x7f6d700c68c0 2024-05-27T00:21:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.166+0000 7f6d75000640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6d500678c0 0x7f6d50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.166+0000 7f6d75000640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6d500678c0 0x7f6d50069d80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c0ae000 tx=0x7f6d5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:33.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.170+0000 7f6d6d600640 1 -- 172.21.15.98:0/971965005 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f6d64022020 con 0x7f6d700c68c0 2024-05-27T00:21:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.370+0000 7f6d78400640 1 -- 172.21.15.98:0/971965005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d701581d0 con 0x7f6d700c68c0 2024-05-27T00:21:33.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d6d600640 1 -- 172.21.15.98:0/971965005 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6006 (secure 0 0 0) 0x7f6d64056740 con 0x7f6d700c68c0 2024-05-27T00:21:33.374 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:33.374 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:25.205553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-05-27T00:21:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 -- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6d500678c0 msgr2=0x7f6d50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6d500678c0 0x7f6d50069d80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c0ae000 tx=0x7f6d5c009040 comp rx=0 tx=0).stop 2024-05-27T00:21:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 -- 172.21.15.98:0/971965005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700c68c0 msgr2=0x7f6d7015d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700c68c0 0x7f6d7015d630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400d830 tx=0x7f6d6400dcf0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 -- 172.21.15.98:0/971965005 shutdown_connections 2024-05-27T00:21:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6d500678c0 0x7f6d50069d80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6d70154830 0x7f6d701571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d700cbbb0 0x7f6d70156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 --2- 172.21.15.98:0/971965005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d700c68c0 0x7f6d7015d630 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 -- 172.21.15.98:0/971965005 >> 172.21.15.98:0/971965005 conn(0x7f6d70067530 msgr2=0x7f6d7006b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 -- 172.21.15.98:0/971965005 shutdown_connections 2024-05-27T00:21:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:33.374+0000 7f6d78400640 1 -- 172.21.15.98:0/971965005 wait complete. 2024-05-27T00:21:33.403 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:21:35.404 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faefd200640 1 Processor -- start 2024-05-27T00:21:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faefd200640 1 -- start start 2024-05-27T00:21:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faefd200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef80cc1f0 0x7faef81592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faefd200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7faef8073570 0x7faef81559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faefd200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faef80c1f60 con 0x7faef80cbe10 2024-05-27T00:21:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faefd200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7faef8159810 con 0x7faef8073570 2024-05-27T00:21:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faefd200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faef80c6dd0 con 0x7faef80cc1f0 2024-05-27T00:21:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faeefe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef80cc1f0 0x7faef81592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faef6800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7faef8073570 0x7faef81559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faeefe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef80cc1f0 0x7faef81592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:39234/0 (socket says 172.21.15.98:39234) 2024-05-27T00:21:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faeefe00640 1 -- 172.21.15.98:0/1122885464 learned_addr learned my addr 172.21.15.98:0/1122885464 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faef5e00640 1 --1- 172.21.15.98:0/1122885464 >> v1:172.21.15.204:6789/0 conn(0x7faef80cbe10 0x7faef80c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:39604/0 (socket says 172.21.15.98:39604) 2024-05-27T00:21:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faeefe00640 1 -- 172.21.15.98:0/1122885464 >> v2:172.21.15.204:3300/0 conn(0x7faef8073570 msgr2=0x7faef81559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faeefe00640 1 --2- 172.21.15.98:0/1122885464 >> v2:172.21.15.204:3300/0 conn(0x7faef8073570 0x7faef81559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faeefe00640 1 -- 172.21.15.98:0/1122885464 >> v1:172.21.15.204:6789/0 conn(0x7faef80cbe10 legacy=0x7faef80c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.546+0000 7faeefe00640 1 -- 172.21.15.98:0/1122885464 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef80c70b0 con 0x7faef80cc1f0 2024-05-27T00:21:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.550+0000 7faeefe00640 1 --2- 172.21.15.98:0/1122885464 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef80cc1f0 0x7faef81592d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faee0009870 tx=0x7faee001ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=858dcb0029e00289 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.550+0000 7faef5400640 1 -- 172.21.15.98:0/1122885464 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee00b5070 con 0x7faef80cc1f0 2024-05-27T00:21:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.550+0000 7faef5400640 1 -- 172.21.15.98:0/1122885464 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faee00a8a50 con 0x7faef80cc1f0 2024-05-27T00:21:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.550+0000 7faef5400640 1 -- 172.21.15.98:0/1122885464 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee00a8e20 con 0x7faef80cc1f0 2024-05-27T00:21:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.554+0000 7faefd200640 1 -- 172.21.15.98:0/1122885464 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef80cc1f0 msgr2=0x7faef81592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.554+0000 7faefd200640 1 --2- 172.21.15.98:0/1122885464 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef80cc1f0 0x7faef81592d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faee0009870 tx=0x7faee001ed50 comp rx=0 tx=0).stop 2024-05-27T00:21:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.554+0000 7faefd200640 1 -- 172.21.15.98:0/1122885464 shutdown_connections 2024-05-27T00:21:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.554+0000 7faefd200640 1 --2- 172.21.15.98:0/1122885464 >> v2:172.21.15.204:3300/0 conn(0x7faef8073570 0x7faef81559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.554+0000 7faefd200640 1 --2- 172.21.15.98:0/1122885464 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef80cc1f0 0x7faef81592d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.554+0000 7faefd200640 1 -- 172.21.15.98:0/1122885464 >> 172.21.15.98:0/1122885464 conn(0x7faef8067530 msgr2=0x7faef806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.554+0000 7faefd200640 1 -- 172.21.15.98:0/1122885464 shutdown_connections 2024-05-27T00:21:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.554+0000 7faefd200640 1 -- 172.21.15.98:0/1122885464 wait complete. 2024-05-27T00:21:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.554+0000 7faefd200640 1 Processor -- start 2024-05-27T00:21:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.554+0000 7faefd200640 1 -- start start 2024-05-27T00:21:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faefd200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef8073570 0x7faef81f0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faefd200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faef80cbe10 0x7faef81f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faefd200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef80cc1f0 0x7faef81570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faefd200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faef80c1330 con 0x7faef80cc1f0 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faefd200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faef80c1050 con 0x7faef80cbe10 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faefd200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faef80c79d0 con 0x7faef8073570 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faef6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef80cc1f0 0x7faef81570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faef6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef80cc1f0 0x7faef81570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41562/0 (socket says 172.21.15.98:41562) 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faef6800640 1 -- 172.21.15.98:0/28757999 learned_addr learned my addr 172.21.15.98:0/28757999 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faef5e00640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef8073570 0x7faef81f0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faeefe00640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faef80cbe10 0x7faef81f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faef6800640 1 -- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef8073570 msgr2=0x7faef81f0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faef6800640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef8073570 0x7faef81f0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faef6800640 1 -- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faef80cbe10 msgr2=0x7faef81f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faef6800640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faef80cbe10 0x7faef81f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faef6800640 1 -- 172.21.15.98:0/28757999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef806f620 con 0x7faef80cc1f0 2024-05-27T00:21:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faef6800640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef80cc1f0 0x7faef81570b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faef0009e30 tx=0x7faef000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faeee000640 1 -- 172.21.15.98:0/28757999 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0018020 con 0x7faef80cc1f0 2024-05-27T00:21:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faeee000640 1 -- 172.21.15.98:0/28757999 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef0014480 con 0x7faef80cc1f0 2024-05-27T00:21:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faeee000640 1 -- 172.21.15.98:0/28757999 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0014870 con 0x7faef80cc1f0 2024-05-27T00:21:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faefd200640 1 -- 172.21.15.98:0/28757999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef80c8b40 con 0x7faef80cc1f0 2024-05-27T00:21:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faefd200640 1 -- 172.21.15.98:0/28757999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef80c3d80 con 0x7faef80cc1f0 2024-05-27T00:21:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faefd200640 1 -- 172.21.15.98:0/28757999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef8075cb0 con 0x7faef80cc1f0 2024-05-27T00:21:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.558+0000 7faeee000640 1 -- 172.21.15.98:0/28757999 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faef001b020 con 0x7faef80cc1f0 2024-05-27T00:21:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.562+0000 7faeee000640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faed40678f0 0x7faed4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.562+0000 7faeee000640 1 -- 172.21.15.98:0/28757999 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7faef0086c50 con 0x7faef80cc1f0 2024-05-27T00:21:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.562+0000 7faef5e00640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faed40678f0 0x7faed4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:35.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.562+0000 7faef5e00640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faed40678f0 0x7faed4069db0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faee8000f70 tx=0x7faee8007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:35.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.566+0000 7faeee000640 1 -- 172.21.15.98:0/28757999 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7faef0022030 con 0x7faef80cc1f0 2024-05-27T00:21:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.762+0000 7faefd200640 1 -- 172.21.15.98:0/28757999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faef8152190 con 0x7faef80cc1f0 2024-05-27T00:21:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.762+0000 7faeee000640 1 -- 172.21.15.98:0/28757999 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6006 (secure 0 0 0) 0x7faef0010040 con 0x7faef80cc1f0 2024-05-27T00:21:35.766 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:35.766 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:25.205553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 24 2024-05-27T00:21:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 -- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faed40678f0 msgr2=0x7faed4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faed40678f0 0x7faed4069db0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faee8000f70 tx=0x7faee8007e90 comp rx=0 tx=0).stop 2024-05-27T00:21:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 -- 172.21.15.98:0/28757999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef80cc1f0 msgr2=0x7faef81570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef80cc1f0 0x7faef81570b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faef0009e30 tx=0x7faef000c490 comp rx=0 tx=0).stop 2024-05-27T00:21:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 -- 172.21.15.98:0/28757999 shutdown_connections 2024-05-27T00:21:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faef80cc1f0 0x7faef81570b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faed40678f0 0x7faed4069db0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faef80cbe10 0x7faef81f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 --2- 172.21.15.98:0/28757999 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faef8073570 0x7faef81f0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 -- 172.21.15.98:0/28757999 >> 172.21.15.98:0/28757999 conn(0x7faef8067530 msgr2=0x7faef8152f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 -- 172.21.15.98:0/28757999 shutdown_connections 2024-05-27T00:21:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:35.766+0000 7faefd200640 1 -- 172.21.15.98:0/28757999 wait complete. 2024-05-27T00:21:35.790 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:21:37.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.934+0000 7f1cc0a00640 1 Processor -- start 2024-05-27T00:21:37.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.934+0000 7f1cc0a00640 1 -- start start 2024-05-27T00:21:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cc0a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1cbc0731f0 0x7f1cbc0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cc0a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 0x7f1cbc151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cc0a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cbc0c17d0 con 0x7f1cbc0c79f0 2024-05-27T00:21:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cc0a00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f1cbc071990 con 0x7f1cbc0731f0 2024-05-27T00:21:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cc0a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1cbc070e10 con 0x7f1cbc0735d0 2024-05-27T00:21:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cb9e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1cbc0731f0 0x7f1cbc0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cb1400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 0x7f1cbc151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cb1400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 0x7f1cbc151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:39244/0 (socket says 172.21.15.98:39244) 2024-05-27T00:21:37.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cb1400640 1 -- 172.21.15.98:0/3380301313 learned_addr learned my addr 172.21.15.98:0/3380301313 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:37.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cb1400640 1 -- 172.21.15.98:0/3380301313 >> v2:172.21.15.204:3300/0 conn(0x7f1cbc0731f0 msgr2=0x7f1cbc0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:37.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cb1400640 1 --2- 172.21.15.98:0/3380301313 >> v2:172.21.15.204:3300/0 conn(0x7f1cbc0731f0 0x7f1cbc0c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:37.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cb1400640 1 -- 172.21.15.98:0/3380301313 >> v1:172.21.15.204:6789/0 conn(0x7f1cbc0c79f0 legacy=0x7f1cbc157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:37.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.938+0000 7f1cb1400640 1 -- 172.21.15.98:0/3380301313 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cbc0710f0 con 0x7f1cbc0735d0 2024-05-27T00:21:37.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cb1400640 1 --2- 172.21.15.98:0/3380301313 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 0x7f1cbc151430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009870 tx=0x7f1cac01ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a89e41a3b95dc09 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cb9400640 1 -- 172.21.15.98:0/3380301313 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac0b5070 con 0x7f1cbc0735d0 2024-05-27T00:21:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cb9400640 1 -- 172.21.15.98:0/3380301313 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cac0a8a50 con 0x7f1cbc0735d0 2024-05-27T00:21:37.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cb9400640 1 -- 172.21.15.98:0/3380301313 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cac0a8e20 con 0x7f1cbc0735d0 2024-05-27T00:21:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cc0a00640 1 -- 172.21.15.98:0/3380301313 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 msgr2=0x7f1cbc151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cc0a00640 1 --2- 172.21.15.98:0/3380301313 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 0x7f1cbc151430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009870 tx=0x7f1cac01ed50 comp rx=0 tx=0).stop 2024-05-27T00:21:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cc0a00640 1 -- 172.21.15.98:0/3380301313 shutdown_connections 2024-05-27T00:21:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cc0a00640 1 --2- 172.21.15.98:0/3380301313 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 0x7f1cbc151430 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cc0a00640 1 --2- 172.21.15.98:0/3380301313 >> v2:172.21.15.204:3300/0 conn(0x7f1cbc0731f0 0x7f1cbc0c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cc0a00640 1 -- 172.21.15.98:0/3380301313 >> 172.21.15.98:0/3380301313 conn(0x7f1cbc067530 msgr2=0x7f1cbc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cc0a00640 1 -- 172.21.15.98:0/3380301313 shutdown_connections 2024-05-27T00:21:37.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cc0a00640 1 -- 172.21.15.98:0/3380301313 wait complete. 2024-05-27T00:21:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.942+0000 7f1cc0a00640 1 Processor -- start 2024-05-27T00:21:37.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cc0a00640 1 -- start start 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cc0a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cbc0731f0 0x7f1cbc158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cc0a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 0x7f1cbc1f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cc0a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1cbc0c79f0 0x7f1cbc151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cc0a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc0c8400 con 0x7f1cbc0731f0 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cc0a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc0716b0 con 0x7f1cbc0c79f0 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cc0a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1cbc0c13f0 con 0x7f1cbc0735d0 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cbc0731f0 0x7f1cbc158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cbc0731f0 0x7f1cbc158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41564/0 (socket says 172.21.15.98:41564) 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb9e00640 1 -- 172.21.15.98:0/2511708984 learned_addr learned my addr 172.21.15.98:0/2511708984 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb1400640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 0x7f1cbc1f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:37.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cba800640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1cbc0c79f0 0x7f1cbc151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb9e00640 1 -- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 msgr2=0x7f1cbc1f18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb9e00640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 0x7f1cbc1f18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb9e00640 1 -- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1cbc0c79f0 msgr2=0x7f1cbc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb9e00640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1cbc0c79f0 0x7f1cbc151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb9e00640 1 -- 172.21.15.98:0/2511708984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cbc15b070 con 0x7f1cbc0731f0 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb9e00640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cbc0731f0 0x7f1cbc158ff0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400e960 tx=0x7f1ca400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb2a00640 1 -- 172.21.15.98:0/2511708984 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4017070 con 0x7f1cbc0731f0 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb2a00640 1 -- 172.21.15.98:0/2511708984 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca400ca00 con 0x7f1cbc0731f0 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cb2a00640 1 -- 172.21.15.98:0/2511708984 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca400cdb0 con 0x7f1cbc0731f0 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cc0a00640 1 -- 172.21.15.98:0/2511708984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cbc15b350 con 0x7f1cbc0731f0 2024-05-27T00:21:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cc0a00640 1 -- 172.21.15.98:0/2511708984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cbc1587f0 con 0x7f1cbc0731f0 2024-05-27T00:21:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.946+0000 7f1cc0a00640 1 -- 172.21.15.98:0/2511708984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cbc075cb0 con 0x7f1cbc0731f0 2024-05-27T00:21:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.950+0000 7f1cb2a00640 1 -- 172.21.15.98:0/2511708984 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1ca401a070 con 0x7f1cbc0731f0 2024-05-27T00:21:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.950+0000 7f1cb2a00640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1c980678c0 0x7f1c98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.950+0000 7f1cb2a00640 1 -- 172.21.15.98:0/2511708984 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f1ca4086350 con 0x7f1cbc0731f0 2024-05-27T00:21:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.950+0000 7f1cb1400640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1c980678c0 0x7f1c98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:37.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.950+0000 7f1cb1400640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1c980678c0 0x7f1c98069d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0b2000 tx=0x7f1cac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:37.954+0000 7f1cb2a00640 1 -- 172.21.15.98:0/2511708984 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1ca4056b90 con 0x7f1cbc0731f0 2024-05-27T00:21:38.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 -- 172.21.15.98:0/2511708984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cbc158510 con 0x7f1cbc0731f0 2024-05-27T00:21:38.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cb2a00640 1 -- 172.21.15.98:0/2511708984 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6010 (secure 0 0 0) 0x7f1ca4056490 con 0x7f1cbc0731f0 2024-05-27T00:21:38.143 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:38.143 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:36.756076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:38.143 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 25 2024-05-27T00:21:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 -- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1c980678c0 msgr2=0x7f1c98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1c980678c0 0x7f1c98069d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0b2000 tx=0x7f1cac008040 comp rx=0 tx=0).stop 2024-05-27T00:21:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 -- 172.21.15.98:0/2511708984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cbc0731f0 msgr2=0x7f1cbc158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cbc0731f0 0x7f1cbc158ff0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400e960 tx=0x7f1ca400ee20 comp rx=0 tx=0).stop 2024-05-27T00:21:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 -- 172.21.15.98:0/2511708984 shutdown_connections 2024-05-27T00:21:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1c980678c0 0x7f1c98069d80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1cbc0c79f0 0x7f1cbc151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1cbc0735d0 0x7f1cbc1f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 --2- 172.21.15.98:0/2511708984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1cbc0731f0 0x7f1cbc158ff0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 -- 172.21.15.98:0/2511708984 >> 172.21.15.98:0/2511708984 conn(0x7f1cbc067530 msgr2=0x7f1cbc14b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:38.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 -- 172.21.15.98:0/2511708984 shutdown_connections 2024-05-27T00:21:38.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:38.142+0000 7f1cc0a00640 1 -- 172.21.15.98:0/2511708984 wait complete. 2024-05-27T00:21:38.164 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5202": { "addr": "172.21.15.98:6837/1698253956", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 1698253956, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 1698253956, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5202, "incarnation": 24, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:21:36.756076+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5202 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:21:40.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:40.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa94ba00640 1 Processor -- start 2024-05-27T00:21:40.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa94ba00640 1 -- start start 2024-05-27T00:21:40.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa94ba00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 0x7fa9440c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:40.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa94ba00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9440c79f0 0x7fa944157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:40.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa94ba00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9440c17d0 con 0x7fa9440735d0 2024-05-27T00:21:40.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa94ba00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa944071990 con 0x7fa9440c79f0 2024-05-27T00:21:40.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa94ba00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa944070e10 con 0x7fa9440731f0 2024-05-27T00:21:40.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa949a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9440c79f0 0x7fa944157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:40.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa949a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9440c79f0 0x7fa944157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41578/0 (socket says 172.21.15.98:41578) 2024-05-27T00:21:40.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa949a00640 1 -- 172.21.15.98:0/661814467 learned_addr learned my addr 172.21.15.98:0/661814467 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:40.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.314+0000 7fa949000640 1 --2- 172.21.15.98:0/661814467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 0x7fa9440c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:40.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa949a00640 1 -- 172.21.15.98:0/661814467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 msgr2=0x7fa9440c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:40.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa949a00640 1 --2- 172.21.15.98:0/661814467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 0x7fa9440c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:40.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa949a00640 1 -- 172.21.15.98:0/661814467 >> v1:172.21.15.204:6789/0 conn(0x7fa9440735d0 legacy=0x7fa944151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:40.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa949a00640 1 -- 172.21.15.98:0/661814467 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9440710f0 con 0x7fa9440c79f0 2024-05-27T00:21:40.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa949000640 1 --2- 172.21.15.98:0/661814467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 0x7fa9440c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa949a00640 1 --2- 172.21.15.98:0/661814467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440c79f0 0x7fa944157970 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00d150 tx=0x7fa93c021420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f25829572ab03c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:40.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa943e00640 1 -- 172.21.15.98:0/661814467 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c0b6070 con 0x7fa9440c79f0 2024-05-27T00:21:40.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa943e00640 1 -- 172.21.15.98:0/661814467 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa93c00b2d0 con 0x7fa9440c79f0 2024-05-27T00:21:40.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa943e00640 1 -- 172.21.15.98:0/661814467 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c00b480 con 0x7fa9440c79f0 2024-05-27T00:21:40.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa94ba00640 1 -- 172.21.15.98:0/661814467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440c79f0 msgr2=0x7fa944157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa94ba00640 1 --2- 172.21.15.98:0/661814467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440c79f0 0x7fa944157970 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00d150 tx=0x7fa93c021420 comp rx=0 tx=0).stop 2024-05-27T00:21:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa94ba00640 1 -- 172.21.15.98:0/661814467 shutdown_connections 2024-05-27T00:21:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa94ba00640 1 --2- 172.21.15.98:0/661814467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440c79f0 0x7fa944157970 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa94ba00640 1 --2- 172.21.15.98:0/661814467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 0x7fa9440c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.318+0000 7fa94ba00640 1 -- 172.21.15.98:0/661814467 >> 172.21.15.98:0/661814467 conn(0x7fa944067530 msgr2=0x7fa94406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.322+0000 7fa94ba00640 1 -- 172.21.15.98:0/661814467 shutdown_connections 2024-05-27T00:21:40.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.322+0000 7fa94ba00640 1 -- 172.21.15.98:0/661814467 wait complete. 2024-05-27T00:21:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.322+0000 7fa94ba00640 1 Processor -- start 2024-05-27T00:21:40.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.322+0000 7fa94ba00640 1 -- start start 2024-05-27T00:21:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.322+0000 7fa94ba00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 0x7fa944147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa94ba00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440735d0 0x7fa94414c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa94ba00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9440c79f0 0x7fa94414ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa94ba00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9440c8400 con 0x7fa9440735d0 2024-05-27T00:21:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa94ba00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa9440716b0 con 0x7fa9440c79f0 2024-05-27T00:21:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa94ba00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa9440c13f0 con 0x7fa9440731f0 2024-05-27T00:21:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa93be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440735d0 0x7fa94414c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa93be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440735d0 0x7fa94414c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41592/0 (socket says 172.21.15.98:41592) 2024-05-27T00:21:40.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa93be00640 1 -- 172.21.15.98:0/2275599394 learned_addr learned my addr 172.21.15.98:0/2275599394 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa949000640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 0x7fa944147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa949a00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9440c79f0 0x7fa94414ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa93be00640 1 -- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 msgr2=0x7fa944147060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa93be00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 0x7fa944147060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa93be00640 1 -- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9440c79f0 msgr2=0x7fa94414ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa93be00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9440c79f0 0x7fa94414ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa93be00640 1 -- 172.21.15.98:0/2275599394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa944153070 con 0x7fa9440735d0 2024-05-27T00:21:40.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa949a00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9440c79f0 0x7fa94414ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:40.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa93be00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440735d0 0x7fa94414c990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00c460 tx=0x7fa92c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:40.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa941600640 1 -- 172.21.15.98:0/2275599394 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c016020 con 0x7fa9440735d0 2024-05-27T00:21:40.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa941600640 1 -- 172.21.15.98:0/2275599394 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa92c0052d0 con 0x7fa9440735d0 2024-05-27T00:21:40.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa94ba00640 1 -- 172.21.15.98:0/2275599394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa944153350 con 0x7fa9440735d0 2024-05-27T00:21:40.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa941600640 1 -- 172.21.15.98:0/2275599394 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c007260 con 0x7fa9440735d0 2024-05-27T00:21:40.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa94ba00640 1 -- 172.21.15.98:0/2275599394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9441ed490 con 0x7fa9440735d0 2024-05-27T00:21:40.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa94ba00640 1 -- 172.21.15.98:0/2275599394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa944075cb0 con 0x7fa9440735d0 2024-05-27T00:21:40.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.326+0000 7fa941600640 1 -- 172.21.15.98:0/2275599394 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa92c007400 con 0x7fa9440735d0 2024-05-27T00:21:40.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.330+0000 7fa941600640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9200678c0 0x7fa920069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:40.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.330+0000 7fa941600640 1 -- 172.21.15.98:0/2275599394 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7fa92c085770 con 0x7fa9440735d0 2024-05-27T00:21:40.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.330+0000 7fa949000640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9200678c0 0x7fa920069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:40.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.330+0000 7fa949000640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9200678c0 0x7fa920069d80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa94414e6d0 tx=0x7fa934008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:40.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.334+0000 7fa941600640 1 -- 172.21.15.98:0/2275599394 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa92c020020 con 0x7fa9440735d0 2024-05-27T00:21:40.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.530+0000 7fa94ba00640 1 -- 172.21.15.98:0/2275599394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa944158510 con 0x7fa9440735d0 2024-05-27T00:21:40.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.530+0000 7fa941600640 1 -- 172.21.15.98:0/2275599394 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6010 (secure 0 0 0) 0x7fa92c005480 con 0x7fa9440735d0 2024-05-27T00:21:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:40.533 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:36.756076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:40.533 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 25 2024-05-27T00:21:40.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 -- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9200678c0 msgr2=0x7fa920069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:40.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9200678c0 0x7fa920069d80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa94414e6d0 tx=0x7fa934008040 comp rx=0 tx=0).stop 2024-05-27T00:21:40.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 -- 172.21.15.98:0/2275599394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440735d0 msgr2=0x7fa94414c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:40.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440735d0 0x7fa94414c990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00c460 tx=0x7fa92c00c920 comp rx=0 tx=0).stop 2024-05-27T00:21:40.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 -- 172.21.15.98:0/2275599394 shutdown_connections 2024-05-27T00:21:40.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9200678c0 0x7fa920069d80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:40.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9440c79f0 0x7fa94414ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:40.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9440735d0 0x7fa94414c990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:40.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 --2- 172.21.15.98:0/2275599394 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9440731f0 0x7fa944147060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:40.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 -- 172.21.15.98:0/2275599394 >> 172.21.15.98:0/2275599394 conn(0x7fa944067530 msgr2=0x7fa944151200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:40.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 -- 172.21.15.98:0/2275599394 shutdown_connections 2024-05-27T00:21:40.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:40.534+0000 7fa94ba00640 1 -- 172.21.15.98:0/2275599394 wait complete. 2024-05-27T00:21:40.553 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:21:42.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:42.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa751000640 1 Processor -- start 2024-05-27T00:21:42.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa751000640 1 -- start start 2024-05-27T00:21:42.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa751000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c158290 0x7fa74c15b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:42.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa751000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa74c0c90b0 con 0x7fa74c0c3a60 2024-05-27T00:21:42.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa751000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa74c0c9350 con 0x7fa74c151950 2024-05-27T00:21:42.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa751000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa74c15bad0 con 0x7fa74c158290 2024-05-27T00:21:42.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa743e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa74c0c3a60 0x7fa74c0c8070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38432/0 (socket says 172.21.15.98:38432) 2024-05-27T00:21:42.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa743e00640 1 -- 172.21.15.98:0/2793558764 learned_addr learned my addr 172.21.15.98:0/2793558764 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:42.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74be00640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c158290 0x7fa74c15b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:42.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2374160061 0 0) 0x7fa74c0c90b0 con 0x7fa74c0c3a60 2024-05-27T00:21:42.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa730003530 con 0x7fa74c0c3a60 2024-05-27T00:21:42.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4059299254 0 0) 0x7fa74c0c9350 con 0x7fa74c151950 2024-05-27T00:21:42.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa74c0c90b0 con 0x7fa74c151950 2024-05-27T00:21:42.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2805332246 0 0) 0x7fa730003530 con 0x7fa74c0c3a60 2024-05-27T00:21:42.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c158290 msgr2=0x7fa74c15b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:42.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c158290 0x7fa74c15b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:42.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 >> v1:172.21.15.204:6789/0 conn(0x7fa74c151950 legacy=0x7fa74c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:42.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74c1577b0 con 0x7fa74c0c3a60 2024-05-27T00:21:42.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fa734001680 con 0x7fa74c0c3a60 2024-05-27T00:21:42.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 >> v1:172.21.15.98:6789/0 conn(0x7fa74c0c3a60 legacy=0x7fa74c0c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:42.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c158290 0x7fa730003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:42.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.702+0000 7fa74aa00640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa730004130 0x7fa730004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:42.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa74aa00640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730004d50 0x7fa7300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:42.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c15bad0 con 0x7fa730004130 2024-05-27T00:21:42.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa73000bb10 con 0x7fa730004d50 2024-05-27T00:21:42.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa73000bd10 con 0x7fa74c158290 2024-05-27T00:21:42.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa743e00640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa730004130 0x7fa730004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:42.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa74b400640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c158290 0x7fa730003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:42.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa74be00640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730004d50 0x7fa7300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:42.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa743e00640 1 -- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c158290 msgr2=0x7fa730003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:42.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa743e00640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c158290 0x7fa730003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:42.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa743e00640 1 -- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730004d50 msgr2=0x7fa7300051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:42.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa743e00640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730004d50 0x7fa7300051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:42.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa743e00640 1 -- 172.21.15.98:0/2793558764 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa730003530 con 0x7fa730004130 2024-05-27T00:21:42.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa743e00640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa730004130 0x7fa730004810 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa734007f80 tx=0x7fa73401ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10ee2362c12be99 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:42.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa734007bd0 con 0x7fa730004130 2024-05-27T00:21:42.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7340a9020 con 0x7fa730004130 2024-05-27T00:21:42.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.706+0000 7fa74aa00640 1 -- 172.21.15.98:0/2793558764 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa734005b00 con 0x7fa730004130 2024-05-27T00:21:42.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 -- 172.21.15.98:0/2793558764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa730004130 msgr2=0x7fa730004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:42.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 --2- 172.21.15.98:0/2793558764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa730004130 0x7fa730004810 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa734007f80 tx=0x7fa73401ca20 comp rx=0 tx=0).stop 2024-05-27T00:21:42.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa74be00640 1 -- 172.21.15.98:0/2793558764 reap_dead start 2024-05-27T00:21:42.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 -- 172.21.15.98:0/2793558764 shutdown_connections 2024-05-27T00:21:42.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 -- 172.21.15.98:0/2793558764 >> 172.21.15.98:0/2793558764 conn(0x7fa74c06dd00 msgr2=0x7fa74c06b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:42.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 -- 172.21.15.98:0/2793558764 shutdown_connections 2024-05-27T00:21:42.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 -- 172.21.15.98:0/2793558764 wait complete. 2024-05-27T00:21:42.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 Processor -- start 2024-05-27T00:21:42.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 -- start start 2024-05-27T00:21:42.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730004130 0x7fa74c159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:42.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa74c0c6260 0x7fa74c1f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:42.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c0c66a0 0x7fa74c1f0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:42.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c151d30 con 0x7fa74c0c6260 2024-05-27T00:21:42.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c0c8e60 con 0x7fa730004130 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa751000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa74c0c9e00 con 0x7fa74c0c66a0 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa743e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa74c0c6260 0x7fa74c1f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa743e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa74c0c6260 0x7fa74c1f0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41614/0 (socket says 172.21.15.98:41614) 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa743e00640 1 -- 172.21.15.98:0/3074698681 learned_addr learned my addr 172.21.15.98:0/3074698681 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa74be00640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c0c66a0 0x7fa74c1f0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa74b400640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730004130 0x7fa74c159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa743e00640 1 -- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c0c66a0 msgr2=0x7fa74c1f0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa743e00640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c0c66a0 0x7fa74c1f0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa743e00640 1 -- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730004130 msgr2=0x7fa74c159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa743e00640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730004130 0x7fa74c159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa743e00640 1 -- 172.21.15.98:0/3074698681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74c159900 con 0x7fa74c0c6260 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.710+0000 7fa74be00640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c0c66a0 0x7fa74c1f0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:42.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa743e00640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa74c0c6260 0x7fa74c1f0210 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa734007040 tx=0x7fa730005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa743400640 1 -- 172.21.15.98:0/3074698681 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa734001b30 con 0x7fa74c0c6260 2024-05-27T00:21:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa743400640 1 -- 172.21.15.98:0/3074698681 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7340072f0 con 0x7fa74c0c6260 2024-05-27T00:21:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa743400640 1 -- 172.21.15.98:0/3074698681 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7340076e0 con 0x7fa74c0c6260 2024-05-27T00:21:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa751000640 1 -- 172.21.15.98:0/3074698681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa74c1f42c0 con 0x7fa74c0c6260 2024-05-27T00:21:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa751000640 1 -- 172.21.15.98:0/3074698681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa74c1f45a0 con 0x7fa74c0c6260 2024-05-27T00:21:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa751000640 1 -- 172.21.15.98:0/3074698681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa74c075c40 con 0x7fa74c0c6260 2024-05-27T00:21:42.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa743400640 1 -- 172.21.15.98:0/3074698681 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa7340a5710 con 0x7fa74c0c6260 2024-05-27T00:21:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa743400640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7280678f0 0x7fa728069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa743400640 1 -- 172.21.15.98:0/3074698681 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7fa73411f5f0 con 0x7fa74c0c6260 2024-05-27T00:21:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.714+0000 7fa74b400640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7280678f0 0x7fa728069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:42.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.718+0000 7fa74b400640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7280678f0 0x7fa728069db0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa73c005630 tx=0x7fa73c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:42.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.718+0000 7fa743400640 1 -- 172.21.15.98:0/3074698681 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa7340bf030 con 0x7fa74c0c6260 2024-05-27T00:21:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.918+0000 7fa751000640 1 -- 172.21.15.98:0/3074698681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa74c0c9b50 con 0x7fa74c0c6260 2024-05-27T00:21:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.918+0000 7fa743400640 1 -- 172.21.15.98:0/3074698681 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6007 (secure 0 0 0) 0x7fa734016730 con 0x7fa74c0c6260 2024-05-27T00:21:42.921 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:42.921 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:42.490758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 26 2024-05-27T00:21:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 -- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7280678f0 msgr2=0x7fa728069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7280678f0 0x7fa728069db0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa73c005630 tx=0x7fa73c009040 comp rx=0 tx=0).stop 2024-05-27T00:21:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 -- 172.21.15.98:0/3074698681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa74c0c6260 msgr2=0x7fa74c1f0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa74c0c6260 0x7fa74c1f0210 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa734007040 tx=0x7fa730005840 comp rx=0 tx=0).stop 2024-05-27T00:21:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 -- 172.21.15.98:0/3074698681 shutdown_connections 2024-05-27T00:21:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c0c66a0 0x7fa74c1f0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa74c0c6260 0x7fa74c1f0210 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7280678f0 0x7fa728069db0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 --2- 172.21.15.98:0/3074698681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730004130 0x7fa74c159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 -- 172.21.15.98:0/3074698681 >> 172.21.15.98:0/3074698681 conn(0x7fa74c06dd00 msgr2=0x7fa74c1501e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 -- 172.21.15.98:0/3074698681 shutdown_connections 2024-05-27T00:21:42.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:42.922+0000 7fa751000640 1 -- 172.21.15.98:0/3074698681 wait complete. 2024-05-27T00:21:42.942 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:21:44.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc37200640 1 Processor -- start 2024-05-27T00:21:45.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc37200640 1 -- start start 2024-05-27T00:21:45.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc37200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc300c76b0 0x7fbc300629d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc37200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc300c62c0 con 0x7fbc300c90f0 2024-05-27T00:21:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc37200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc300c7b80 con 0x7fbc301525a0 2024-05-27T00:21:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc37200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbc30159190 con 0x7fbc300c76b0 2024-05-27T00:21:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc36800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbc300c90f0 0x7fbc3006f000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60992/0 (socket says 172.21.15.98:60992) 2024-05-27T00:21:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc36800640 1 -- 172.21.15.98:0/2066264574 learned_addr learned my addr 172.21.15.98:0/2066264574 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:45.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc35400640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc300c76b0 0x7fbc300629d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308524085 0 0) 0x7fbc300c62c0 con 0x7fbc300c90f0 2024-05-27T00:21:45.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc18003530 con 0x7fbc300c90f0 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4248025564 0 0) 0x7fbc300c7b80 con 0x7fbc301525a0 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc300c62c0 con 0x7fbc301525a0 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3878346190 0 0) 0x7fbc18003530 con 0x7fbc300c90f0 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc300c76b0 msgr2=0x7fbc300629d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc300c76b0 0x7fbc300629d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 >> v1:172.21.15.204:6789/0 conn(0x7fbc301525a0 legacy=0x7fbc300c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc30159430 con 0x7fbc300c90f0 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fbc2c001a00 con 0x7fbc300c90f0 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 >> v1:172.21.15.98:6789/0 conn(0x7fbc300c90f0 legacy=0x7fbc3006f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc300c76b0 0x7fbc18003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:45.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 0x7fbc18004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc18004d50 0x7fbc180051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc30159190 con 0x7fbc18004130 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1800bb10 con 0x7fbc18004d50 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1800bd10 con 0x7fbc300c76b0 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc35400640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 0x7fbc18004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc35e00640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc300c76b0 0x7fbc18003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc36800640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc18004d50 0x7fbc180051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc35400640 1 -- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc300c76b0 msgr2=0x7fbc18003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc35400640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc300c76b0 0x7fbc18003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc35400640 1 -- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc18004d50 msgr2=0x7fbc180051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc35400640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc18004d50 0x7fbc180051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:45.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.090+0000 7fbc35400640 1 -- 172.21.15.98:0/2066264574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc18003530 con 0x7fbc18004130 2024-05-27T00:21:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc35400640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 0x7fbc18004810 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009ab0 tx=0x7fbc2001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131222614ad629e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:45.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc200b2070 con 0x7fbc18004130 2024-05-27T00:21:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc200a5580 con 0x7fbc18004130 2024-05-27T00:21:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc34a00640 1 -- 172.21.15.98:0/2066264574 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc200096b0 con 0x7fbc18004130 2024-05-27T00:21:45.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc37200640 1 -- 172.21.15.98:0/2066264574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 msgr2=0x7fbc18004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc37200640 1 --2- 172.21.15.98:0/2066264574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 0x7fbc18004810 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009ab0 tx=0x7fbc2001c990 comp rx=0 tx=0).stop 2024-05-27T00:21:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc36800640 1 -- 172.21.15.98:0/2066264574 reap_dead start 2024-05-27T00:21:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc37200640 1 -- 172.21.15.98:0/2066264574 shutdown_connections 2024-05-27T00:21:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc37200640 1 -- 172.21.15.98:0/2066264574 >> 172.21.15.98:0/2066264574 conn(0x7fbc30067730 msgr2=0x7fbc3006b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc37200640 1 -- 172.21.15.98:0/2066264574 shutdown_connections 2024-05-27T00:21:45.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.094+0000 7fbc37200640 1 -- 172.21.15.98:0/2066264574 wait complete. 2024-05-27T00:21:45.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc37200640 1 Processor -- start 2024-05-27T00:21:45.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc37200640 1 -- start start 2024-05-27T00:21:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc37200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 0x7fbc301e33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc37200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc301e3ca0 0x7fbc30155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc37200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc30072ce0 0x7fbc30073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:45.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc37200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc30152980 con 0x7fbc18004130 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc37200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc300c6810 con 0x7fbc301e3ca0 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc37200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbc30152b00 con 0x7fbc30072ce0 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc35e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 0x7fbc301e33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc35e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 0x7fbc301e33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51296/0 (socket says 172.21.15.98:51296) 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc35e00640 1 -- 172.21.15.98:0/4006763262 learned_addr learned my addr 172.21.15.98:0/4006763262 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc36800640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc30072ce0 0x7fbc30073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc35400640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc301e3ca0 0x7fbc30155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc35e00640 1 -- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc30072ce0 msgr2=0x7fbc30073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc35e00640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc30072ce0 0x7fbc30073160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc35e00640 1 -- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc301e3ca0 msgr2=0x7fbc30155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc35e00640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc301e3ca0 0x7fbc30155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc35e00640 1 -- 172.21.15.98:0/4006763262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3014eb10 con 0x7fbc18004130 2024-05-27T00:21:45.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc36800640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc30072ce0 0x7fbc30073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc35e00640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 0x7fbc301e33b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbc24007860 tx=0x7fbc2400f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc2a000640 1 -- 172.21.15.98:0/4006763262 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24010040 con 0x7fbc18004130 2024-05-27T00:21:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc2a000640 1 -- 172.21.15.98:0/4006763262 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc2401a070 con 0x7fbc18004130 2024-05-27T00:21:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc2a000640 1 -- 172.21.15.98:0/4006763262 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24015870 con 0x7fbc18004130 2024-05-27T00:21:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc37200640 1 -- 172.21.15.98:0/4006763262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc3014edf0 con 0x7fbc18004130 2024-05-27T00:21:45.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.098+0000 7fbc37200640 1 -- 172.21.15.98:0/4006763262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc3014f3b0 con 0x7fbc18004130 2024-05-27T00:21:45.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.102+0000 7fbc37200640 1 -- 172.21.15.98:0/4006763262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc30075c40 con 0x7fbc18004130 2024-05-27T00:21:45.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.102+0000 7fbc2a000640 1 -- 172.21.15.98:0/4006763262 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbc24015a10 con 0x7fbc18004130 2024-05-27T00:21:45.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.102+0000 7fbc2a000640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc140678c0 0x7fbc14069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:45.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.102+0000 7fbc2a000640 1 -- 172.21.15.98:0/4006763262 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7fbc24089740 con 0x7fbc18004130 2024-05-27T00:21:45.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.102+0000 7fbc35400640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc140678c0 0x7fbc14069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:45.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.102+0000 7fbc35400640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc140678c0 0x7fbc14069d80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbc200af000 tx=0x7fbc200046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:45.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.106+0000 7fbc2a000640 1 -- 172.21.15.98:0/4006763262 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fbc2405a010 con 0x7fbc18004130 2024-05-27T00:21:45.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.294+0000 7fbc37200640 1 -- 172.21.15.98:0/4006763262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc3014e7b0 con 0x7fbc18004130 2024-05-27T00:21:45.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.294+0000 7fbc2a000640 1 -- 172.21.15.98:0/4006763262 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6007 (secure 0 0 0) 0x7fbc24087e20 con 0x7fbc18004130 2024-05-27T00:21:45.297 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:45.297 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:42.490758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:45.297 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 26 2024-05-27T00:21:45.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 -- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc140678c0 msgr2=0x7fbc14069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:45.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc140678c0 0x7fbc14069d80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbc200af000 tx=0x7fbc200046d0 comp rx=0 tx=0).stop 2024-05-27T00:21:45.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 -- 172.21.15.98:0/4006763262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 msgr2=0x7fbc301e33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:45.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 0x7fbc301e33b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbc24007860 tx=0x7fbc2400f540 comp rx=0 tx=0).stop 2024-05-27T00:21:45.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 -- 172.21.15.98:0/4006763262 shutdown_connections 2024-05-27T00:21:45.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc30072ce0 0x7fbc30073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:45.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc140678c0 0x7fbc14069d80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:45.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc301e3ca0 0x7fbc30155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:45.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 --2- 172.21.15.98:0/4006763262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18004130 0x7fbc301e33b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:45.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 -- 172.21.15.98:0/4006763262 >> 172.21.15.98:0/4006763262 conn(0x7fbc30067730 msgr2=0x7fbc30150490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:45.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 -- 172.21.15.98:0/4006763262 shutdown_connections 2024-05-27T00:21:45.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:45.298+0000 7fbc37200640 1 -- 172.21.15.98:0/4006763262 wait complete. 2024-05-27T00:21:45.319 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:21:47.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:47.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:21:47.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:21:47.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.466+0000 7f4794c00640 1 Processor -- start 2024-05-27T00:21:47.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.466+0000 7f4794c00640 1 -- start start 2024-05-27T00:21:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.466+0000 7f4794c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f479006b470 0x7f479015b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.466+0000 7f4794c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f479006b8b0 con 0x7f479015bb90 2024-05-27T00:21:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.466+0000 7f4794c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4790154de0 con 0x7f4790154a00 2024-05-27T00:21:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.466+0000 7f4794c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f479015b650 con 0x7f479006b470 2024-05-27T00:21:47.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478fe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f479015bb90 0x7f479015c020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:32774/0 (socket says 172.21.15.98:32774) 2024-05-27T00:21:47.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478fe00640 1 -- 172.21.15.98:0/1318171811 learned_addr learned my addr 172.21.15.98:0/1318171811 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:47.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f4786a00640 1 --2- 172.21.15.98:0/1318171811 >> v2:172.21.15.204:3300/0 conn(0x7f479006b470 0x7f479015b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:47.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 747988693 0 0) 0x7f479006b8b0 con 0x7f479015bb90 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4774003530 con 0x7f479015bb90 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3507356443 0 0) 0x7f4790154de0 con 0x7f4790154a00 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f479006b8b0 con 0x7f4790154a00 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2275614707 0 0) 0x7f4774003530 con 0x7f479015bb90 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 >> v2:172.21.15.204:3300/0 conn(0x7f479006b470 msgr2=0x7f479015b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 --2- 172.21.15.98:0/1318171811 >> v2:172.21.15.204:3300/0 conn(0x7f479006b470 0x7f479015b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 >> v1:172.21.15.204:6789/0 conn(0x7f4790154a00 legacy=0x7f47900c1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479006bb20 con 0x7f479015bb90 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f4788001680 con 0x7f479015bb90 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 >> v1:172.21.15.98:6789/0 conn(0x7f479015bb90 legacy=0x7f479015c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4774003b50 0x7f4774003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4774004810 0x7f4774004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47740051d0 0x7f477400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f479015b650 con 0x7f47740051d0 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f477400c050 con 0x7f4774003b50 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f477400c250 con 0x7f4774004810 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478f400640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4774003b50 0x7f4774003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f4786a00640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4774004810 0x7f4774004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478fe00640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47740051d0 0x7f477400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478fe00640 1 -- 172.21.15.98:0/1318171811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4774004810 msgr2=0x7f4774004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478fe00640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4774004810 0x7f4774004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478fe00640 1 -- 172.21.15.98:0/1318171811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4774003b50 msgr2=0x7f4774003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478fe00640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4774003b50 0x7f4774003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:47.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.470+0000 7f478fe00640 1 -- 172.21.15.98:0/1318171811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4774003530 con 0x7f47740051d0 2024-05-27T00:21:47.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f478fe00640 1 -- 172.21.15.98:0/1318171811 reap_dead start 2024-05-27T00:21:47.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f478fe00640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47740051d0 0x7f477400b9d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f478800c160 tx=0x7f478801f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563a63934dd6a22f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:47.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788007cb0 con 0x7f47740051d0 2024-05-27T00:21:47.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4788005b00 con 0x7f47740051d0 2024-05-27T00:21:47.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f478ea00640 1 -- 172.21.15.98:0/1318171811 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788007070 con 0x7f47740051d0 2024-05-27T00:21:47.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f4794c00640 1 -- 172.21.15.98:0/1318171811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47740051d0 msgr2=0x7f477400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:47.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f4794c00640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47740051d0 0x7f477400b9d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f478800c160 tx=0x7f478801f350 comp rx=0 tx=0).stop 2024-05-27T00:21:47.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f4794c00640 1 -- 172.21.15.98:0/1318171811 shutdown_connections 2024-05-27T00:21:47.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f4794c00640 1 --2- 172.21.15.98:0/1318171811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47740051d0 0x7f477400b9d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:47.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f4794c00640 1 -- 172.21.15.98:0/1318171811 >> 172.21.15.98:0/1318171811 conn(0x7f479006eb50 msgr2=0x7f479006ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:47.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f4794c00640 1 -- 172.21.15.98:0/1318171811 shutdown_connections 2024-05-27T00:21:47.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f4794c00640 1 -- 172.21.15.98:0/1318171811 wait complete. 2024-05-27T00:21:47.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f4794c00640 1 Processor -- start 2024-05-27T00:21:47.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.474+0000 7f4794c00640 1 -- start start 2024-05-27T00:21:47.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4794c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47740051d0 0x7f47901f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:47.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4794c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47900c7420 0x7f47901f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:47.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4794c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4790159d00 0x7f4790158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:47.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4794c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f479006c6a0 con 0x7f47900c7420 2024-05-27T00:21:47.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4794c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f479006c3c0 con 0x7f4790159d00 2024-05-27T00:21:47.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4794c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f479006c980 con 0x7f47740051d0 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4786a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47900c7420 0x7f47901f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4786a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47900c7420 0x7f47901f4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51310/0 (socket says 172.21.15.98:51310) 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4786a00640 1 -- 172.21.15.98:0/817312809 learned_addr learned my addr 172.21.15.98:0/817312809 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f478fe00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4790159d00 0x7f4790158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f478f400640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47740051d0 0x7f47901f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4786a00640 1 -- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47740051d0 msgr2=0x7f47901f0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4786a00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47740051d0 0x7f47901f0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4786a00640 1 -- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4790159d00 msgr2=0x7f4790158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4786a00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4790159d00 0x7f4790158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4786a00640 1 -- 172.21.15.98:0/817312809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47901513f0 con 0x7f47900c7420 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4786a00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47900c7420 0x7f47901f4120 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f47780119c0 tx=0x7f4778011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4787e00640 1 -- 172.21.15.98:0/817312809 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477800fb50 con 0x7f47900c7420 2024-05-27T00:21:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4787e00640 1 -- 172.21.15.98:0/817312809 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f477800fcf0 con 0x7f47900c7420 2024-05-27T00:21:47.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4787e00640 1 -- 172.21.15.98:0/817312809 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477800fea0 con 0x7f47900c7420 2024-05-27T00:21:47.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4794c00640 1 -- 172.21.15.98:0/817312809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47901516d0 con 0x7f47900c7420 2024-05-27T00:21:47.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4794c00640 1 -- 172.21.15.98:0/817312809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47901f04f0 con 0x7f47900c7420 2024-05-27T00:21:47.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4794c00640 1 -- 172.21.15.98:0/817312809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4790075c40 con 0x7f47900c7420 2024-05-27T00:21:47.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.478+0000 7f4787e00640 1 -- 172.21.15.98:0/817312809 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4778012040 con 0x7f47900c7420 2024-05-27T00:21:47.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.482+0000 7f4787e00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f476c0678c0 0x7f476c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:47.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.482+0000 7f4787e00640 1 -- 172.21.15.98:0/817312809 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f4778086530 con 0x7f47900c7420 2024-05-27T00:21:47.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.482+0000 7f478f400640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f476c0678c0 0x7f476c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:47.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.482+0000 7f478f400640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f476c0678c0 0x7f476c069d80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4780000f90 tx=0x7f4780009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:47.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.486+0000 7f4787e00640 1 -- 172.21.15.98:0/817312809 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4778019030 con 0x7f47900c7420 2024-05-27T00:21:47.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.698+0000 7f4794c00640 1 -- 172.21.15.98:0/817312809 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47900b1160 con 0x7f47900c7420 2024-05-27T00:21:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.698+0000 7f4787e00640 1 -- 172.21.15.98:0/817312809 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6007 (secure 0 0 0) 0x7f4778056e90 con 0x7f47900c7420 2024-05-27T00:21:47.700 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:47.700 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:42.490758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:47.700 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 26 2024-05-27T00:21:47.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 -- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f476c0678c0 msgr2=0x7f476c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:47.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f476c0678c0 0x7f476c069d80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4780000f90 tx=0x7f4780009040 comp rx=0 tx=0).stop 2024-05-27T00:21:47.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 -- 172.21.15.98:0/817312809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47900c7420 msgr2=0x7f47901f4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:47.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47900c7420 0x7f47901f4120 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f47780119c0 tx=0x7f4778011e80 comp rx=0 tx=0).stop 2024-05-27T00:21:47.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 -- 172.21.15.98:0/817312809 shutdown_connections 2024-05-27T00:21:47.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f476c0678c0 0x7f476c069d80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:47.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4790159d00 0x7f4790158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:47.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47900c7420 0x7f47901f4120 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:47.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 --2- 172.21.15.98:0/817312809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47740051d0 0x7f47901f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:47.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 -- 172.21.15.98:0/817312809 >> 172.21.15.98:0/817312809 conn(0x7f479006eb50 msgr2=0x7f4790071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:47.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 -- 172.21.15.98:0/817312809 shutdown_connections 2024-05-27T00:21:47.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:47.702+0000 7f4794c00640 1 -- 172.21.15.98:0/817312809 wait complete. 2024-05-27T00:21:47.722 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:21:49.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f87a1800640 1 Processor -- start 2024-05-27T00:21:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f87a1800640 1 -- start start 2024-05-27T00:21:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f87a1800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f879c0cbe10 0x7f879c1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f87a1800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f879c0c1f60 con 0x7f879c073570 2024-05-27T00:21:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f87a1800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f879c0c6dd0 con 0x7f879c0cc1f0 2024-05-27T00:21:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f87a1800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f879c159810 con 0x7f879c0cbe10 2024-05-27T00:21:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f879aa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f879c0cbe10 0x7f879c1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f879b400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f879c073570 0x7f879c1559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:32780/0 (socket says 172.21.15.98:32780) 2024-05-27T00:21:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f879b400640 1 -- 172.21.15.98:0/3925449601 learned_addr learned my addr 172.21.15.98:0/3925449601 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f879aa00640 1 -- 172.21.15.98:0/3925449601 >> v1:172.21.15.204:6789/0 conn(0x7f879c0cc1f0 legacy=0x7f879c0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f879aa00640 1 -- 172.21.15.98:0/3925449601 >> v1:172.21.15.98:6789/0 conn(0x7f879c073570 legacy=0x7f879c1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f879aa00640 1 -- 172.21.15.98:0/3925449601 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879c0c70b0 con 0x7f879c0cbe10 2024-05-27T00:21:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.870+0000 7f8799600640 1 -- 172.21.15.98:0/3925449601 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164342143 0 0) 0x7f879c0c1f60 con 0x7f879c073570 2024-05-27T00:21:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f879aa00640 1 --2- 172.21.15.98:0/3925449601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cbe10 0x7f879c1592d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f878c009d90 tx=0x7f878c01f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e5306c1a0b043f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f8799600640 1 -- 172.21.15.98:0/3925449601 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c0b4070 con 0x7f879c0cbe10 2024-05-27T00:21:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f8799600640 1 -- 172.21.15.98:0/3925449601 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f878c0af600 con 0x7f879c0cbe10 2024-05-27T00:21:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f8799600640 1 -- 172.21.15.98:0/3925449601 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c0af9d0 con 0x7f879c0cbe10 2024-05-27T00:21:49.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f87a1800640 1 -- 172.21.15.98:0/3925449601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cbe10 msgr2=0x7f879c1592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:49.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f87a1800640 1 --2- 172.21.15.98:0/3925449601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cbe10 0x7f879c1592d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f878c009d90 tx=0x7f878c01f120 comp rx=0 tx=0).stop 2024-05-27T00:21:49.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f87a1800640 1 -- 172.21.15.98:0/3925449601 shutdown_connections 2024-05-27T00:21:49.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f87a1800640 1 --2- 172.21.15.98:0/3925449601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cbe10 0x7f879c1592d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:49.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f87a1800640 1 -- 172.21.15.98:0/3925449601 >> 172.21.15.98:0/3925449601 conn(0x7f879c067530 msgr2=0x7f879c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:49.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f87a1800640 1 -- 172.21.15.98:0/3925449601 shutdown_connections 2024-05-27T00:21:49.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.874+0000 7f87a1800640 1 -- 172.21.15.98:0/3925449601 wait complete. 2024-05-27T00:21:49.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 Processor -- start 2024-05-27T00:21:49.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 -- start start 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f879c073570 0x7f879c1e3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f879c0cbe10 0x7f879c1f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cc1f0 0x7f879c14f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f879c0c1330 con 0x7f879c0cc1f0 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f879c0c1050 con 0x7f879c0cbe10 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f879c0c79d0 con 0x7f879c073570 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cc1f0 0x7f879c14f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cc1f0 0x7f879c14f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51328/0 (socket says 172.21.15.98:51328) 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879b400640 1 -- 172.21.15.98:0/565505548 learned_addr learned my addr 172.21.15.98:0/565505548 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879aa00640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f879c073570 0x7f879c1e3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879a000640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f879c0cbe10 0x7f879c1f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879b400640 1 -- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f879c073570 msgr2=0x7f879c1e3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879b400640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f879c073570 0x7f879c1e3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879b400640 1 -- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f879c0cbe10 msgr2=0x7f879c1f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879b400640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f879c0cbe10 0x7f879c1f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879b400640 1 -- 172.21.15.98:0/565505548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879c151db0 con 0x7f879c0cc1f0 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f879b400640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cc1f0 0x7f879c14f8d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f879400c480 tx=0x7f879400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f8792a00640 1 -- 172.21.15.98:0/565505548 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8794018070 con 0x7f879c0cc1f0 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f8792a00640 1 -- 172.21.15.98:0/565505548 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87940074e0 con 0x7f879c0cc1f0 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 -- 172.21.15.98:0/565505548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f879c150ff0 con 0x7f879c0cc1f0 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f8792a00640 1 -- 172.21.15.98:0/565505548 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8794004a30 con 0x7f879c0cc1f0 2024-05-27T00:21:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 -- 172.21.15.98:0/565505548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f879c1515b0 con 0x7f879c0cc1f0 2024-05-27T00:21:49.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.878+0000 7f87a1800640 1 -- 172.21.15.98:0/565505548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f879c075cb0 con 0x7f879c0cc1f0 2024-05-27T00:21:49.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.882+0000 7f8792a00640 1 -- 172.21.15.98:0/565505548 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f879401b070 con 0x7f879c0cc1f0 2024-05-27T00:21:49.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.882+0000 7f8792a00640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f877c0678c0 0x7f877c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:49.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.882+0000 7f8792a00640 1 -- 172.21.15.98:0/565505548 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f8794085190 con 0x7f879c0cc1f0 2024-05-27T00:21:49.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.882+0000 7f879aa00640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f877c0678c0 0x7f877c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:49.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.882+0000 7f879aa00640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f877c0678c0 0x7f877c069d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f878c0b1000 tx=0x7f878c0aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:49.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:49.886+0000 7f8792a00640 1 -- 172.21.15.98:0/565505548 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8794055af0 con 0x7f879c0cc1f0 2024-05-27T00:21:50.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 -- 172.21.15.98:0/565505548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f879c06c220 con 0x7f879c0cc1f0 2024-05-27T00:21:50.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f8792a00640 1 -- 172.21.15.98:0/565505548 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6013 (secure 0 0 0) 0x7f87940553f0 con 0x7f879c0cc1f0 2024-05-27T00:21:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:50.075 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:47.644571+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:clientreplay","state_seq":29,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:50.076 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 27 2024-05-27T00:21:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 -- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f877c0678c0 msgr2=0x7f877c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f877c0678c0 0x7f877c069d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f878c0b1000 tx=0x7f878c0aa000 comp rx=0 tx=0).stop 2024-05-27T00:21:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 -- 172.21.15.98:0/565505548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cc1f0 msgr2=0x7f879c14f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cc1f0 0x7f879c14f8d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f879400c480 tx=0x7f879400c940 comp rx=0 tx=0).stop 2024-05-27T00:21:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 -- 172.21.15.98:0/565505548 shutdown_connections 2024-05-27T00:21:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f879c0cc1f0 0x7f879c14f8d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f877c0678c0 0x7f877c069d80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f879c0cbe10 0x7f879c1f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 --2- 172.21.15.98:0/565505548 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f879c073570 0x7f879c1e3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 -- 172.21.15.98:0/565505548 >> 172.21.15.98:0/565505548 conn(0x7f879c067530 msgr2=0x7f879c0caa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 -- 172.21.15.98:0/565505548 shutdown_connections 2024-05-27T00:21:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:50.074+0000 7f87a1800640 1 -- 172.21.15.98:0/565505548 wait complete. 2024-05-27T00:21:50.097 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5202": { "addr": "172.21.15.98:6837/1698253956", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 1698253956, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 1698253956, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5202, "incarnation": 24, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:21:47.644571+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5202 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:21:52.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.242+0000 7f91ba800640 1 Processor -- start 2024-05-27T00:21:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.242+0000 7f91ba800640 1 -- start start 2024-05-27T00:21:52.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91ba800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91b4154830 0x7f91b415b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:52.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91ba800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f91b40cc010 0x7f91b4071e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:52.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91ba800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b40c1a90 con 0x7f91b40cbbb0 2024-05-27T00:21:52.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91ba800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f91b40c2b30 con 0x7f91b4154830 2024-05-27T00:21:52.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91ba800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f91b40c26a0 con 0x7f91b40cc010 2024-05-27T00:21:52.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91b3400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91b4154830 0x7f91b415b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:52.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91b3400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91b4154830 0x7f91b415b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51334/0 (socket says 172.21.15.98:51334) 2024-05-27T00:21:52.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91b3400640 1 -- 172.21.15.98:0/4059933604 learned_addr learned my addr 172.21.15.98:0/4059933604 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:52.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91b3e00640 1 --2- 172.21.15.98:0/4059933604 >> v2:172.21.15.204:3300/0 conn(0x7f91b40cc010 0x7f91b4071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:52.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91b3400640 1 -- 172.21.15.98:0/4059933604 >> v2:172.21.15.204:3300/0 conn(0x7f91b40cc010 msgr2=0x7f91b4071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:52.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91b3400640 1 --2- 172.21.15.98:0/4059933604 >> v2:172.21.15.204:3300/0 conn(0x7f91b40cc010 0x7f91b4071e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91b3400640 1 -- 172.21.15.98:0/4059933604 >> v1:172.21.15.204:6789/0 conn(0x7f91b40cbbb0 legacy=0x7f91b40c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:21:52.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.246+0000 7f91b3400640 1 -- 172.21.15.98:0/4059933604 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b40c1d70 con 0x7f91b4154830 2024-05-27T00:21:52.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91b3400640 1 --2- 172.21.15.98:0/4059933604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 0x7f91b415b470 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f91b40b2190 tx=0x7f91a401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f45abf99139798 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91b2000640 1 -- 172.21.15.98:0/4059933604 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a40a5670 con 0x7f91b4154830 2024-05-27T00:21:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91b2000640 1 -- 172.21.15.98:0/4059933604 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a40a5810 con 0x7f91b4154830 2024-05-27T00:21:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91b2000640 1 -- 172.21.15.98:0/4059933604 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a40a5be0 con 0x7f91b4154830 2024-05-27T00:21:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 -- 172.21.15.98:0/4059933604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 msgr2=0x7f91b415b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 --2- 172.21.15.98:0/4059933604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 0x7f91b415b470 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f91b40b2190 tx=0x7f91a401c9b0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 -- 172.21.15.98:0/4059933604 shutdown_connections 2024-05-27T00:21:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 --2- 172.21.15.98:0/4059933604 >> v2:172.21.15.204:3300/0 conn(0x7f91b40cc010 0x7f91b4071e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 --2- 172.21.15.98:0/4059933604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 0x7f91b415b470 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 -- 172.21.15.98:0/4059933604 >> 172.21.15.98:0/4059933604 conn(0x7f91b4067530 msgr2=0x7f91b406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 -- 172.21.15.98:0/4059933604 shutdown_connections 2024-05-27T00:21:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 -- 172.21.15.98:0/4059933604 wait complete. 2024-05-27T00:21:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 Processor -- start 2024-05-27T00:21:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 -- start start 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.250+0000 7f91ba800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91b40cbbb0 0x7f91b415e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ba800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91b40cc010 0x7f91b4155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ba800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 0x7f91b4156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ba800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f91b40c6fd0 con 0x7f91b4154830 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ba800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f91b40cc420 con 0x7f91b40cbbb0 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ba800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f91b40c7150 con 0x7f91b40cc010 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 0x7f91b4156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 0x7f91b4156480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51350/0 (socket says 172.21.15.98:51350) 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3e00640 1 -- 172.21.15.98:0/4023522304 learned_addr learned my addr 172.21.15.98:0/4023522304 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b2a00640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91b40cc010 0x7f91b4155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3400640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91b40cbbb0 0x7f91b415e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3e00640 1 -- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91b40cc010 msgr2=0x7f91b4155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3e00640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91b40cc010 0x7f91b4155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3e00640 1 -- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91b40cbbb0 msgr2=0x7f91b415e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3e00640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91b40cbbb0 0x7f91b415e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3e00640 1 -- 172.21.15.98:0/4023522304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b415b020 con 0x7f91b4154830 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3400640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91b40cbbb0 0x7f91b415e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b2a00640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91b40cc010 0x7f91b4155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91b3e00640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 0x7f91b4156480 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f91ac00d800 tx=0x7f91ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ab400640 1 -- 172.21.15.98:0/4023522304 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ac00b6c0 con 0x7f91b4154830 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ab400640 1 -- 172.21.15.98:0/4023522304 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91ac019070 con 0x7f91b4154830 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ab400640 1 -- 172.21.15.98:0/4023522304 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ac00baa0 con 0x7f91b4154830 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ba800640 1 -- 172.21.15.98:0/4023522304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b4155410 con 0x7f91b4154830 2024-05-27T00:21:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ba800640 1 -- 172.21.15.98:0/4023522304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b406f4e0 con 0x7f91b4154830 2024-05-27T00:21:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ba800640 1 -- 172.21.15.98:0/4023522304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91b4075cb0 con 0x7f91b4154830 2024-05-27T00:21:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ab400640 1 -- 172.21.15.98:0/4023522304 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f91ac00bc40 con 0x7f91b4154830 2024-05-27T00:21:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.254+0000 7f91ab400640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f91940678c0 0x7f9194069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.258+0000 7f91ab400640 1 -- 172.21.15.98:0/4023522304 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f91ac086380 con 0x7f91b4154830 2024-05-27T00:21:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.258+0000 7f91b3400640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f91940678c0 0x7f9194069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.258+0000 7f91b3400640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f91940678c0 0x7f9194069d80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f91a40ae000 tx=0x7f91a40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:52.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.262+0000 7f91ab400640 1 -- 172.21.15.98:0/4023522304 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f91ac056c50 con 0x7f91b4154830 2024-05-27T00:21:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 -- 172.21.15.98:0/4023522304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91b40c1300 con 0x7f91b4154830 2024-05-27T00:21:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ab400640 1 -- 172.21.15.98:0/4023522304 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6013 (secure 0 0 0) 0x7f91ac056550 con 0x7f91b4154830 2024-05-27T00:21:52.454 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:52.454 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:47.644571+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:clientreplay","state_seq":29,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:21:52.455 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 27 2024-05-27T00:21:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 -- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f91940678c0 msgr2=0x7f9194069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f91940678c0 0x7f9194069d80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f91a40ae000 tx=0x7f91a40a7070 comp rx=0 tx=0).stop 2024-05-27T00:21:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 -- 172.21.15.98:0/4023522304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 msgr2=0x7f91b4156480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 0x7f91b4156480 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f91ac00d800 tx=0x7f91ac00dcc0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 -- 172.21.15.98:0/4023522304 shutdown_connections 2024-05-27T00:21:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b4154830 0x7f91b4156480 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91b40cc010 0x7f91b4155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f91940678c0 0x7f9194069d80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 --2- 172.21.15.98:0/4023522304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91b40cbbb0 0x7f91b415e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 -- 172.21.15.98:0/4023522304 >> 172.21.15.98:0/4023522304 conn(0x7f91b4067530 msgr2=0x7f91b4150be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 -- 172.21.15.98:0/4023522304 shutdown_connections 2024-05-27T00:21:52.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:52.454+0000 7f91ba800640 1 -- 172.21.15.98:0/4023522304 wait complete. 2024-05-27T00:21:52.474 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:21:54.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:21:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff96ac00640 1 Processor -- start 2024-05-27T00:21:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff96ac00640 1 -- start start 2024-05-27T00:21:54.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff96ac00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640ca480 0x7ff96415e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:54.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff96ac00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9641583f0 con 0x7ff964158010 2024-05-27T00:21:54.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff96ac00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9640c94b0 con 0x7ff9640c32d0 2024-05-27T00:21:54.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff96ac00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff96406f660 con 0x7ff9640ca480 2024-05-27T00:21:54.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff963e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff964158010 0x7ff96406faf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47894/0 (socket says 172.21.15.98:47894) 2024-05-27T00:21:54.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff963e00640 1 -- 172.21.15.98:0/1937007449 learned_addr learned my addr 172.21.15.98:0/1937007449 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:21:54.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff968c00640 1 --2- 172.21.15.98:0/1937007449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640ca480 0x7ff96415e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:54.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff962a00640 1 -- 172.21.15.98:0/1937007449 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2621905454 0 0) 0x7ff9641583f0 con 0x7ff964158010 2024-05-27T00:21:54.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.630+0000 7ff962a00640 1 -- 172.21.15.98:0/1937007449 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff948003530 con 0x7ff964158010 2024-05-27T00:21:54.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.634+0000 7ff968c00640 1 -- 172.21.15.98:0/1937007449 >> v1:172.21.15.204:6789/0 conn(0x7ff9640c32d0 legacy=0x7ff96415def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:54.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.634+0000 7ff962a00640 1 -- 172.21.15.98:0/1937007449 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1249144435 0 0) 0x7ff9640c94b0 con 0x7ff9640c32d0 2024-05-27T00:21:54.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.634+0000 7ff968c00640 1 -- 172.21.15.98:0/1937007449 >> v1:172.21.15.98:6789/0 conn(0x7ff964158010 legacy=0x7ff96406faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:54.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.634+0000 7ff968c00640 1 -- 172.21.15.98:0/1937007449 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9640c9790 con 0x7ff9640ca480 2024-05-27T00:21:54.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.634+0000 7ff968c00640 1 --2- 172.21.15.98:0/1937007449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640ca480 0x7ff96415e600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009dc0 tx=0x7ff95c01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=514b1c4bef326bf4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:54.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.634+0000 7ff962a00640 1 -- 172.21.15.98:0/1937007449 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0b2070 con 0x7ff9640ca480 2024-05-27T00:21:54.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.634+0000 7ff962a00640 1 -- 172.21.15.98:0/1937007449 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff95c0a5850 con 0x7ff9640ca480 2024-05-27T00:21:54.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.634+0000 7ff962a00640 1 -- 172.21.15.98:0/1937007449 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0a5c00 con 0x7ff9640ca480 2024-05-27T00:21:54.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 -- 172.21.15.98:0/1937007449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640ca480 msgr2=0x7ff96415e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:54.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 --2- 172.21.15.98:0/1937007449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640ca480 0x7ff96415e600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009dc0 tx=0x7ff95c01c990 comp rx=0 tx=0).stop 2024-05-27T00:21:54.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 -- 172.21.15.98:0/1937007449 shutdown_connections 2024-05-27T00:21:54.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 --2- 172.21.15.98:0/1937007449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640ca480 0x7ff96415e600 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:54.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 -- 172.21.15.98:0/1937007449 >> 172.21.15.98:0/1937007449 conn(0x7ff9640ce000 msgr2=0x7ff9640ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:54.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 -- 172.21.15.98:0/1937007449 shutdown_connections 2024-05-27T00:21:54.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 -- 172.21.15.98:0/1937007449 wait complete. 2024-05-27T00:21:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 Processor -- start 2024-05-27T00:21:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 -- start start 2024-05-27T00:21:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640c32d0 0x7ff964153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9640ca480 0x7ff9641f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964158010 0x7ff96414bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9640c5e60 con 0x7ff964158010 2024-05-27T00:21:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff9640c5ce0 con 0x7ff9640ca480 2024-05-27T00:21:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff96ac00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff964168a50 con 0x7ff9640c32d0 2024-05-27T00:21:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff968c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964158010 0x7ff96414bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff968c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964158010 0x7ff96414bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50334/0 (socket says 172.21.15.98:50334) 2024-05-27T00:21:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff963e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640c32d0 0x7ff964153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff968c00640 1 -- 172.21.15.98:0/1659908672 learned_addr learned my addr 172.21.15.98:0/1659908672 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:21:54.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.638+0000 7ff963400640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9640ca480 0x7ff9641f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:54.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff968c00640 1 -- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640c32d0 msgr2=0x7ff964153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:54.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff968c00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640c32d0 0x7ff964153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff968c00640 1 -- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9640ca480 msgr2=0x7ff9641f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff968c00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9640ca480 0x7ff9641f4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff968c00640 1 -- 172.21.15.98:0/1659908672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96414c1e0 con 0x7ff964158010 2024-05-27T00:21:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff968c00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964158010 0x7ff96414bca0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff95c01a5c0 tx=0x7ff95c019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff95be00640 1 -- 172.21.15.98:0/1659908672 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0b2040 con 0x7ff964158010 2024-05-27T00:21:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff95be00640 1 -- 172.21.15.98:0/1659908672 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff95c0bf070 con 0x7ff964158010 2024-05-27T00:21:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff95be00640 1 -- 172.21.15.98:0/1659908672 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0b8670 con 0x7ff964158010 2024-05-27T00:21:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff96ac00640 1 -- 172.21.15.98:0/1659908672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff964153010 con 0x7ff964158010 2024-05-27T00:21:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff96ac00640 1 -- 172.21.15.98:0/1659908672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9641532b0 con 0x7ff964158010 2024-05-27T00:21:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff96ac00640 1 -- 172.21.15.98:0/1659908672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff964075cb0 con 0x7ff964158010 2024-05-27T00:21:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff95be00640 1 -- 172.21.15.98:0/1659908672 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff95c0b8810 con 0x7ff964158010 2024-05-27T00:21:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.642+0000 7ff95be00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9440678c0 0x7ff944069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:21:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.646+0000 7ff95be00640 1 -- 172.21.15.98:0/1659908672 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7ff95c120f10 con 0x7ff964158010 2024-05-27T00:21:54.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.646+0000 7ff963e00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9440678c0 0x7ff944069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:21:54.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.646+0000 7ff963e00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9440678c0 0x7ff944069d80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff954007fd0 tx=0x7ff954009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:21:54.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.650+0000 7ff95be00640 1 -- 172.21.15.98:0/1659908672 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7ff95c016580 con 0x7ff964158010 2024-05-27T00:21:54.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 -- 172.21.15.98:0/1659908672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff96415b890 con 0x7ff964158010 2024-05-27T00:21:54.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff95be00640 1 -- 172.21.15.98:0/1659908672 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6014 (secure 0 0 0) 0x7ff95c0f1220 con 0x7ff964158010 2024-05-27T00:21:54.875 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:21:54.876 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:54.706568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5202,"qdb_cluster":[5202]},"id":1}]} 2024-05-27T00:21:54.876 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-05-27T00:21:54.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 -- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9440678c0 msgr2=0x7ff944069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:54.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9440678c0 0x7ff944069d80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff954007fd0 tx=0x7ff954009040 comp rx=0 tx=0).stop 2024-05-27T00:21:54.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 -- 172.21.15.98:0/1659908672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964158010 msgr2=0x7ff96414bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:21:54.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964158010 0x7ff96414bca0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff95c01a5c0 tx=0x7ff95c019730 comp rx=0 tx=0).stop 2024-05-27T00:21:54.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 -- 172.21.15.98:0/1659908672 shutdown_connections 2024-05-27T00:21:54.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff964158010 0x7ff96414bca0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:54.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9440678c0 0x7ff944069d80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:54.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9640ca480 0x7ff9641f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:54.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 --2- 172.21.15.98:0/1659908672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff9640c32d0 0x7ff964153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:21:54.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.874+0000 7ff96ac00640 1 -- 172.21.15.98:0/1659908672 >> 172.21.15.98:0/1659908672 conn(0x7ff9640ce000 msgr2=0x7ff9640c7250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:21:54.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.878+0000 7ff96ac00640 1 -- 172.21.15.98:0/1659908672 shutdown_connections 2024-05-27T00:21:54.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:21:54.878+0000 7ff96ac00640 1 -- 172.21.15.98:0/1659908672 wait complete. 2024-05-27T00:21:54.897 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=5073 2024-05-27T00:21:54.897 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2024-05-27T00:22:01.268 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 63% (141/222) Updating files: 64% (143/222) Updating files: 65% (145/222) Updating files: 66% (147/222) Updating files: 67% (149/222) Updating files: 68% (151/222) Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 90% (200/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2024-05-27T00:22:01.518 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 2b5f76f48e9 readme 2024-05-27T00:22:01.522 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.2 2024-05-27T00:22:01.533 INFO:tasks.ceph.mds.c.smithi098.stderr:2024-05-27T00:22:01.530+0000 7fd14e200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:22:01.535 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 3 -eq 48 ] 2024-05-27T00:22:01.535 INFO:tasks.workunit.client.0.smithi098.stdout:v0.3 2024-05-27T00:22:01.536 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.3 2024-05-27T00:22:01.536 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.3 2024-05-27T00:22:01.536 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.3 2024-05-27T00:22:02.755 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2024-05-27T00:22:02.760 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.3 2024-05-27T00:22:02.897 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-05-27T00:22:02.898 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-27T00:22:02.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-27T00:22:02.902 INFO:tasks.ceph.mds.a:Started 2024-05-27T00:22:02.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:02.994 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2024-05-27T00:22:03.029 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 4 -eq 48 ] 2024-05-27T00:22:03.029 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.4 2024-05-27T00:22:03.029 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.4 2024-05-27T00:22:03.029 INFO:tasks.workunit.client.0.smithi098.stdout:v0.4 2024-05-27T00:22:03.030 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.4 2024-05-27T00:22:03.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.078+0000 7fdb67a00640 1 Processor -- start 2024-05-27T00:22:03.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.078+0000 7fdb67a00640 1 -- start start 2024-05-27T00:22:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.078+0000 7fdb67a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdb60073570 0x7fdb601592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.078+0000 7fdb67a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb600c1f60 con 0x7fdb600cbe10 2024-05-27T00:22:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.078+0000 7fdb67a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb600c6dd0 con 0x7fdb600cc1f0 2024-05-27T00:22:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.078+0000 7fdb67a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdb60159810 con 0x7fdb60073570 2024-05-27T00:22:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.082+0000 7fdb65000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdb600cbe10 0x7fdb600c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47912/0 (socket says 172.21.15.98:47912) 2024-05-27T00:22:03.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.082+0000 7fdb65000640 1 -- 172.21.15.98:0/1269412307 learned_addr learned my addr 172.21.15.98:0/1269412307 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:03.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.082+0000 7fdb65a00640 1 --2- 172.21.15.98:0/1269412307 >> v2:172.21.15.98:3300/0 conn(0x7fdb60073570 0x7fdb601592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:03.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.082+0000 7fdb5f400640 1 -- 172.21.15.98:0/1269412307 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3031551155 0 0) 0x7fdb600c1f60 con 0x7fdb600cbe10 2024-05-27T00:22:03.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.082+0000 7fdb65a00640 1 -- 172.21.15.98:0/1269412307 >> v1:172.21.15.204:6789/0 conn(0x7fdb600cc1f0 legacy=0x7fdb601559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:03.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.082+0000 7fdb65a00640 1 -- 172.21.15.98:0/1269412307 >> v1:172.21.15.98:6789/0 conn(0x7fdb600cbe10 legacy=0x7fdb600c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:03.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.082+0000 7fdb65a00640 1 -- 172.21.15.98:0/1269412307 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb600c70b0 con 0x7fdb60073570 2024-05-27T00:22:03.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.082+0000 7fdb65a00640 1 --2- 172.21.15.98:0/1269412307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 0x7fdb601592d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdb58009d90 tx=0x7fdb5801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f213a7e008aca99 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:03.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.082+0000 7fdb5f400640 1 -- 172.21.15.98:0/1269412307 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb580b4070 con 0x7fdb60073570 2024-05-27T00:22:03.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.082+0000 7fdb5f400640 1 -- 172.21.15.98:0/1269412307 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb580af600 con 0x7fdb60073570 2024-05-27T00:22:03.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb5f400640 1 -- 172.21.15.98:0/1269412307 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb580af9d0 con 0x7fdb60073570 2024-05-27T00:22:03.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 -- 172.21.15.98:0/1269412307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 msgr2=0x7fdb601592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:03.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 --2- 172.21.15.98:0/1269412307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 0x7fdb601592d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdb58009d90 tx=0x7fdb5801f150 comp rx=0 tx=0).stop 2024-05-27T00:22:03.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 -- 172.21.15.98:0/1269412307 shutdown_connections 2024-05-27T00:22:03.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 --2- 172.21.15.98:0/1269412307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 0x7fdb601592d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:03.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 -- 172.21.15.98:0/1269412307 >> 172.21.15.98:0/1269412307 conn(0x7fdb60067530 msgr2=0x7fdb6006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:03.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 -- 172.21.15.98:0/1269412307 shutdown_connections 2024-05-27T00:22:03.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 -- 172.21.15.98:0/1269412307 wait complete. 2024-05-27T00:22:03.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 Processor -- start 2024-05-27T00:22:03.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 -- start start 2024-05-27T00:22:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 0x7fdb600718f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.086+0000 7fdb67a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb600cbe10 0x7fdb600c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb67a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb600cc1f0 0x7fdb60072760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb67a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb600c1330 con 0x7fdb60073570 2024-05-27T00:22:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb67a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdb600c1050 con 0x7fdb600cc1f0 2024-05-27T00:22:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb67a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdb600c79d0 con 0x7fdb600cbe10 2024-05-27T00:22:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb65000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 0x7fdb600718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb65000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 0x7fdb600718f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50362/0 (socket says 172.21.15.98:50362) 2024-05-27T00:22:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb65000640 1 -- 172.21.15.98:0/3640937193 learned_addr learned my addr 172.21.15.98:0/3640937193 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb5fe00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb600cbe10 0x7fdb600c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb65a00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb600cc1f0 0x7fdb60072760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb65000640 1 -- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb600cbe10 msgr2=0x7fdb600c4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb65000640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb600cbe10 0x7fdb600c4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb65000640 1 -- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb600cc1f0 msgr2=0x7fdb60072760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb65000640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb600cc1f0 0x7fdb60072760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb65000640 1 -- 172.21.15.98:0/3640937193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb600c3820 con 0x7fdb60073570 2024-05-27T00:22:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb65000640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 0x7fdb600718f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400c4d0 tx=0x7fdb5400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb5cc00640 1 -- 172.21.15.98:0/3640937193 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54018070 con 0x7fdb60073570 2024-05-27T00:22:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb5cc00640 1 -- 172.21.15.98:0/3640937193 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb540074e0 con 0x7fdb60073570 2024-05-27T00:22:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb67a00640 1 -- 172.21.15.98:0/3640937193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb600c3b00 con 0x7fdb60073570 2024-05-27T00:22:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb5cc00640 1 -- 172.21.15.98:0/3640937193 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54004a30 con 0x7fdb60073570 2024-05-27T00:22:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb67a00640 1 -- 172.21.15.98:0/3640937193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb60070e10 con 0x7fdb60073570 2024-05-27T00:22:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb67a00640 1 -- 172.21.15.98:0/3640937193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb60075cb0 con 0x7fdb60073570 2024-05-27T00:22:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.090+0000 7fdb5cc00640 1 -- 172.21.15.98:0/3640937193 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdb5401b070 con 0x7fdb60073570 2024-05-27T00:22:03.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.094+0000 7fdb5cc00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb400678f0 0x7fdb40069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:03.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.094+0000 7fdb5cc00640 1 -- 172.21.15.98:0/3640937193 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7fdb540852c0 con 0x7fdb60073570 2024-05-27T00:22:03.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.094+0000 7fdb5fe00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb400678f0 0x7fdb40069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:03.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.094+0000 7fdb5fe00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb400678f0 0x7fdb40069db0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdb600c7980 tx=0x7fdb4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:03.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.098+0000 7fdb5cc00640 1 -- 172.21.15.98:0/3640937193 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fdb54055b00 con 0x7fdb60073570 2024-05-27T00:22:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.370+0000 7fdb67a00640 1 -- 172.21.15.98:0/3640937193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb601e6730 con 0x7fdb60073570 2024-05-27T00:22:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.370+0000 7fdb5cc00640 1 -- 172.21.15.98:0/3640937193 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6839 (secure 0 0 0) 0x7fdb54055400 con 0x7fdb60073570 2024-05-27T00:22:03.373 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:03.374 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:54.706568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5202,"qdb_cluster":[5202]},"id":1}]} 2024-05-27T00:22:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-27T00:22:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 -- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb400678f0 msgr2=0x7fdb40069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb400678f0 0x7fdb40069db0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdb600c7980 tx=0x7fdb4c009040 comp rx=0 tx=0).stop 2024-05-27T00:22:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 -- 172.21.15.98:0/3640937193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 msgr2=0x7fdb600718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 0x7fdb600718f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400c4d0 tx=0x7fdb5400c990 comp rx=0 tx=0).stop 2024-05-27T00:22:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 -- 172.21.15.98:0/3640937193 shutdown_connections 2024-05-27T00:22:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb400678f0 0x7fdb40069db0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb600cc1f0 0x7fdb60072760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb600cbe10 0x7fdb600c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 --2- 172.21.15.98:0/3640937193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb60073570 0x7fdb600718f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 -- 172.21.15.98:0/3640937193 >> 172.21.15.98:0/3640937193 conn(0x7fdb60067530 msgr2=0x7fdb601588c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 -- 172.21.15.98:0/3640937193 shutdown_connections 2024-05-27T00:22:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:03.374+0000 7fdb67a00640 1 -- 172.21.15.98:0/3640937193 wait complete. 2024-05-27T00:22:03.397 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-05-27T00:22:03.397 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before thrashing 2024-05-27T00:22:03.817 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2024-05-27T00:22:03.823 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.4 2024-05-27T00:22:04.051 INFO:tasks.workunit.client.0.smithi098.stdout:v0.5 2024-05-27T00:22:04.052 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 5 -eq 48 ] 2024-05-27T00:22:04.052 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.5 2024-05-27T00:22:04.052 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.5 2024-05-27T00:22:04.052 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.5 2024-05-27T00:22:04.910 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2024-05-27T00:22:04.915 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.5 2024-05-27T00:22:05.012 INFO:tasks.workunit.client.0.smithi098.stdout:v0.6 2024-05-27T00:22:05.013 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 6 -eq 48 ] 2024-05-27T00:22:05.013 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.6 2024-05-27T00:22:05.013 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.6 2024-05-27T00:22:05.013 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.6 2024-05-27T00:22:05.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fadad400640 1 Processor -- start 2024-05-27T00:22:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fadad400640 1 -- start start 2024-05-27T00:22:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fadad400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fadad400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fada815bf40 0x7fada815c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fadad400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fada80ca440 con 0x7fada8152850 2024-05-27T00:22:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fadad400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fada80c58f0 con 0x7fada815bf40 2024-05-27T00:22:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fadad400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fada815bab0 con 0x7fada80c9f30 2024-05-27T00:22:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fadaca00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fada815bf40 0x7fada815c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fada7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fada7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:58206/0 (socket says 172.21.15.98:58206) 2024-05-27T00:22:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fada7400640 1 -- 172.21.15.98:0/2635316892 learned_addr learned my addr 172.21.15.98:0/2635316892 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fada7e00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fada8152850 0x7fada8070660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:38466/0 (socket says 172.21.15.98:38466) 2024-05-27T00:22:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fada7400640 1 -- 172.21.15.98:0/2635316892 >> v2:172.21.15.204:3300/0 conn(0x7fada815bf40 msgr2=0x7fada815c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fada7400640 1 --2- 172.21.15.98:0/2635316892 >> v2:172.21.15.204:3300/0 conn(0x7fada815bf40 0x7fada815c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fada6a00640 1 -- 172.21.15.98:0/2635316892 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165818250 0 0) 0x7fada80ca440 con 0x7fada8152850 2024-05-27T00:22:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fada7400640 1 -- 172.21.15.98:0/2635316892 >> v1:172.21.15.204:6789/0 conn(0x7fada8152850 legacy=0x7fada8070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.562+0000 7fada7400640 1 -- 172.21.15.98:0/2635316892 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada815c8d0 con 0x7fada80c9f30 2024-05-27T00:22:05.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fada7400640 1 --2- 172.21.15.98:0/2635316892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80c53b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad94009870 tx=0x7fad9401ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d52bdcb510c6c3e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:05.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fada6a00640 1 -- 172.21.15.98:0/2635316892 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad940b5070 con 0x7fada80c9f30 2024-05-27T00:22:05.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fada6a00640 1 -- 172.21.15.98:0/2635316892 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad940a8a50 con 0x7fada80c9f30 2024-05-27T00:22:05.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fada6a00640 1 -- 172.21.15.98:0/2635316892 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad940a8e20 con 0x7fada80c9f30 2024-05-27T00:22:05.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fadad400640 1 -- 172.21.15.98:0/2635316892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 msgr2=0x7fada80c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:05.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fadad400640 1 --2- 172.21.15.98:0/2635316892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80c53b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad94009870 tx=0x7fad9401ed50 comp rx=0 tx=0).stop 2024-05-27T00:22:05.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fadad400640 1 -- 172.21.15.98:0/2635316892 shutdown_connections 2024-05-27T00:22:05.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fadad400640 1 --2- 172.21.15.98:0/2635316892 >> v2:172.21.15.204:3300/0 conn(0x7fada815bf40 0x7fada815c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:05.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fadad400640 1 --2- 172.21.15.98:0/2635316892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80c53b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:05.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fadad400640 1 -- 172.21.15.98:0/2635316892 >> 172.21.15.98:0/2635316892 conn(0x7fada80cdeb0 msgr2=0x7fada80ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:05.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fadad400640 1 -- 172.21.15.98:0/2635316892 shutdown_connections 2024-05-27T00:22:05.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.566+0000 7fadad400640 1 -- 172.21.15.98:0/2635316892 wait complete. 2024-05-27T00:22:05.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadad400640 1 Processor -- start 2024-05-27T00:22:05.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadad400640 1 -- start start 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadad400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadad400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada8152850 0x7fada81f0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadad400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fada815bf40 0x7fada815acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadad400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fada806f870 con 0x7fada8152850 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadad400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fada806f590 con 0x7fada815bf40 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadad400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fada806fb50 con 0x7fada80c9f30 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada8152850 0x7fada81f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada8152850 0x7fada81f0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36848/0 (socket says 172.21.15.98:36848) 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7400640 1 -- 172.21.15.98:0/2000582348 learned_addr learned my addr 172.21.15.98:0/2000582348 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadaca00640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fada815bf40 0x7fada815acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:05.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7e00640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:05.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7400640 1 -- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 msgr2=0x7fada80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7400640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7400640 1 -- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fada815bf40 msgr2=0x7fada815acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7400640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fada815bf40 0x7fada815acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7400640 1 -- 172.21.15.98:0/2000582348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada815a210 con 0x7fada8152850 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7e00640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadaca00640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fada815bf40 0x7fada815acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fada7400640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada8152850 0x7fada81f0410 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fad9401d400 tx=0x7fad9401baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fad9fe00640 1 -- 172.21.15.98:0/2000582348 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad940c0070 con 0x7fada8152850 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fad9fe00640 1 -- 172.21.15.98:0/2000582348 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad940b0b70 con 0x7fada8152850 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fad9fe00640 1 -- 172.21.15.98:0/2000582348 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad940b5040 con 0x7fada8152850 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.570+0000 7fadad400640 1 -- 172.21.15.98:0/2000582348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada81f0d30 con 0x7fada8152850 2024-05-27T00:22:05.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.574+0000 7fadad400640 1 -- 172.21.15.98:0/2000582348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada815a4b0 con 0x7fada8152850 2024-05-27T00:22:05.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.574+0000 7fadad400640 1 -- 172.21.15.98:0/2000582348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fada8075c40 con 0x7fada8152850 2024-05-27T00:22:05.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.574+0000 7fad9fe00640 1 -- 172.21.15.98:0/2000582348 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fad94004030 con 0x7fada8152850 2024-05-27T00:22:05.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.574+0000 7fad9fe00640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad880678c0 0x7fad88069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:05.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.574+0000 7fad9fe00640 1 -- 172.21.15.98:0/2000582348 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7fad94122bd0 con 0x7fada8152850 2024-05-27T00:22:05.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.578+0000 7fada7e00640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad880678c0 0x7fad88069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.578+0000 7fada7e00640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad880678c0 0x7fad88069d80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fad98007fd0 tx=0x7fad98007c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:05.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.582+0000 7fad9fe00640 1 -- 172.21.15.98:0/2000582348 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fad940188b0 con 0x7fada8152850 2024-05-27T00:22:05.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.794+0000 7fadad400640 1 -- 172.21.15.98:0/2000582348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fada80c7640 con 0x7fada8152850 2024-05-27T00:22:05.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.794+0000 7fad9fe00640 1 -- 172.21.15.98:0/2000582348 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6839 (secure 0 0 0) 0x7fad940f3490 con 0x7fada8152850 2024-05-27T00:22:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:05.797 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:54.706568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5202,"qdb_cluster":[5202]},"id":1}]} 2024-05-27T00:22:05.797 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 -- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad880678c0 msgr2=0x7fad88069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad880678c0 0x7fad88069d80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fad98007fd0 tx=0x7fad98007c20 comp rx=0 tx=0).stop 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 -- 172.21.15.98:0/2000582348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada8152850 msgr2=0x7fada81f0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada8152850 0x7fada81f0410 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fad9401d400 tx=0x7fad9401baf0 comp rx=0 tx=0).stop 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 -- 172.21.15.98:0/2000582348 shutdown_connections 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad880678c0 0x7fad88069d80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fada815bf40 0x7fada815acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fada8152850 0x7fada81f0410 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 --2- 172.21.15.98:0/2000582348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fada80c9f30 0x7fada80ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 -- 172.21.15.98:0/2000582348 >> 172.21.15.98:0/2000582348 conn(0x7fada80cdeb0 msgr2=0x7fada80c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 -- 172.21.15.98:0/2000582348 shutdown_connections 2024-05-27T00:22:05.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:05.798+0000 7fadad400640 1 -- 172.21.15.98:0/2000582348 wait complete. 2024-05-27T00:22:05.820 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-05-27T00:22:05.821 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-27T00:22:05.821 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:22:05.841 INFO:tasks.ceph.mds.c:Stopped 2024-05-27T00:22:05.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.018+0000 7f72ecc00640 1 Processor -- start 2024-05-27T00:22:06.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.018+0000 7f72ecc00640 1 -- start start 2024-05-27T00:22:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.018+0000 7f72ecc00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f72e80cbe10 0x7f72e81592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.018+0000 7f72ecc00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e80c1f60 con 0x7f72e80cc1f0 2024-05-27T00:22:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.018+0000 7f72ecc00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e80c6dd0 con 0x7f72e8073570 2024-05-27T00:22:06.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.018+0000 7f72ecc00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f72e8159810 con 0x7f72e80cbe10 2024-05-27T00:22:06.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72dd400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f72e80cc1f0 0x7f72e80c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57984/0 (socket says 172.21.15.98:57984) 2024-05-27T00:22:06.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72dd400640 1 -- 172.21.15.98:0/4014855781 learned_addr learned my addr 172.21.15.98:0/4014855781 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:06.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5e00640 1 --2- 172.21.15.98:0/4014855781 >> v2:172.21.15.204:3300/0 conn(0x7f72e80cbe10 0x7f72e81592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2854037775 0 0) 0x7f72e80c1f60 con 0x7f72e80cc1f0 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72cc003530 con 0x7f72e80cc1f0 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1216861916 0 0) 0x7f72cc003530 con 0x7f72e80cc1f0 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 >> v2:172.21.15.204:3300/0 conn(0x7f72e80cbe10 msgr2=0x7f72e81592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 --2- 172.21.15.98:0/4014855781 >> v2:172.21.15.204:3300/0 conn(0x7f72e80cbe10 0x7f72e81592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 >> v1:172.21.15.204:6789/0 conn(0x7f72e8073570 legacy=0x7f72e81559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e80c70b0 con 0x7f72e80cc1f0 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f72d0001680 con 0x7f72e80cc1f0 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 >> v1:172.21.15.98:6789/0 conn(0x7f72e80cc1f0 legacy=0x7f72e80c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72cc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72cc004810 0x7f72cc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f72cc0051d0 0x7f72cc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f72e8159810 con 0x7f72cc0039c0 2024-05-27T00:22:06.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc00c080 con 0x7f72cc004810 2024-05-27T00:22:06.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc00c280 con 0x7f72cc0051d0 2024-05-27T00:22:06.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5e00640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72cc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:06.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72dd400640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72cc004810 0x7f72cc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:06.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e6800640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f72cc0051d0 0x7f72cc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:06.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5e00640 1 -- 172.21.15.98:0/4014855781 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f72cc0051d0 msgr2=0x7f72cc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:06.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5e00640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f72cc0051d0 0x7f72cc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:06.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5e00640 1 -- 172.21.15.98:0/4014855781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72cc004810 msgr2=0x7f72cc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:06.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5e00640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72cc004810 0x7f72cc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:06.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e5e00640 1 -- 172.21.15.98:0/4014855781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc003530 con 0x7f72cc0039c0 2024-05-27T00:22:06.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.022+0000 7f72e6800640 1 -- 172.21.15.98:0/4014855781 reap_dead start 2024-05-27T00:22:06.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72e5e00640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72cc003de0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009ae0 tx=0x7f72d801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e343e2e76f76e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:06.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d80b2070 con 0x7f72cc0039c0 2024-05-27T00:22:06.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72d80a5550 con 0x7f72cc0039c0 2024-05-27T00:22:06.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72e5400640 1 -- 172.21.15.98:0/4014855781 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d80096b0 con 0x7f72cc0039c0 2024-05-27T00:22:06.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72ecc00640 1 -- 172.21.15.98:0/4014855781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 msgr2=0x7f72cc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:06.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72ecc00640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72cc003de0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009ae0 tx=0x7f72d801ca70 comp rx=0 tx=0).stop 2024-05-27T00:22:06.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72ecc00640 1 -- 172.21.15.98:0/4014855781 shutdown_connections 2024-05-27T00:22:06.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72ecc00640 1 --2- 172.21.15.98:0/4014855781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72cc003de0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:06.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72ecc00640 1 -- 172.21.15.98:0/4014855781 >> 172.21.15.98:0/4014855781 conn(0x7f72e8067530 msgr2=0x7f72e806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:06.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72ecc00640 1 -- 172.21.15.98:0/4014855781 shutdown_connections 2024-05-27T00:22:06.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72ecc00640 1 -- 172.21.15.98:0/4014855781 wait complete. 2024-05-27T00:22:06.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72ecc00640 1 Processor -- start 2024-05-27T00:22:06.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.026+0000 7f72ecc00640 1 -- start start 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72ecc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72e80c3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72ecc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72e81f0480 0x7f72e814da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72ecc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f72e8072090 0x7f72e81f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72ecc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f72e80c1330 con 0x7f72cc0039c0 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72ecc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f72e80c1050 con 0x7f72e81f0480 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72ecc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f72e80c79d0 con 0x7f72e8072090 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72e5e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72e80c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72e5e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72e80c3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36864/0 (socket says 172.21.15.98:36864) 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72e5e00640 1 -- 172.21.15.98:0/1411837824 learned_addr learned my addr 172.21.15.98:0/1411837824 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72e6800640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f72e8072090 0x7f72e81f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:06.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72dd400640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72e81f0480 0x7f72e814da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72e5e00640 1 -- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f72e8072090 msgr2=0x7f72e81f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72e5e00640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f72e8072090 0x7f72e81f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72e5e00640 1 -- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72e81f0480 msgr2=0x7f72e814da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72e5e00640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72e81f0480 0x7f72e814da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72e5e00640 1 -- 172.21.15.98:0/1411837824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e80c3c50 con 0x7f72cc0039c0 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72dd400640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72e81f0480 0x7f72e814da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72e5e00640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72e80c3710 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f72d801c8a0 tx=0x7f72d8019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72dea00640 1 -- 172.21.15.98:0/1411837824 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d80b2040 con 0x7f72cc0039c0 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72dea00640 1 -- 172.21.15.98:0/1411837824 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72d800f000 con 0x7f72cc0039c0 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72ecc00640 1 -- 172.21.15.98:0/1411837824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e8070ff0 con 0x7f72cc0039c0 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72dea00640 1 -- 172.21.15.98:0/1411837824 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d8004260 con 0x7f72cc0039c0 2024-05-27T00:22:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72ecc00640 1 -- 172.21.15.98:0/1411837824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e80715b0 con 0x7f72cc0039c0 2024-05-27T00:22:06.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72ecc00640 1 -- 172.21.15.98:0/1411837824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e8075cb0 con 0x7f72cc0039c0 2024-05-27T00:22:06.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.030+0000 7f72dea00640 1 -- 172.21.15.98:0/1411837824 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f72d80bf030 con 0x7f72cc0039c0 2024-05-27T00:22:06.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.034+0000 7f72dea00640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f72c40678c0 0x7f72c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:06.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.034+0000 7f72dea00640 1 -- 172.21.15.98:0/1411837824 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f72d811fcb0 con 0x7f72cc0039c0 2024-05-27T00:22:06.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.034+0000 7f72dd400640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f72c40678c0 0x7f72c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:06.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.034+0000 7f72dd400640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f72c40678c0 0x7f72c4069d80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f72d0000c10 tx=0x7f72d0001450 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:06.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.038+0000 7f72dea00640 1 -- 172.21.15.98:0/1411837824 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f72d80c6060 con 0x7f72cc0039c0 2024-05-27T00:22:06.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.226+0000 7f72ecc00640 1 -- 172.21.15.98:0/1411837824 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72e806eba0 con 0x7f72cc0039c0 2024-05-27T00:22:06.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.226+0000 7f72dea00640 1 -- 172.21.15.98:0/1411837824 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6839 (secure 0 0 0) 0x7f72d8016780 con 0x7f72cc0039c0 2024-05-27T00:22:06.227 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:06.228 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:54.706568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5202,"qdb_cluster":[5202]},"id":1}]} 2024-05-27T00:22:06.228 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-27T00:22:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.226+0000 7f72ecc00640 1 -- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f72c40678c0 msgr2=0x7f72c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:06.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.226+0000 7f72ecc00640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f72c40678c0 0x7f72c4069d80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f72d0000c10 tx=0x7f72d0001450 comp rx=0 tx=0).stop 2024-05-27T00:22:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.226+0000 7f72ecc00640 1 -- 172.21.15.98:0/1411837824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 msgr2=0x7f72e80c3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.226+0000 7f72ecc00640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72e80c3710 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f72d801c8a0 tx=0x7f72d8019810 comp rx=0 tx=0).stop 2024-05-27T00:22:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.226+0000 7f72ecc00640 1 -- 172.21.15.98:0/1411837824 shutdown_connections 2024-05-27T00:22:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.226+0000 7f72ecc00640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f72e8072090 0x7f72e81f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.226+0000 7f72ecc00640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f72c40678c0 0x7f72c4069d80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.226+0000 7f72ecc00640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72e81f0480 0x7f72e814da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.230+0000 7f72ecc00640 1 --2- 172.21.15.98:0/1411837824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72cc0039c0 0x7f72e80c3710 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.230+0000 7f72ecc00640 1 -- 172.21.15.98:0/1411837824 >> 172.21.15.98:0/1411837824 conn(0x7f72e8067530 msgr2=0x7f72e80cb260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.230+0000 7f72ecc00640 1 -- 172.21.15.98:0/1411837824 shutdown_connections 2024-05-27T00:22:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:06.230+0000 7f72ecc00640 1 -- 172.21.15.98:0/1411837824 wait complete. 2024-05-27T00:22:06.249 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:22:08.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:08.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb7d600640 1 Processor -- start 2024-05-27T00:22:08.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb7d600640 1 -- start start 2024-05-27T00:22:08.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb7d600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb780c67c0 0x7feb780c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:08.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb7d600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb780c70b0 con 0x7feb780cc4f0 2024-05-27T00:22:08.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb7d600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb780c8930 con 0x7feb780629d0 2024-05-27T00:22:08.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb7d600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7feb780c6330 con 0x7feb780c67c0 2024-05-27T00:22:08.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb77400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb780c67c0 0x7feb780c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:08.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb77e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7feb780cc4f0 0x7feb7815bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57986/0 (socket says 172.21.15.98:57986) 2024-05-27T00:22:08.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb77e00640 1 -- 172.21.15.98:0/3518774288 learned_addr learned my addr 172.21.15.98:0/3518774288 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:08.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb77400640 1 -- 172.21.15.98:0/3518774288 >> v1:172.21.15.204:6789/0 conn(0x7feb780629d0 legacy=0x7feb78162ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:08.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb77400640 1 -- 172.21.15.98:0/3518774288 >> v1:172.21.15.98:6789/0 conn(0x7feb780cc4f0 legacy=0x7feb7815bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:08.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb77400640 1 -- 172.21.15.98:0/3518774288 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb780c8c10 con 0x7feb780c67c0 2024-05-27T00:22:08.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.398+0000 7feb76a00640 1 -- 172.21.15.98:0/3518774288 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1582115214 0 0) 0x7feb780c70b0 con 0x7feb780cc4f0 2024-05-27T00:22:08.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb77400640 1 --2- 172.21.15.98:0/3518774288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780c67c0 0x7feb780c7720 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7feb640097c0 tx=0x7feb6401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be5c1e0e2a0eec7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:08.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb76a00640 1 -- 172.21.15.98:0/3518774288 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb640b0070 con 0x7feb780c67c0 2024-05-27T00:22:08.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb76a00640 1 -- 172.21.15.98:0/3518774288 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb640ab600 con 0x7feb780c67c0 2024-05-27T00:22:08.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb76a00640 1 -- 172.21.15.98:0/3518774288 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb640ab9d0 con 0x7feb780c67c0 2024-05-27T00:22:08.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb7d600640 1 -- 172.21.15.98:0/3518774288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780c67c0 msgr2=0x7feb780c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:08.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb7d600640 1 --2- 172.21.15.98:0/3518774288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780c67c0 0x7feb780c7720 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7feb640097c0 tx=0x7feb6401c460 comp rx=0 tx=0).stop 2024-05-27T00:22:08.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb7d600640 1 -- 172.21.15.98:0/3518774288 shutdown_connections 2024-05-27T00:22:08.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb7d600640 1 --2- 172.21.15.98:0/3518774288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780c67c0 0x7feb780c7720 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:08.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb7d600640 1 -- 172.21.15.98:0/3518774288 >> 172.21.15.98:0/3518774288 conn(0x7feb7806dd00 msgr2=0x7feb7806b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:08.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb7d600640 1 -- 172.21.15.98:0/3518774288 shutdown_connections 2024-05-27T00:22:08.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb7d600640 1 -- 172.21.15.98:0/3518774288 wait complete. 2024-05-27T00:22:08.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.402+0000 7feb7d600640 1 Processor -- start 2024-05-27T00:22:08.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7d600640 1 -- start start 2024-05-27T00:22:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb780629d0 0x7feb7813bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feb780c67c0 0x7feb7814d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7d600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780cc4f0 0x7feb7814d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7d600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb78167040 con 0x7feb780cc4f0 2024-05-27T00:22:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feb780c6e60 con 0x7feb780629d0 2024-05-27T00:22:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7feb781671c0 con 0x7feb780c67c0 2024-05-27T00:22:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7cc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780cc4f0 0x7feb7814d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7cc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780cc4f0 0x7feb7814d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36880/0 (socket says 172.21.15.98:36880) 2024-05-27T00:22:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7cc00640 1 -- 172.21.15.98:0/2324021210 learned_addr learned my addr 172.21.15.98:0/2324021210 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb77e00640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb780629d0 0x7feb7813bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb77400640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feb780c67c0 0x7feb7814d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7cc00640 1 -- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feb780c67c0 msgr2=0x7feb7814d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7cc00640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feb780c67c0 0x7feb7814d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7cc00640 1 -- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb780629d0 msgr2=0x7feb7813bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7cc00640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb780629d0 0x7feb7813bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7cc00640 1 -- 172.21.15.98:0/2324021210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7813e070 con 0x7feb780cc4f0 2024-05-27T00:22:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7cc00640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780cc4f0 0x7feb7814d670 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7feb7000b970 tx=0x7feb7000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:08.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb6fe00640 1 -- 172.21.15.98:0/2324021210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb70019070 con 0x7feb780cc4f0 2024-05-27T00:22:08.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb6fe00640 1 -- 172.21.15.98:0/2324021210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb7000cb40 con 0x7feb780cc4f0 2024-05-27T00:22:08.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb6fe00640 1 -- 172.21.15.98:0/2324021210 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb70014440 con 0x7feb780cc4f0 2024-05-27T00:22:08.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7d600640 1 -- 172.21.15.98:0/2324021210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb780c0600 con 0x7feb780cc4f0 2024-05-27T00:22:08.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7d600640 1 -- 172.21.15.98:0/2324021210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb780c10e0 con 0x7feb780cc4f0 2024-05-27T00:22:08.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.406+0000 7feb7d600640 1 -- 172.21.15.98:0/2324021210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb78075c40 con 0x7feb780cc4f0 2024-05-27T00:22:08.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.410+0000 7feb6fe00640 1 -- 172.21.15.98:0/2324021210 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feb7001c070 con 0x7feb780cc4f0 2024-05-27T00:22:08.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.410+0000 7feb6fe00640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feb580678c0 0x7feb58069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:08.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.410+0000 7feb6fe00640 1 -- 172.21.15.98:0/2324021210 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7feb70085b70 con 0x7feb780cc4f0 2024-05-27T00:22:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.410+0000 7feb77e00640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feb580678c0 0x7feb58069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.410+0000 7feb77e00640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feb580678c0 0x7feb58069d80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feb68000f70 tx=0x7feb68007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:08.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.414+0000 7feb6fe00640 1 -- 172.21.15.98:0/2324021210 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7feb70056440 con 0x7feb780cc4f0 2024-05-27T00:22:08.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 -- 172.21.15.98:0/2324021210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb780c7500 con 0x7feb780cc4f0 2024-05-27T00:22:08.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb6fe00640 1 -- 172.21.15.98:0/2324021210 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6839 (secure 0 0 0) 0x7feb70055d40 con 0x7feb780cc4f0 2024-05-27T00:22:08.603 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:08.603 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:54.706568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5202,"qdb_cluster":[5202]},"id":1}]} 2024-05-27T00:22:08.603 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-27T00:22:08.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 -- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feb580678c0 msgr2=0x7feb58069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:08.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feb580678c0 0x7feb58069d80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feb68000f70 tx=0x7feb68007c10 comp rx=0 tx=0).stop 2024-05-27T00:22:08.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 -- 172.21.15.98:0/2324021210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780cc4f0 msgr2=0x7feb7814d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:08.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780cc4f0 0x7feb7814d670 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7feb7000b970 tx=0x7feb7000be30 comp rx=0 tx=0).stop 2024-05-27T00:22:08.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 -- 172.21.15.98:0/2324021210 shutdown_connections 2024-05-27T00:22:08.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb780cc4f0 0x7feb7814d670 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:08.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feb780c67c0 0x7feb7814d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:08.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feb580678c0 0x7feb58069d80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:08.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 --2- 172.21.15.98:0/2324021210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb780629d0 0x7feb7813bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:08.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 -- 172.21.15.98:0/2324021210 >> 172.21.15.98:0/2324021210 conn(0x7feb7806dd00 msgr2=0x7feb780cba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:08.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 -- 172.21.15.98:0/2324021210 shutdown_connections 2024-05-27T00:22:08.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:08.602+0000 7feb7d600640 1 -- 172.21.15.98:0/2324021210 wait complete. 2024-05-27T00:22:08.624 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:22:10.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.770+0000 7fd3cc400640 1 Processor -- start 2024-05-27T00:22:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.770+0000 7fd3cc400640 1 -- start start 2024-05-27T00:22:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3cc400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd3c40cc1f0 0x7fd3c41592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3cc400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3c4073570 0x7fd3c41559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3cc400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3c40c1f60 con 0x7fd3c40cbe10 2024-05-27T00:22:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3cc400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd3c4159810 con 0x7fd3c4073570 2024-05-27T00:22:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3cc400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd3c40c6dd0 con 0x7fd3c40cc1f0 2024-05-27T00:22:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3ca400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3c4073570 0x7fd3c41559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3ca400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd3c4073570 0x7fd3c41559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36888/0 (socket says 172.21.15.98:36888) 2024-05-27T00:22:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3ca400640 1 -- 172.21.15.98:0/1378250354 learned_addr learned my addr 172.21.15.98:0/1378250354 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3c9000640 1 --2- 172.21.15.98:0/1378250354 >> v2:172.21.15.204:3300/0 conn(0x7fd3c40cc1f0 0x7fd3c41592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3ca400640 1 -- 172.21.15.98:0/1378250354 >> v2:172.21.15.204:3300/0 conn(0x7fd3c40cc1f0 msgr2=0x7fd3c41592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3ca400640 1 --2- 172.21.15.98:0/1378250354 >> v2:172.21.15.204:3300/0 conn(0x7fd3c40cc1f0 0x7fd3c41592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3ca400640 1 -- 172.21.15.98:0/1378250354 >> v1:172.21.15.204:6789/0 conn(0x7fd3c40cbe10 legacy=0x7fd3c40c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3ca400640 1 -- 172.21.15.98:0/1378250354 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c40c70b0 con 0x7fd3c4073570 2024-05-27T00:22:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.774+0000 7fd3c9000640 1 --2- 172.21.15.98:0/1378250354 >> v2:172.21.15.204:3300/0 conn(0x7fd3c40cc1f0 0x7fd3c41592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3ca400640 1 --2- 172.21.15.98:0/1378250354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 0x7fd3c41559e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40cc6d0 tx=0x7fd3bc01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405e0cd8bbc3c8b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3c3e00640 1 -- 172.21.15.98:0/1378250354 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0b4070 con 0x7fd3c4073570 2024-05-27T00:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3c3e00640 1 -- 172.21.15.98:0/1378250354 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc0af400 con 0x7fd3c4073570 2024-05-27T00:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3c3e00640 1 -- 172.21.15.98:0/1378250354 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0af7d0 con 0x7fd3c4073570 2024-05-27T00:22:10.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3cc400640 1 -- 172.21.15.98:0/1378250354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 msgr2=0x7fd3c41559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3cc400640 1 --2- 172.21.15.98:0/1378250354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 0x7fd3c41559e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40cc6d0 tx=0x7fd3bc01f1e0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3cc400640 1 -- 172.21.15.98:0/1378250354 shutdown_connections 2024-05-27T00:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3cc400640 1 --2- 172.21.15.98:0/1378250354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 0x7fd3c41559e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3cc400640 1 --2- 172.21.15.98:0/1378250354 >> v2:172.21.15.204:3300/0 conn(0x7fd3c40cc1f0 0x7fd3c41592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3cc400640 1 -- 172.21.15.98:0/1378250354 >> 172.21.15.98:0/1378250354 conn(0x7fd3c4067530 msgr2=0x7fd3c406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3cc400640 1 -- 172.21.15.98:0/1378250354 shutdown_connections 2024-05-27T00:22:10.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3cc400640 1 -- 172.21.15.98:0/1378250354 wait complete. 2024-05-27T00:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3cc400640 1 Processor -- start 2024-05-27T00:22:10.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.778+0000 7fd3cc400640 1 -- start start 2024-05-27T00:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3cc400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 0x7fd3c4071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3cc400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3c40cbe10 0x7fd3c41544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:10.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3cc400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3c40cc1f0 0x7fd3c414b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3cc400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c40c1330 con 0x7fd3c4073570 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3cc400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c40c1050 con 0x7fd3c40cc1f0 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3cc400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd3c40c79d0 con 0x7fd3c40cbe10 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 0x7fd3c4071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 0x7fd3c4071eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36896/0 (socket says 172.21.15.98:36896) 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9a00640 1 -- 172.21.15.98:0/571137604 learned_addr learned my addr 172.21.15.98:0/571137604 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9000640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3c40cbe10 0x7fd3c41544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3ca400640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3c40cc1f0 0x7fd3c414b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9a00640 1 -- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3c40cbe10 msgr2=0x7fd3c41544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9a00640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3c40cbe10 0x7fd3c41544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9a00640 1 -- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3c40cc1f0 msgr2=0x7fd3c414b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9a00640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3c40cc1f0 0x7fd3c414b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9a00640 1 -- 172.21.15.98:0/571137604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c41550b0 con 0x7fd3c4073570 2024-05-27T00:22:10.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9000640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3c40cbe10 0x7fd3c41544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c9a00640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 0x7fd3c4071eb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800c500 tx=0x7fd3b800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c1600640 1 -- 172.21.15.98:0/571137604 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8016020 con 0x7fd3c4073570 2024-05-27T00:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c1600640 1 -- 172.21.15.98:0/571137604 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3b8005220 con 0x7fd3c4073570 2024-05-27T00:22:10.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c1600640 1 -- 172.21.15.98:0/571137604 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8007030 con 0x7fd3c4073570 2024-05-27T00:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3cc400640 1 -- 172.21.15.98:0/571137604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c41f1450 con 0x7fd3c4073570 2024-05-27T00:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3cc400640 1 -- 172.21.15.98:0/571137604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c40710f0 con 0x7fd3c4073570 2024-05-27T00:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3cc400640 1 -- 172.21.15.98:0/571137604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3c40c5b00 con 0x7fd3c4073570 2024-05-27T00:22:10.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.782+0000 7fd3c1600640 1 -- 172.21.15.98:0/571137604 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd3b80071d0 con 0x7fd3c4073570 2024-05-27T00:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.786+0000 7fd3c1600640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3a40678c0 0x7fd3a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.786+0000 7fd3c1600640 1 -- 172.21.15.98:0/571137604 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7fd3b8085810 con 0x7fd3c4073570 2024-05-27T00:22:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.786+0000 7fd3c9000640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3a40678c0 0x7fd3a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.786+0000 7fd3c9000640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3a40678c0 0x7fd3a4069d80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40c2310 tx=0x7fd3b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:10.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.790+0000 7fd3c1600640 1 -- 172.21.15.98:0/571137604 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd3c40c5b00 con 0x7fd3c4073570 2024-05-27T00:22:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.974+0000 7fd3cc400640 1 -- 172.21.15.98:0/571137604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3c4075cb0 con 0x7fd3c4073570 2024-05-27T00:22:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3c1600640 1 -- 172.21.15.98:0/571137604 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6839 (secure 0 0 0) 0x7fd3b8020020 con 0x7fd3c4073570 2024-05-27T00:22:10.978 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:10.978 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:54.706568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5202,"qdb_cluster":[5202]},"id":1}]} 2024-05-27T00:22:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-27T00:22:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 -- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3a40678c0 msgr2=0x7fd3a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3a40678c0 0x7fd3a4069d80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40c2310 tx=0x7fd3b0008040 comp rx=0 tx=0).stop 2024-05-27T00:22:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 -- 172.21.15.98:0/571137604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 msgr2=0x7fd3c4071eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 0x7fd3c4071eb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800c500 tx=0x7fd3b800c9c0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 -- 172.21.15.98:0/571137604 shutdown_connections 2024-05-27T00:22:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3a40678c0 0x7fd3a4069d80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3c40cc1f0 0x7fd3c414b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3c40cbe10 0x7fd3c41544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 --2- 172.21.15.98:0/571137604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4073570 0x7fd3c4071eb0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 -- 172.21.15.98:0/571137604 >> 172.21.15.98:0/571137604 conn(0x7fd3c4067530 msgr2=0x7fd3c406b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 -- 172.21.15.98:0/571137604 shutdown_connections 2024-05-27T00:22:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:10.978+0000 7fd3cc400640 1 -- 172.21.15.98:0/571137604 wait complete. 2024-05-27T00:22:10.999 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:22:13.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:13.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.142+0000 7f7417a00640 1 Processor -- start 2024-05-27T00:22:13.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.142+0000 7f7417a00640 1 -- start start 2024-05-27T00:22:13.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f7417a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f74100cc8f0 0x7f7410070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:13.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f7417a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74100c1000 con 0x7f74101511d0 2024-05-27T00:22:13.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f7417a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74101515b0 con 0x7f74100c8670 2024-05-27T00:22:13.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f7417a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f74100b1dd0 con 0x7f74100cc8f0 2024-05-27T00:22:13.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f7415000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f74101511d0 0x7f74100c1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57998/0 (socket says 172.21.15.98:57998) 2024-05-27T00:22:13.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f7415000640 1 -- 172.21.15.98:0/2476167310 learned_addr learned my addr 172.21.15.98:0/2476167310 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:13.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f740fe00640 1 --2- 172.21.15.98:0/2476167310 >> v2:172.21.15.98:3300/0 conn(0x7f74100cc8f0 0x7f7410070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:13.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f740fe00640 1 -- 172.21.15.98:0/2476167310 >> v1:172.21.15.204:6789/0 conn(0x7f74100c8670 legacy=0x7f74101617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:13.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f740fe00640 1 -- 172.21.15.98:0/2476167310 >> v1:172.21.15.98:6789/0 conn(0x7f74101511d0 legacy=0x7f74100c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:13.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f740fe00640 1 -- 172.21.15.98:0/2476167310 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74100c83c0 con 0x7f74100cc8f0 2024-05-27T00:22:13.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.146+0000 7f740f400640 1 -- 172.21.15.98:0/2476167310 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3466524870 0 0) 0x7f74100c1000 con 0x7f74101511d0 2024-05-27T00:22:13.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f740fe00640 1 --2- 172.21.15.98:0/2476167310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 0x7f7410070e10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0097c0 tx=0x7f73fc01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814487166ef19e7f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:13.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f740f400640 1 -- 172.21.15.98:0/2476167310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc0b0070 con 0x7f74100cc8f0 2024-05-27T00:22:13.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f740f400640 1 -- 172.21.15.98:0/2476167310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73fc0ab600 con 0x7f74100cc8f0 2024-05-27T00:22:13.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f740f400640 1 -- 172.21.15.98:0/2476167310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc0ab9d0 con 0x7f74100cc8f0 2024-05-27T00:22:13.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 -- 172.21.15.98:0/2476167310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 msgr2=0x7f7410070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:13.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 --2- 172.21.15.98:0/2476167310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 0x7f7410070e10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0097c0 tx=0x7f73fc01c3d0 comp rx=0 tx=0).stop 2024-05-27T00:22:13.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 -- 172.21.15.98:0/2476167310 shutdown_connections 2024-05-27T00:22:13.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 --2- 172.21.15.98:0/2476167310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 0x7f7410070e10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:13.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 -- 172.21.15.98:0/2476167310 >> 172.21.15.98:0/2476167310 conn(0x7f7410067530 msgr2=0x7f741006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:13.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 -- 172.21.15.98:0/2476167310 shutdown_connections 2024-05-27T00:22:13.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 -- 172.21.15.98:0/2476167310 wait complete. 2024-05-27T00:22:13.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 Processor -- start 2024-05-27T00:22:13.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 -- start start 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74100c8670 0x7f74100ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 0x7f7410145f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74101511d0 0x7f74101464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f74100c8eb0 con 0x7f74100cc8f0 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f74100c1a20 con 0x7f74101511d0 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.150+0000 7f7417a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f74100c2150 con 0x7f74100c8670 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 0x7f7410145f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 0x7f7410145f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36904/0 (socket says 172.21.15.98:36904) 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740fe00640 1 -- 172.21.15.98:0/3227419269 learned_addr learned my addr 172.21.15.98:0/3227419269 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f7415000640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74100c8670 0x7f74100ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:13.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f7415a00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74101511d0 0x7f74101464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740fe00640 1 -- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74100c8670 msgr2=0x7f74100ca780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740fe00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74100c8670 0x7f74100ca780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740fe00640 1 -- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74101511d0 msgr2=0x7f74101464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740fe00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74101511d0 0x7f74101464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740fe00640 1 -- 172.21.15.98:0/3227419269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7410140370 con 0x7f74100cc8f0 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740fe00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 0x7f7410145f70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f73fc018fe0 tx=0x7f73fc01ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740cc00640 1 -- 172.21.15.98:0/3227419269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc0b0050 con 0x7f74100cc8f0 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740cc00640 1 -- 172.21.15.98:0/3227419269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73fc0bc070 con 0x7f74100cc8f0 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740cc00640 1 -- 172.21.15.98:0/3227419269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc0b5a50 con 0x7f74100cc8f0 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f7417a00640 1 -- 172.21.15.98:0/3227419269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7410145a90 con 0x7f74100cc8f0 2024-05-27T00:22:13.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f7417a00640 1 -- 172.21.15.98:0/3227419269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74100c9f20 con 0x7f74100cc8f0 2024-05-27T00:22:13.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f7417a00640 1 -- 172.21.15.98:0/3227419269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7410145c60 con 0x7f74100cc8f0 2024-05-27T00:22:13.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740cc00640 1 -- 172.21.15.98:0/3227419269 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f73fc0ab4a0 con 0x7f74100cc8f0 2024-05-27T00:22:13.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740cc00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f73f00678c0 0x7f73f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:13.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.154+0000 7f740cc00640 1 -- 172.21.15.98:0/3227419269 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f73fc120920 con 0x7f74100cc8f0 2024-05-27T00:22:13.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.158+0000 7f7415000640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f73f00678c0 0x7f73f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:13.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.158+0000 7f7415000640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f73f00678c0 0x7f73f0069d80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7404000f70 tx=0x7f7404007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:13.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.158+0000 7f740cc00640 1 -- 172.21.15.98:0/3227419269 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f73fc016050 con 0x7f74100cc8f0 2024-05-27T00:22:13.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.346+0000 7f7417a00640 1 -- 172.21.15.98:0/3227419269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7410075cb0 con 0x7f74100cc8f0 2024-05-27T00:22:13.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f740cc00640 1 -- 172.21.15.98:0/3227419269 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6839 (secure 0 0 0) 0x7f73fc0f1160 con 0x7f74100cc8f0 2024-05-27T00:22:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:13.350 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:54.706568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5202,"qdb_cluster":[5202]},"id":1}]} 2024-05-27T00:22:13.350 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-27T00:22:13.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 -- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f73f00678c0 msgr2=0x7f73f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:13.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f73f00678c0 0x7f73f0069d80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7404000f70 tx=0x7f7404007c10 comp rx=0 tx=0).stop 2024-05-27T00:22:13.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 -- 172.21.15.98:0/3227419269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 msgr2=0x7f7410145f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:13.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 0x7f7410145f70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f73fc018fe0 tx=0x7f73fc01ac10 comp rx=0 tx=0).stop 2024-05-27T00:22:13.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 -- 172.21.15.98:0/3227419269 shutdown_connections 2024-05-27T00:22:13.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f73f00678c0 0x7f73f0069d80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:13.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74101511d0 0x7f74101464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:13.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74100cc8f0 0x7f7410145f70 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:13.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 --2- 172.21.15.98:0/3227419269 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74100c8670 0x7f74100ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:13.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 -- 172.21.15.98:0/3227419269 >> 172.21.15.98:0/3227419269 conn(0x7f7410067530 msgr2=0x7f74101539f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:13.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 -- 172.21.15.98:0/3227419269 shutdown_connections 2024-05-27T00:22:13.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:13.350+0000 7f7417a00640 1 -- 172.21.15.98:0/3227419269 wait complete. 2024-05-27T00:22:13.371 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:22:15.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.526+0000 7f232fe00640 1 Processor -- start 2024-05-27T00:22:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.526+0000 7f232fe00640 1 -- start start 2024-05-27T00:22:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.526+0000 7f232fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2328073570 0x7f23281592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.526+0000 7f232fe00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23280c1f60 con 0x7f23280cc1f0 2024-05-27T00:22:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f232fe00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23280c6dd0 con 0x7f23280cbe10 2024-05-27T00:22:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f232fe00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2328159810 con 0x7f2328073570 2024-05-27T00:22:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f232de00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2328073570 0x7f23281592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f232de00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2328073570 0x7f23281592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59136/0 (socket says 172.21.15.98:59136) 2024-05-27T00:22:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f232de00640 1 -- 172.21.15.98:0/4091633955 learned_addr learned my addr 172.21.15.98:0/4091633955 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f232ca00640 1 --1- 172.21.15.98:0/4091633955 >> v1:172.21.15.98:6789/0 conn(0x7f23280cc1f0 0x7f23281559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48684/0 (socket says 172.21.15.98:48684) 2024-05-27T00:22:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f232de00640 1 -- 172.21.15.98:0/4091633955 >> v1:172.21.15.204:6789/0 conn(0x7f23280cbe10 legacy=0x7f23280c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f232de00640 1 -- 172.21.15.98:0/4091633955 >> v1:172.21.15.98:6789/0 conn(0x7f23280cc1f0 legacy=0x7f23281559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:15.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f232de00640 1 -- 172.21.15.98:0/4091633955 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23280c70b0 con 0x7f2328073570 2024-05-27T00:22:15.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f232de00640 1 --2- 172.21.15.98:0/4091633955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 0x7f23281592d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2320009e50 tx=0x7f232001f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a019f7ea1161190e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f2327e00640 1 -- 172.21.15.98:0/4091633955 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23200b4070 con 0x7f2328073570 2024-05-27T00:22:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f2327e00640 1 -- 172.21.15.98:0/4091633955 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23200af400 con 0x7f2328073570 2024-05-27T00:22:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.530+0000 7f2327e00640 1 -- 172.21.15.98:0/4091633955 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23200af7d0 con 0x7f2328073570 2024-05-27T00:22:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 -- 172.21.15.98:0/4091633955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 msgr2=0x7f23281592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 --2- 172.21.15.98:0/4091633955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 0x7f23281592d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2320009e50 tx=0x7f232001f210 comp rx=0 tx=0).stop 2024-05-27T00:22:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 -- 172.21.15.98:0/4091633955 shutdown_connections 2024-05-27T00:22:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 --2- 172.21.15.98:0/4091633955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 0x7f23281592d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 -- 172.21.15.98:0/4091633955 >> 172.21.15.98:0/4091633955 conn(0x7f2328067530 msgr2=0x7f232806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 -- 172.21.15.98:0/4091633955 shutdown_connections 2024-05-27T00:22:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 -- 172.21.15.98:0/4091633955 wait complete. 2024-05-27T00:22:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 Processor -- start 2024-05-27T00:22:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 -- start start 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 0x7f23280c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23280cbe10 0x7f23281e4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23280cc1f0 0x7f23280c3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f23280c1330 con 0x7f2328073570 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f23280c1050 con 0x7f23280cc1f0 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f23280c79d0 con 0x7f23280cbe10 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 0x7f23280c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 0x7f23280c52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59148/0 (socket says 172.21.15.98:59148) 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232d400640 1 -- 172.21.15.98:0/233809867 learned_addr learned my addr 172.21.15.98:0/233809867 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232ca00640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23280cbe10 0x7f23281e4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232de00640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23280cc1f0 0x7f23280c3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232d400640 1 -- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23280cbe10 msgr2=0x7f23281e4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232d400640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23280cbe10 0x7f23281e4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232d400640 1 -- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23280cc1f0 msgr2=0x7f23280c3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232d400640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23280cc1f0 0x7f23280c3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.534+0000 7f232d400640 1 -- 172.21.15.98:0/233809867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23281ee7b0 con 0x7f2328073570 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f232d400640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 0x7f23280c52c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f231c00c530 tx=0x7f231c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f2325600640 1 -- 172.21.15.98:0/233809867 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c016020 con 0x7f2328073570 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f2325600640 1 -- 172.21.15.98:0/233809867 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f231c005220 con 0x7f2328073570 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f2325600640 1 -- 172.21.15.98:0/233809867 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c007030 con 0x7f2328073570 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f232fe00640 1 -- 172.21.15.98:0/233809867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23281eea90 con 0x7f2328073570 2024-05-27T00:22:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f232fe00640 1 -- 172.21.15.98:0/233809867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23281ef050 con 0x7f2328073570 2024-05-27T00:22:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f232fe00640 1 -- 172.21.15.98:0/233809867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2328075cb0 con 0x7f2328073570 2024-05-27T00:22:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f2325600640 1 -- 172.21.15.98:0/233809867 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f231c0071d0 con 0x7f2328073570 2024-05-27T00:22:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f2325600640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f23080678c0 0x7f2308069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f2325600640 1 -- 172.21.15.98:0/233809867 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7f231c085910 con 0x7f2328073570 2024-05-27T00:22:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.538+0000 7f232ca00640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f23080678c0 0x7f2308069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:15.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.542+0000 7f232ca00640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f23080678c0 0x7f2308069d80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2328157b90 tx=0x7f2314007cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.542+0000 7f2325600640 1 -- 172.21.15.98:0/233809867 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f231c020020 con 0x7f2328073570 2024-05-27T00:22:15.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.734+0000 7f232fe00640 1 -- 172.21.15.98:0/233809867 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23281ef250 con 0x7f2328073570 2024-05-27T00:22:15.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f2325600640 1 -- 172.21.15.98:0/233809867 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6839 (secure 0 0 0) 0x7f231c0074c0 con 0x7f2328073570 2024-05-27T00:22:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:15.738 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:54.706568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5202,"qdb_cluster":[5202]},"id":1}]} 2024-05-27T00:22:15.738 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-27T00:22:15.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 -- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f23080678c0 msgr2=0x7f2308069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:15.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f23080678c0 0x7f2308069d80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2328157b90 tx=0x7f2314007cf0 comp rx=0 tx=0).stop 2024-05-27T00:22:15.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 -- 172.21.15.98:0/233809867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 msgr2=0x7f23280c52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 0x7f23280c52c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f231c00c530 tx=0x7f231c00c9f0 comp rx=0 tx=0).stop 2024-05-27T00:22:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 -- 172.21.15.98:0/233809867 shutdown_connections 2024-05-27T00:22:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f23080678c0 0x7f2308069d80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23280cc1f0 0x7f23280c3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23280cbe10 0x7f23281e4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 --2- 172.21.15.98:0/233809867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2328073570 0x7f23280c52c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 -- 172.21.15.98:0/233809867 >> 172.21.15.98:0/233809867 conn(0x7f2328067530 msgr2=0x7f23280c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 -- 172.21.15.98:0/233809867 shutdown_connections 2024-05-27T00:22:15.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:15.738+0000 7f232fe00640 1 -- 172.21.15.98:0/233809867 wait complete. 2024-05-27T00:22:15.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:22:17.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:22:17.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:22:17.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa197000640 1 Processor -- start 2024-05-27T00:22:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa197000640 1 -- start start 2024-05-27T00:22:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa197000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900735d0 0x7fa190151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa197000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1900c79f0 0x7fa190151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa197000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1900c17d0 con 0x7fa1900731f0 2024-05-27T00:22:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa197000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa190071990 con 0x7fa1900c79f0 2024-05-27T00:22:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa197000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa190070e10 con 0x7fa1900735d0 2024-05-27T00:22:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa195000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1900c79f0 0x7fa190151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa195000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1900c79f0 0x7fa190151970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59154/0 (socket says 172.21.15.98:59154) 2024-05-27T00:22:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa195000640 1 -- 172.21.15.98:0/3365265778 learned_addr learned my addr 172.21.15.98:0/3365265778 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa18f400640 1 --2- 172.21.15.98:0/3365265778 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900735d0 0x7fa190151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa195000640 1 -- 172.21.15.98:0/3365265778 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900735d0 msgr2=0x7fa190151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa195000640 1 --2- 172.21.15.98:0/3365265778 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900735d0 0x7fa190151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa195000640 1 -- 172.21.15.98:0/3365265778 >> v1:172.21.15.204:6789/0 conn(0x7fa1900731f0 legacy=0x7fa1900c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.906+0000 7fa195000640 1 -- 172.21.15.98:0/3365265778 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1900710f0 con 0x7fa1900c79f0 2024-05-27T00:22:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa195000640 1 --2- 172.21.15.98:0/3365265778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 0x7fa190151970 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa188009e50 tx=0x7fa18801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593c7f26c08f1868 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa18ea00640 1 -- 172.21.15.98:0/3365265778 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1880b4070 con 0x7fa1900c79f0 2024-05-27T00:22:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa18ea00640 1 -- 172.21.15.98:0/3365265778 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1880af400 con 0x7fa1900c79f0 2024-05-27T00:22:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa18ea00640 1 -- 172.21.15.98:0/3365265778 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1880af7d0 con 0x7fa1900c79f0 2024-05-27T00:22:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa197000640 1 -- 172.21.15.98:0/3365265778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 msgr2=0x7fa190151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa197000640 1 --2- 172.21.15.98:0/3365265778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 0x7fa190151970 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa188009e50 tx=0x7fa18801f210 comp rx=0 tx=0).stop 2024-05-27T00:22:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa197000640 1 -- 172.21.15.98:0/3365265778 shutdown_connections 2024-05-27T00:22:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa197000640 1 --2- 172.21.15.98:0/3365265778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 0x7fa190151970 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa197000640 1 --2- 172.21.15.98:0/3365265778 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900735d0 0x7fa190151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa197000640 1 -- 172.21.15.98:0/3365265778 >> 172.21.15.98:0/3365265778 conn(0x7fa190067530 msgr2=0x7fa19006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa197000640 1 -- 172.21.15.98:0/3365265778 shutdown_connections 2024-05-27T00:22:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa197000640 1 -- 172.21.15.98:0/3365265778 wait complete. 2024-05-27T00:22:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa197000640 1 Processor -- start 2024-05-27T00:22:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.910+0000 7fa197000640 1 -- start start 2024-05-27T00:22:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa197000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900731f0 0x7fa190152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa197000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1900735d0 0x7fa19014a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa197000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 0x7fa19015b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa197000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa1900c8400 con 0x7fa1900c79f0 2024-05-27T00:22:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa197000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa1900716b0 con 0x7fa1900735d0 2024-05-27T00:22:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa197000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa190157970 con 0x7fa1900731f0 2024-05-27T00:22:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa195000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 0x7fa19015b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa195000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 0x7fa19015b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59168/0 (socket says 172.21.15.98:59168) 2024-05-27T00:22:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa195000640 1 -- 172.21.15.98:0/2937594443 learned_addr learned my addr 172.21.15.98:0/2937594443 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa18fe00640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900731f0 0x7fa190152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa18f400640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1900735d0 0x7fa19014a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa195000640 1 -- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900731f0 msgr2=0x7fa190152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa195000640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900731f0 0x7fa190152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa195000640 1 -- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1900735d0 msgr2=0x7fa19014a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa195000640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1900735d0 0x7fa19014a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa195000640 1 -- 172.21.15.98:0/2937594443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1901484f0 con 0x7fa1900c79f0 2024-05-27T00:22:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa18fe00640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900731f0 0x7fa190152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa195000640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 0x7fa19015b120 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa18801ce40 tx=0x7fa18801bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa187e00640 1 -- 172.21.15.98:0/2937594443 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1880b4040 con 0x7fa1900c79f0 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa187e00640 1 -- 172.21.15.98:0/2937594443 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1880afd00 con 0x7fa1900c79f0 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa197000640 1 -- 172.21.15.98:0/2937594443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1901487d0 con 0x7fa1900c79f0 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa187e00640 1 -- 172.21.15.98:0/2937594443 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188009510 con 0x7fa1900c79f0 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa197000640 1 -- 172.21.15.98:0/2937594443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa190148d90 con 0x7fa1900c79f0 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.914+0000 7fa197000640 1 -- 172.21.15.98:0/2937594443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa190075cb0 con 0x7fa1900c79f0 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.918+0000 7fa187e00640 1 -- 172.21.15.98:0/2937594443 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa1880c3020 con 0x7fa1900c79f0 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.918+0000 7fa187e00640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1700678f0 0x7fa170069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.918+0000 7fa187e00640 1 -- 172.21.15.98:0/2937594443 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6169+0+0 (secure 0 0 0) 0x7fa188123950 con 0x7fa1900c79f0 2024-05-27T00:22:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.918+0000 7fa18fe00640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1700678f0 0x7fa170069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.918+0000 7fa18fe00640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1700678f0 0x7fa170069db0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa180001710 tx=0x7fa180009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:17.922+0000 7fa187e00640 1 -- 172.21.15.98:0/2937594443 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa1880ca020 con 0x7fa1900c79f0 2024-05-27T00:22:18.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.110+0000 7fa197000640 1 -- 172.21.15.98:0/2937594443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa190158510 con 0x7fa1900c79f0 2024-05-27T00:22:18.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.110+0000 7fa187e00640 1 -- 172.21.15.98:0/2937594443 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6839 (secure 0 0 0) 0x7fa188018d70 con 0x7fa1900c79f0 2024-05-27T00:22:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:18.112 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:21:54.706568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":31,"addr":"172.21.15.98:6837/1698253956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1698253956},{"type":"v1","addr":"172.21.15.98:6837","nonce":1698253956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5202,"qdb_cluster":[5202]},"id":1}]} 2024-05-27T00:22:18.113 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-05-27T00:22:18.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 -- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1700678f0 msgr2=0x7fa170069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:18.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1700678f0 0x7fa170069db0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa180001710 tx=0x7fa180009040 comp rx=0 tx=0).stop 2024-05-27T00:22:18.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 -- 172.21.15.98:0/2937594443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 msgr2=0x7fa19015b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:18.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 0x7fa19015b120 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa18801ce40 tx=0x7fa18801bfb0 comp rx=0 tx=0).stop 2024-05-27T00:22:18.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 -- 172.21.15.98:0/2937594443 shutdown_connections 2024-05-27T00:22:18.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1900c79f0 0x7fa19015b120 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:18.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa1700678f0 0x7fa170069db0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:18.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1900735d0 0x7fa19014a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:18.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 --2- 172.21.15.98:0/2937594443 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900731f0 0x7fa190152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:18.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 -- 172.21.15.98:0/2937594443 >> 172.21.15.98:0/2937594443 conn(0x7fa190067530 msgr2=0x7fa1900c6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:18.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 -- 172.21.15.98:0/2937594443 shutdown_connections 2024-05-27T00:22:18.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:18.114+0000 7fa197000640 1 -- 172.21.15.98:0/2937594443 wait complete. 2024-05-27T00:22:18.135 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:22:20.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.282+0000 7f36f7400640 1 Processor -- start 2024-05-27T00:22:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.282+0000 7f36f7400640 1 -- start start 2024-05-27T00:22:20.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36f7400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36f00cc1f0 0x7f36f01592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36f7400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f36f0073570 0x7f36f01559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36f7400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36f00c1f60 con 0x7f36f00cbe10 2024-05-27T00:22:20.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36f7400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f36f0159810 con 0x7f36f00cc1f0 2024-05-27T00:22:20.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36f7400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f36f00c6dd0 con 0x7f36f0073570 2024-05-27T00:22:20.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36e7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36f00cc1f0 0x7f36f01592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36e7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f36f00cc1f0 0x7f36f01592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59176/0 (socket says 172.21.15.98:59176) 2024-05-27T00:22:20.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36e7e00640 1 -- 172.21.15.98:0/4235291054 learned_addr learned my addr 172.21.15.98:0/4235291054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:20.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36f4a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f36f00cbe10 0x7f36f00c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48690/0 (socket says 172.21.15.98:48690) 2024-05-27T00:22:20.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36f5400640 1 --2- 172.21.15.98:0/4235291054 >> v2:172.21.15.204:3300/0 conn(0x7f36f0073570 0x7f36f01559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36e7e00640 1 -- 172.21.15.98:0/4235291054 >> v2:172.21.15.204:3300/0 conn(0x7f36f0073570 msgr2=0x7f36f01559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:20.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36e7e00640 1 --2- 172.21.15.98:0/4235291054 >> v2:172.21.15.204:3300/0 conn(0x7f36f0073570 0x7f36f01559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36e7e00640 1 -- 172.21.15.98:0/4235291054 >> v1:172.21.15.98:6789/0 conn(0x7f36f00cbe10 legacy=0x7f36f00c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.286+0000 7f36e7e00640 1 -- 172.21.15.98:0/4235291054 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f00c70b0 con 0x7f36f00cc1f0 2024-05-27T00:22:20.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36e7e00640 1 --2- 172.21.15.98:0/4235291054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cc1f0 0x7f36f01592d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f36f006e230 tx=0x7f36d801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71caf52fc0242cd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:20.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36efe00640 1 -- 172.21.15.98:0/4235291054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d80b5070 con 0x7f36f00cc1f0 2024-05-27T00:22:20.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36efe00640 1 -- 172.21.15.98:0/4235291054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36d80a8a10 con 0x7f36f00cc1f0 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36efe00640 1 -- 172.21.15.98:0/4235291054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d80a8de0 con 0x7f36f00cc1f0 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36f7400640 1 -- 172.21.15.98:0/4235291054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cc1f0 msgr2=0x7f36f01592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36f7400640 1 --2- 172.21.15.98:0/4235291054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cc1f0 0x7f36f01592d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f36f006e230 tx=0x7f36d801ed10 comp rx=0 tx=0).stop 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36f7400640 1 -- 172.21.15.98:0/4235291054 shutdown_connections 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36f7400640 1 --2- 172.21.15.98:0/4235291054 >> v2:172.21.15.204:3300/0 conn(0x7f36f0073570 0x7f36f01559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36f7400640 1 --2- 172.21.15.98:0/4235291054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cc1f0 0x7f36f01592d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36f7400640 1 -- 172.21.15.98:0/4235291054 >> 172.21.15.98:0/4235291054 conn(0x7f36f0067530 msgr2=0x7f36f006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36f7400640 1 -- 172.21.15.98:0/4235291054 shutdown_connections 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36f7400640 1 -- 172.21.15.98:0/4235291054 wait complete. 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36f7400640 1 Processor -- start 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.290+0000 7f36f7400640 1 -- start start 2024-05-27T00:22:20.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36f0073570 0x7f36f0157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cbe10 0x7f36f01533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36f00cc1f0 0x7f36f0153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f7400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36f00c1330 con 0x7f36f00cbe10 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f7400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f36f00c1050 con 0x7f36f0073570 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f7400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f36f00c79d0 con 0x7f36f00cc1f0 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cbe10 0x7f36f01533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cbe10 0x7f36f01533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59188/0 (socket says 172.21.15.98:59188) 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36e7e00640 1 -- 172.21.15.98:0/243378702 learned_addr learned my addr 172.21.15.98:0/243378702 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f5400640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36f00cc1f0 0x7f36f0153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f4a00640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36f0073570 0x7f36f0157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36e7e00640 1 -- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36f00cc1f0 msgr2=0x7f36f0153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36e7e00640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36f00cc1f0 0x7f36f0153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36e7e00640 1 -- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36f0073570 msgr2=0x7f36f0157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36e7e00640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36f0073570 0x7f36f0157a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36e7e00640 1 -- 172.21.15.98:0/243378702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f01f11c0 con 0x7f36f00cbe10 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f5400640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36f00cc1f0 0x7f36f0153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f4a00640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36f0073570 0x7f36f0157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36e7e00640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cbe10 0x7f36f01533d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f36d80b3040 tx=0x7f36d801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36ed600640 1 -- 172.21.15.98:0/243378702 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d80b5070 con 0x7f36f00cbe10 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36ed600640 1 -- 172.21.15.98:0/243378702 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36d80b0b30 con 0x7f36f00cbe10 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36ed600640 1 -- 172.21.15.98:0/243378702 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d80bb410 con 0x7f36f00cbe10 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f7400640 1 -- 172.21.15.98:0/243378702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f01f1460 con 0x7f36f00cbe10 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f7400640 1 -- 172.21.15.98:0/243378702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f00c4e20 con 0x7f36f00cbe10 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36f7400640 1 -- 172.21.15.98:0/243378702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f0075cb0 con 0x7f36f00cbe10 2024-05-27T00:22:20.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.294+0000 7f36ed600640 1 -- 172.21.15.98:0/243378702 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f36d80c2060 con 0x7f36f00cbe10 2024-05-27T00:22:20.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.298+0000 7f36ed600640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36cc0678c0 0x7f36cc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.298+0000 7f36ed600640 1 -- 172.21.15.98:0/243378702 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f36d8122fc0 con 0x7f36f00cbe10 2024-05-27T00:22:20.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.298+0000 7f36f4a00640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36cc0678c0 0x7f36cc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.298+0000 7f36f4a00640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36cc0678c0 0x7f36cc069d80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f36e0007fd0 tx=0x7f36e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:20.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.302+0000 7f36ed600640 1 -- 172.21.15.98:0/243378702 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f36d8018870 con 0x7f36f00cbe10 2024-05-27T00:22:20.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.490+0000 7f36f7400640 1 -- 172.21.15.98:0/243378702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36f00c2be0 con 0x7f36f00cbe10 2024-05-27T00:22:20.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.490+0000 7f36ed600640 1 -- 172.21.15.98:0/243378702 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6006 (secure 0 0 0) 0x7f36d80f38d0 con 0x7f36f00cbe10 2024-05-27T00:22:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:20.491 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:20.214130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:20.492 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-05-27T00:22:20.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.490+0000 7f36f7400640 1 -- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36cc0678c0 msgr2=0x7f36cc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.490+0000 7f36f7400640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36cc0678c0 0x7f36cc069d80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f36e0007fd0 tx=0x7f36e0009040 comp rx=0 tx=0).stop 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.490+0000 7f36f7400640 1 -- 172.21.15.98:0/243378702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cbe10 msgr2=0x7f36f01533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.490+0000 7f36f7400640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cbe10 0x7f36f01533d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f36d80b3040 tx=0x7f36d801bab0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.494+0000 7f36f7400640 1 -- 172.21.15.98:0/243378702 shutdown_connections 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.494+0000 7f36f7400640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36f00cc1f0 0x7f36f0153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.494+0000 7f36f7400640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36f00cbe10 0x7f36f01533d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.494+0000 7f36f7400640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36cc0678c0 0x7f36cc069d80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.494+0000 7f36f7400640 1 --2- 172.21.15.98:0/243378702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36f0073570 0x7f36f0157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.494+0000 7f36f7400640 1 -- 172.21.15.98:0/243378702 >> 172.21.15.98:0/243378702 conn(0x7f36f0067530 msgr2=0x7f36f00ae8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.494+0000 7f36f7400640 1 -- 172.21.15.98:0/243378702 shutdown_connections 2024-05-27T00:22:20.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.494+0000 7f36f7400640 1 -- 172.21.15.98:0/243378702 wait complete. 2024-05-27T00:22:20.513 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-05-27T00:22:20.513 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:22:20.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe25ae00640 1 Processor -- start 2024-05-27T00:22:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe25ae00640 1 -- start start 2024-05-27T00:22:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe25ae00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe25406f760 0x7fe2540ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe25ae00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe25406cfb0 con 0x7fe2540cdf90 2024-05-27T00:22:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe25ae00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2540ca840 con 0x7fe2540c4fd0 2024-05-27T00:22:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe25ae00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fe25406d500 con 0x7fe25406f760 2024-05-27T00:22:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe253e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe2540cdf90 0x7fe2540c8fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48694/0 (socket says 172.21.15.98:48694) 2024-05-27T00:22:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe253e00640 1 -- 172.21.15.98:0/1809155536 learned_addr learned my addr 172.21.15.98:0/1809155536 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe258e00640 1 --2- 172.21.15.98:0/1809155536 >> v2:172.21.15.204:3300/0 conn(0x7fe25406f760 0x7fe2540ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2202151474 0 0) 0x7fe25406cfb0 con 0x7fe2540cdf90 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe238003530 con 0x7fe2540cdf90 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2047698660 0 0) 0x7fe238003530 con 0x7fe2540cdf90 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 >> v2:172.21.15.204:3300/0 conn(0x7fe25406f760 msgr2=0x7fe2540ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 --2- 172.21.15.98:0/1809155536 >> v2:172.21.15.204:3300/0 conn(0x7fe25406f760 0x7fe2540ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 >> v1:172.21.15.204:6789/0 conn(0x7fe2540c4fd0 legacy=0x7fe254163130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe25406b610 con 0x7fe2540cdf90 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fe244001a00 con 0x7fe2540cdf90 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe258e00640 1 --2- 172.21.15.98:0/1809155536 >> v2:172.21.15.204:3300/0 conn(0x7fe25406f760 0x7fe2540ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 >> v1:172.21.15.98:6789/0 conn(0x7fe2540cdf90 legacy=0x7fe2540c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2380039c0 0x7fe238003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe238004810 0x7fe238004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2380051d0 0x7fe23800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.650+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe25406d500 con 0x7fe2380039c0 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe23800c080 con 0x7fe238004810 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe23800c280 con 0x7fe2380051d0 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe253e00640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2380039c0 0x7fe238003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe253400640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe238004810 0x7fe238004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe258e00640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2380051d0 0x7fe23800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe253e00640 1 -- 172.21.15.98:0/1809155536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2380051d0 msgr2=0x7fe23800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe253e00640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2380051d0 0x7fe23800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe253e00640 1 -- 172.21.15.98:0/1809155536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe238004810 msgr2=0x7fe238004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe253e00640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe238004810 0x7fe238004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe253e00640 1 -- 172.21.15.98:0/1809155536 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe238003530 con 0x7fe2380039c0 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe258e00640 1 -- 172.21.15.98:0/1809155536 reap_dead start 2024-05-27T00:22:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe253400640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe238004810 0x7fe238004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:22:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe253e00640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2380039c0 0x7fe238003de0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe24400a860 tx=0x7fe24401fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb132568bfb10cf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24400a480 con 0x7fe2380039c0 2024-05-27T00:22:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2440aa690 con 0x7fe2380039c0 2024-05-27T00:22:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.654+0000 7fe252a00640 1 -- 172.21.15.98:0/1809155536 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2440aaae0 con 0x7fe2380039c0 2024-05-27T00:22:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 -- 172.21.15.98:0/1809155536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2380039c0 msgr2=0x7fe238003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2380039c0 0x7fe238003de0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe24400a860 tx=0x7fe24401fa60 comp rx=0 tx=0).stop 2024-05-27T00:22:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 -- 172.21.15.98:0/1809155536 shutdown_connections 2024-05-27T00:22:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 --2- 172.21.15.98:0/1809155536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2380039c0 0x7fe238003de0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 -- 172.21.15.98:0/1809155536 >> 172.21.15.98:0/1809155536 conn(0x7fe25406d680 msgr2=0x7fe25406da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 -- 172.21.15.98:0/1809155536 shutdown_connections 2024-05-27T00:22:20.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 -- 172.21.15.98:0/1809155536 wait complete. 2024-05-27T00:22:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 Processor -- start 2024-05-27T00:22:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 -- start start 2024-05-27T00:22:20.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2380039c0 0x7fe25414e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe254155360 0x7fe25414eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2541f37c0 0x7fe2541f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe25406d220 con 0x7fe2541f37c0 2024-05-27T00:22:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe25406ef50 con 0x7fe254155360 2024-05-27T00:22:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe25ae00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe25406b8b0 con 0x7fe2380039c0 2024-05-27T00:22:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe258e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2541f37c0 0x7fe2541f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe253e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2380039c0 0x7fe25414e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe258e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2541f37c0 0x7fe2541f3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59200/0 (socket says 172.21.15.98:59200) 2024-05-27T00:22:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe258e00640 1 -- 172.21.15.98:0/2365954728 learned_addr learned my addr 172.21.15.98:0/2365954728 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:20.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe253400640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe254155360 0x7fe25414eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe258e00640 1 -- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2380039c0 msgr2=0x7fe25414e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe258e00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2380039c0 0x7fe25414e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe258e00640 1 -- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe254155360 msgr2=0x7fe25414eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe258e00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe254155360 0x7fe25414eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe258e00640 1 -- 172.21.15.98:0/2365954728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe254150ce0 con 0x7fe2541f37c0 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.658+0000 7fe258e00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2541f37c0 0x7fe2541f3c40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe254070740 tx=0x7fe24c011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.662+0000 7fe24be00640 1 -- 172.21.15.98:0/2365954728 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c004230 con 0x7fe2541f37c0 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.662+0000 7fe24be00640 1 -- 172.21.15.98:0/2365954728 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe24c01c070 con 0x7fe2541f37c0 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.662+0000 7fe24be00640 1 -- 172.21.15.98:0/2365954728 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c00f6b0 con 0x7fe2541f37c0 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.662+0000 7fe25ae00640 1 -- 172.21.15.98:0/2365954728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2540c7c50 con 0x7fe2541f37c0 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.662+0000 7fe25ae00640 1 -- 172.21.15.98:0/2365954728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2540c7f10 con 0x7fe2541f37c0 2024-05-27T00:22:20.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.662+0000 7fe25ae00640 1 -- 172.21.15.98:0/2365954728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe25406f540 con 0x7fe2541f37c0 2024-05-27T00:22:20.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.662+0000 7fe24be00640 1 -- 172.21.15.98:0/2365954728 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe24c00f850 con 0x7fe2541f37c0 2024-05-27T00:22:20.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.662+0000 7fe24be00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe2340678c0 0x7fe234069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:20.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.662+0000 7fe24be00640 1 -- 172.21.15.98:0/2365954728 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7fe24c086000 con 0x7fe2541f37c0 2024-05-27T00:22:20.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.662+0000 7fe253e00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe2340678c0 0x7fe234069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:20.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.666+0000 7fe253e00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe2340678c0 0x7fe234069d80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe2440b0000 tx=0x7fe244009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:20.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.666+0000 7fe24be00640 1 -- 172.21.15.98:0/2365954728 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fe24c0562d0 con 0x7fe2541f37c0 2024-05-27T00:22:20.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 -- 172.21.15.98:0/2365954728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe254075c90 con 0x7fe2541f37c0 2024-05-27T00:22:20.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe24be00640 1 -- 172.21.15.98:0/2365954728 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6006 (secure 0 0 0) 0x7fe24c05a5d0 con 0x7fe2541f37c0 2024-05-27T00:22:20.859 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:20.859 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:20.214130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:20.859 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-05-27T00:22:20.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 -- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe2340678c0 msgr2=0x7fe234069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe2340678c0 0x7fe234069d80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe2440b0000 tx=0x7fe244009040 comp rx=0 tx=0).stop 2024-05-27T00:22:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 -- 172.21.15.98:0/2365954728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2541f37c0 msgr2=0x7fe2541f3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2541f37c0 0x7fe2541f3c40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe254070740 tx=0x7fe24c011c90 comp rx=0 tx=0).stop 2024-05-27T00:22:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 -- 172.21.15.98:0/2365954728 shutdown_connections 2024-05-27T00:22:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2541f37c0 0x7fe2541f3c40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe2340678c0 0x7fe234069d80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe254155360 0x7fe25414eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 --2- 172.21.15.98:0/2365954728 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2380039c0 0x7fe25414e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 -- 172.21.15.98:0/2365954728 >> 172.21.15.98:0/2365954728 conn(0x7fe25406d680 msgr2=0x7fe2540b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:20.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 -- 172.21.15.98:0/2365954728 shutdown_connections 2024-05-27T00:22:20.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:20.858+0000 7fe25ae00640 1 -- 172.21.15.98:0/2365954728 wait complete. 2024-05-27T00:22:20.880 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 30, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5343": { "addr": "172.21.15.98:6834/4290077910", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 4290077910, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 4290077910, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5343, "incarnation": 30, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:22:20.214130+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5343 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:22:22.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.026+0000 7fb3a7200640 1 Processor -- start 2024-05-27T00:22:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.026+0000 7fb3a7200640 1 -- start start 2024-05-27T00:22:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a7200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb3a8150c80 0x7fb3a815a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3a806eed0 0x7fb3a806b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a7200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a80c8230 con 0x7fb3a8157710 2024-05-27T00:22:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a7200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb3a80c8000 con 0x7fb3a8150c80 2024-05-27T00:22:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a80c84d0 con 0x7fb3a806eed0 2024-05-27T00:22:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a5400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb3a8157710 0x7fb3a80c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48700/0 (socket says 172.21.15.98:48700) 2024-05-27T00:22:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a5400640 1 -- 172.21.15.98:0/1701150882 learned_addr learned my addr 172.21.15.98:0/1701150882 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a5e00640 1 --2- 172.21.15.98:0/1701150882 >> v2:172.21.15.204:3300/0 conn(0x7fb3a8150c80 0x7fb3a815a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a6800640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3a806eed0 0x7fb3a806b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3374204879 0 0) 0x7fb3a80c8230 con 0x7fb3a8157710 2024-05-27T00:22:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb38c003530 con 0x7fb3a8157710 2024-05-27T00:22:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1032165879 0 0) 0x7fb38c003530 con 0x7fb3a8157710 2024-05-27T00:22:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3a806eed0 msgr2=0x7fb3a806b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3a806eed0 0x7fb3a806b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 >> v2:172.21.15.204:3300/0 conn(0x7fb3a8150c80 msgr2=0x7fb3a815a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 --2- 172.21.15.98:0/1701150882 >> v2:172.21.15.204:3300/0 conn(0x7fb3a8150c80 0x7fb3a815a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a8156c30 con 0x7fb3a8157710 2024-05-27T00:22:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fb3940016a0 con 0x7fb3a8157710 2024-05-27T00:22:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 >> v1:172.21.15.98:6789/0 conn(0x7fb3a8157710 legacy=0x7fb3a80c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb38c0039c0 0x7fb38c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3a806eed0 0x7fb38c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb38c004d50 0x7fb38c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a80c84d0 con 0x7fb38c004d50 2024-05-27T00:22:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb38c00b9d0 con 0x7fb38c0039c0 2024-05-27T00:22:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb38c00bb80 con 0x7fb3a806eed0 2024-05-27T00:22:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a6800640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb38c004d50 0x7fb38c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:23.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a5e00640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb38c0039c0 0x7fb38c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:23.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a5400640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3a806eed0 0x7fb38c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:23.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a6800640 1 -- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3a806eed0 msgr2=0x7fb38c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:23.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a6800640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3a806eed0 0x7fb38c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:23.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a6800640 1 -- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb38c0039c0 msgr2=0x7fb38c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:23.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a6800640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb38c0039c0 0x7fb38c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:23.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a6800640 1 -- 172.21.15.98:0/1701150882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb38c003530 con 0x7fb38c004d50 2024-05-27T00:22:23.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.030+0000 7fb3a5400640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3a806eed0 0x7fb38c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a6800640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb38c004d50 0x7fb38c0051d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009ae0 tx=0x7fb3a001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81dd83816455570a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00b2070 con 0x7fb38c004d50 2024-05-27T00:22:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a00a5660 con 0x7fb38c004d50 2024-05-27T00:22:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a4a00640 1 -- 172.21.15.98:0/1701150882 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00096b0 con 0x7fb38c004d50 2024-05-27T00:22:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a7200640 1 -- 172.21.15.98:0/1701150882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb38c004d50 msgr2=0x7fb38c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a7200640 1 --2- 172.21.15.98:0/1701150882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb38c004d50 0x7fb38c0051d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009ae0 tx=0x7fb3a001ca70 comp rx=0 tx=0).stop 2024-05-27T00:22:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a6800640 1 -- 172.21.15.98:0/1701150882 reap_dead start 2024-05-27T00:22:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a7200640 1 -- 172.21.15.98:0/1701150882 shutdown_connections 2024-05-27T00:22:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a7200640 1 -- 172.21.15.98:0/1701150882 >> 172.21.15.98:0/1701150882 conn(0x7fb3a80cbe00 msgr2=0x7fb3a80cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a7200640 1 -- 172.21.15.98:0/1701150882 shutdown_connections 2024-05-27T00:22:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.034+0000 7fb3a7200640 1 -- 172.21.15.98:0/1701150882 wait complete. 2024-05-27T00:22:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 Processor -- start 2024-05-27T00:22:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 -- start start 2024-05-27T00:22:23.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb38c004d50 0x7fb3a81f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:23.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3a8072eb0 0x7fb3a813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:23.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81424c0 0x7fb3a8140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:23.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a8151060 con 0x7fb3a81424c0 2024-05-27T00:22:23.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a81571f0 con 0x7fb3a8072eb0 2024-05-27T00:22:23.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a81511e0 con 0x7fb38c004d50 2024-05-27T00:22:23.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81424c0 0x7fb3a8140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:23.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81424c0 0x7fb3a8140ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59204/0 (socket says 172.21.15.98:59204) 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a6800640 1 -- 172.21.15.98:0/3832752169 learned_addr learned my addr 172.21.15.98:0/3832752169 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a5400640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3a8072eb0 0x7fb3a813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a5e00640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb38c004d50 0x7fb3a81f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a6800640 1 -- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb38c004d50 msgr2=0x7fb3a81f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a6800640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb38c004d50 0x7fb3a81f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a6800640 1 -- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3a8072eb0 msgr2=0x7fb3a813e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a6800640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3a8072eb0 0x7fb3a813e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a6800640 1 -- 172.21.15.98:0/3832752169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a8073960 con 0x7fb3a81424c0 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a5400640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3a8072eb0 0x7fb3a813e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a5e00640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb38c004d50 0x7fb3a81f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a6800640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81424c0 0x7fb3a8140ff0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb3a001c8a0 tx=0x7fb38c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb39e000640 1 -- 172.21.15.98:0/3832752169 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00bb070 con 0x7fb3a81424c0 2024-05-27T00:22:23.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb39e000640 1 -- 172.21.15.98:0/3832752169 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a0000e50 con 0x7fb3a81424c0 2024-05-27T00:22:23.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 -- 172.21.15.98:0/3832752169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a80733a0 con 0x7fb3a81424c0 2024-05-27T00:22:23.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb39e000640 1 -- 172.21.15.98:0/3832752169 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00b2040 con 0x7fb3a81424c0 2024-05-27T00:22:23.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 -- 172.21.15.98:0/3832752169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a81f35b0 con 0x7fb3a81424c0 2024-05-27T00:22:23.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.038+0000 7fb3a7200640 1 -- 172.21.15.98:0/3832752169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a8075c40 con 0x7fb3a81424c0 2024-05-27T00:22:23.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.042+0000 7fb39e000640 1 -- 172.21.15.98:0/3832752169 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb3a00be070 con 0x7fb3a81424c0 2024-05-27T00:22:23.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.042+0000 7fb39e000640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3880678c0 0x7fb388069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:23.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.042+0000 7fb39e000640 1 -- 172.21.15.98:0/3832752169 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7fb3a011f470 con 0x7fb3a81424c0 2024-05-27T00:22:23.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.042+0000 7fb3a5e00640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3880678c0 0x7fb388069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:23.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.042+0000 7fb3a5e00640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3880678c0 0x7fb388069d80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80b0f40 tx=0x7fb39800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:23.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.046+0000 7fb39e000640 1 -- 172.21.15.98:0/3832752169 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb3a0016780 con 0x7fb3a81424c0 2024-05-27T00:22:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.238+0000 7fb3a7200640 1 -- 172.21.15.98:0/3832752169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3a80c7bd0 con 0x7fb3a81424c0 2024-05-27T00:22:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.238+0000 7fb39e000640 1 -- 172.21.15.98:0/3832752169 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6006 (secure 0 0 0) 0x7fb3a00ef610 con 0x7fb3a81424c0 2024-05-27T00:22:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:23.241 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:20.214130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:23.242 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-05-27T00:22:23.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 -- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3880678c0 msgr2=0x7fb388069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:23.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3880678c0 0x7fb388069d80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80b0f40 tx=0x7fb39800b040 comp rx=0 tx=0).stop 2024-05-27T00:22:23.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 -- 172.21.15.98:0/3832752169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81424c0 msgr2=0x7fb3a8140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:23.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81424c0 0x7fb3a8140ff0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb3a001c8a0 tx=0x7fb38c005840 comp rx=0 tx=0).stop 2024-05-27T00:22:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 -- 172.21.15.98:0/3832752169 shutdown_connections 2024-05-27T00:22:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3a81424c0 0x7fb3a8140ff0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3880678c0 0x7fb388069d80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3a8072eb0 0x7fb3a813e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 --2- 172.21.15.98:0/3832752169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb38c004d50 0x7fb3a81f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 -- 172.21.15.98:0/3832752169 >> 172.21.15.98:0/3832752169 conn(0x7fb3a80cbe00 msgr2=0x7fb3a806e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 -- 172.21.15.98:0/3832752169 shutdown_connections 2024-05-27T00:22:23.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:23.242+0000 7fb3a7200640 1 -- 172.21.15.98:0/3832752169 wait complete. 2024-05-27T00:22:23.263 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:22:25.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:25.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f837f000640 1 Processor -- start 2024-05-27T00:22:25.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f837f000640 1 -- start start 2024-05-27T00:22:25.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f837f000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f83780c9ec0 0x7f837806f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:25.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f837f000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f837806b470 con 0x7f83780c4da0 2024-05-27T00:22:25.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f837f000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f837806d380 con 0x7f83780c9850 2024-05-27T00:22:25.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f837f000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f83780c8c20 con 0x7f83780c9ec0 2024-05-27T00:22:25.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f836f400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f83780c4da0 0x7f83780c9140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37494/0 (socket says 172.21.15.98:37494) 2024-05-27T00:22:25.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f836f400640 1 -- 172.21.15.98:0/360710827 learned_addr learned my addr 172.21.15.98:0/360710827 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:25.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f8377e00640 1 --2- 172.21.15.98:0/360710827 >> v2:172.21.15.204:3300/0 conn(0x7f83780c9ec0 0x7f837806f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:25.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f8377400640 1 -- 172.21.15.98:0/360710827 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 954791102 0 0) 0x7f837806b470 con 0x7f83780c4da0 2024-05-27T00:22:25.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f8377400640 1 -- 172.21.15.98:0/360710827 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f835c003530 con 0x7f83780c4da0 2024-05-27T00:22:25.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f8377e00640 1 -- 172.21.15.98:0/360710827 >> v1:172.21.15.204:6789/0 conn(0x7f83780c9850 legacy=0x7f8378169470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:25.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f8377e00640 1 -- 172.21.15.98:0/360710827 >> v1:172.21.15.98:6789/0 conn(0x7f83780c4da0 legacy=0x7f83780c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:25.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.414+0000 7f8377e00640 1 -- 172.21.15.98:0/360710827 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f837806b730 con 0x7f83780c9ec0 2024-05-27T00:22:25.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f8377e00640 1 --2- 172.21.15.98:0/360710827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83780c9ec0 0x7f837806f760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f83680097c0 tx=0x7f836801ead0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a64a31f009156de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:25.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f8377400640 1 -- 172.21.15.98:0/360710827 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83680b5070 con 0x7f83780c9ec0 2024-05-27T00:22:25.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f8377400640 1 -- 172.21.15.98:0/360710827 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83680a8590 con 0x7f83780c9ec0 2024-05-27T00:22:25.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f8377400640 1 -- 172.21.15.98:0/360710827 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360001680 con 0x7f83780c9ec0 2024-05-27T00:22:25.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f837f000640 1 -- 172.21.15.98:0/360710827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83780c9ec0 msgr2=0x7f837806f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:25.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f837f000640 1 --2- 172.21.15.98:0/360710827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83780c9ec0 0x7f837806f760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f83680097c0 tx=0x7f836801ead0 comp rx=0 tx=0).stop 2024-05-27T00:22:25.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f837f000640 1 -- 172.21.15.98:0/360710827 shutdown_connections 2024-05-27T00:22:25.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f837f000640 1 --2- 172.21.15.98:0/360710827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83780c9ec0 0x7f837806f760 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:25.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f837f000640 1 -- 172.21.15.98:0/360710827 >> 172.21.15.98:0/360710827 conn(0x7f837806d680 msgr2=0x7f837806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:25.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f837f000640 1 -- 172.21.15.98:0/360710827 shutdown_connections 2024-05-27T00:22:25.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.418+0000 7f837f000640 1 -- 172.21.15.98:0/360710827 wait complete. 2024-05-27T00:22:25.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837f000640 1 Processor -- start 2024-05-27T00:22:25.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837f000640 1 -- start start 2024-05-27T00:22:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837f000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83780c4da0 0x7f83780c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837f000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83780c9850 0x7f83781e9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837f000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83780c9ec0 0x7f83781ea1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837f000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83780c51c0 con 0x7f83780c9ec0 2024-05-27T00:22:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837f000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f837806eb30 con 0x7f83780c9850 2024-05-27T00:22:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837f000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f83781591b0 con 0x7f83780c4da0 2024-05-27T00:22:25.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83780c9ec0 0x7f83781ea1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:25.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83780c9ec0 0x7f83781ea1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52518/0 (socket says 172.21.15.98:52518) 2024-05-27T00:22:25.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f836f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83780c9850 0x7f83781e9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:25.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f837d000640 1 -- 172.21.15.98:0/2455349593 learned_addr learned my addr 172.21.15.98:0/2455349593 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:25.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.422+0000 7f8377e00640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83780c4da0 0x7f83780c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f837d000640 1 -- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83780c4da0 msgr2=0x7f83780c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f837d000640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83780c4da0 0x7f83780c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f837d000640 1 -- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83780c9850 msgr2=0x7f83781e9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f837d000640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83780c9850 0x7f83781e9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f837d000640 1 -- 172.21.15.98:0/2455349593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f837814d480 con 0x7f83780c9ec0 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f837d000640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83780c9ec0 0x7f83781ea1f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f837000c770 tx=0x7f837000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f8374c00640 1 -- 172.21.15.98:0/2455349593 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370003b80 con 0x7f83780c9ec0 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f8374c00640 1 -- 172.21.15.98:0/2455349593 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8370003d20 con 0x7f83780c9ec0 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f837f000640 1 -- 172.21.15.98:0/2455349593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f837814d1e0 con 0x7f83780c9ec0 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f8374c00640 1 -- 172.21.15.98:0/2455349593 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370005310 con 0x7f83780c9ec0 2024-05-27T00:22:25.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f837f000640 1 -- 172.21.15.98:0/2455349593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f837815f9a0 con 0x7f83780c9ec0 2024-05-27T00:22:25.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f837f000640 1 -- 172.21.15.98:0/2455349593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8378075cb0 con 0x7f83780c9ec0 2024-05-27T00:22:25.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.426+0000 7f8374c00640 1 -- 172.21.15.98:0/2455349593 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f83700054b0 con 0x7f83780c9ec0 2024-05-27T00:22:25.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.430+0000 7f8374c00640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f83540678f0 0x7f8354069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.430+0000 7f8374c00640 1 -- 172.21.15.98:0/2455349593 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f8370085c50 con 0x7f83780c9ec0 2024-05-27T00:22:25.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.430+0000 7f8377e00640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f83540678f0 0x7f8354069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:25.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.430+0000 7f8377e00640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f83540678f0 0x7f8354069db0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f83680b2000 tx=0x7f83680aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:25.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.434+0000 7f8374c00640 1 -- 172.21.15.98:0/2455349593 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8370056560 con 0x7f83780c9ec0 2024-05-27T00:22:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.634+0000 7f837f000640 1 -- 172.21.15.98:0/2455349593 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83780c4970 con 0x7f83780c9ec0 2024-05-27T00:22:25.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f8374c00640 1 -- 172.21.15.98:0/2455349593 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6006 (secure 0 0 0) 0x7f8370055e60 con 0x7f83780c9ec0 2024-05-27T00:22:25.638 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:25.639 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:20.214130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:25.639 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-05-27T00:22:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 -- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f83540678f0 msgr2=0x7f8354069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f83540678f0 0x7f8354069db0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f83680b2000 tx=0x7f83680aa070 comp rx=0 tx=0).stop 2024-05-27T00:22:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 -- 172.21.15.98:0/2455349593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83780c9ec0 msgr2=0x7f83781ea1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83780c9ec0 0x7f83781ea1f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f837000c770 tx=0x7f837000cb70 comp rx=0 tx=0).stop 2024-05-27T00:22:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 -- 172.21.15.98:0/2455349593 shutdown_connections 2024-05-27T00:22:25.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83780c9ec0 0x7f83781ea1f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f83540678f0 0x7f8354069db0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83780c9850 0x7f83781e9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 --2- 172.21.15.98:0/2455349593 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83780c4da0 0x7f83780c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 -- 172.21.15.98:0/2455349593 >> 172.21.15.98:0/2455349593 conn(0x7f837806d680 msgr2=0x7f83780ca2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 -- 172.21.15.98:0/2455349593 shutdown_connections 2024-05-27T00:22:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:25.638+0000 7f837f000640 1 -- 172.21.15.98:0/2455349593 wait complete. 2024-05-27T00:22:25.667 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:22:27.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdad0800640 1 Processor -- start 2024-05-27T00:22:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdad0800640 1 -- start start 2024-05-27T00:22:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdad0800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdac8154830 0x7fdac815b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdad0800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdac80cc010 0x7fdac8071e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdad0800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac80c1a90 con 0x7fdac80cbbb0 2024-05-27T00:22:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdad0800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdac80c2b30 con 0x7fdac8154830 2024-05-27T00:22:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdad0800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdac80c26a0 con 0x7fdac80cc010 2024-05-27T00:22:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdacde00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdac8154830 0x7fdac815b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdacde00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdac8154830 0x7fdac815b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52524/0 (socket says 172.21.15.98:52524) 2024-05-27T00:22:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdacde00640 1 -- 172.21.15.98:0/287225256 learned_addr learned my addr 172.21.15.98:0/287225256 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdace800640 1 --2- 172.21.15.98:0/287225256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdac80cc010 0x7fdac8071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdacde00640 1 -- 172.21.15.98:0/287225256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdac80cc010 msgr2=0x7fdac8071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdacde00640 1 --2- 172.21.15.98:0/287225256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdac80cc010 0x7fdac8071e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdacde00640 1 -- 172.21.15.98:0/287225256 >> v1:172.21.15.204:6789/0 conn(0x7fdac80cbbb0 legacy=0x7fdac80c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.809+0000 7fdacde00640 1 -- 172.21.15.98:0/287225256 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac80c1d70 con 0x7fdac8154830 2024-05-27T00:22:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdacde00640 1 --2- 172.21.15.98:0/287225256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac8154830 0x7fdac815b470 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdabc009870 tx=0x7fdabc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ee8a85b5da245c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdacca00640 1 -- 172.21.15.98:0/287225256 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc0b2070 con 0x7fdac8154830 2024-05-27T00:22:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdacca00640 1 -- 172.21.15.98:0/287225256 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdabc0a9070 con 0x7fdac8154830 2024-05-27T00:22:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdacca00640 1 -- 172.21.15.98:0/287225256 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc0ad880 con 0x7fdac8154830 2024-05-27T00:22:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdad0800640 1 -- 172.21.15.98:0/287225256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac8154830 msgr2=0x7fdac815b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdad0800640 1 --2- 172.21.15.98:0/287225256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac8154830 0x7fdac815b470 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdabc009870 tx=0x7fdabc01c560 comp rx=0 tx=0).stop 2024-05-27T00:22:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdad0800640 1 -- 172.21.15.98:0/287225256 shutdown_connections 2024-05-27T00:22:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdad0800640 1 --2- 172.21.15.98:0/287225256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdac80cc010 0x7fdac8071e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdad0800640 1 --2- 172.21.15.98:0/287225256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac8154830 0x7fdac815b470 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdad0800640 1 -- 172.21.15.98:0/287225256 >> 172.21.15.98:0/287225256 conn(0x7fdac8067530 msgr2=0x7fdac806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdad0800640 1 -- 172.21.15.98:0/287225256 shutdown_connections 2024-05-27T00:22:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.813+0000 7fdad0800640 1 -- 172.21.15.98:0/287225256 wait complete. 2024-05-27T00:22:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdad0800640 1 Processor -- start 2024-05-27T00:22:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdad0800640 1 -- start start 2024-05-27T00:22:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdad0800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac80cbbb0 0x7fdac80b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdad0800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdac80cc010 0x7fdac815e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdad0800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdac8154830 0x7fdac81f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdad0800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdac80c6fd0 con 0x7fdac80cbbb0 2024-05-27T00:22:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdad0800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdac80cc420 con 0x7fdac80cc010 2024-05-27T00:22:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdad0800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdac80c7150 con 0x7fdac8154830 2024-05-27T00:22:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdacde00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac80cbbb0 0x7fdac80b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdacde00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac80cbbb0 0x7fdac80b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52540/0 (socket says 172.21.15.98:52540) 2024-05-27T00:22:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdacde00640 1 -- 172.21.15.98:0/696652921 learned_addr learned my addr 172.21.15.98:0/696652921 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdacd400640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdac80cc010 0x7fdac815e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdace800640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdac8154830 0x7fdac81f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdacde00640 1 -- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdac8154830 msgr2=0x7fdac81f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdacde00640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdac8154830 0x7fdac81f3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdacde00640 1 -- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdac80cc010 msgr2=0x7fdac815e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdacde00640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdac80cc010 0x7fdac815e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdacde00640 1 -- 172.21.15.98:0/696652921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac815edc0 con 0x7fdac80cbbb0 2024-05-27T00:22:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdacde00640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac80cbbb0 0x7fdac80b0080 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0b0040 tx=0x7fdabc01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdac2000640 1 -- 172.21.15.98:0/696652921 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc0bc020 con 0x7fdac80cbbb0 2024-05-27T00:22:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdac2000640 1 -- 172.21.15.98:0/696652921 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdabc0b8420 con 0x7fdac80cbbb0 2024-05-27T00:22:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdad0800640 1 -- 172.21.15.98:0/696652921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac8156850 con 0x7fdac80cbbb0 2024-05-27T00:22:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.817+0000 7fdac2000640 1 -- 172.21.15.98:0/696652921 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc0b2040 con 0x7fdac80cbbb0 2024-05-27T00:22:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.821+0000 7fdad0800640 1 -- 172.21.15.98:0/696652921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac8156b10 con 0x7fdac80cbbb0 2024-05-27T00:22:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.821+0000 7fdad0800640 1 -- 172.21.15.98:0/696652921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac8075cb0 con 0x7fdac80cbbb0 2024-05-27T00:22:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.821+0000 7fdac2000640 1 -- 172.21.15.98:0/696652921 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdabc0bf020 con 0x7fdac80cbbb0 2024-05-27T00:22:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.821+0000 7fdac2000640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdaac0678c0 0x7fdaac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.821+0000 7fdac2000640 1 -- 172.21.15.98:0/696652921 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7fdabc1201e0 con 0x7fdac80cbbb0 2024-05-27T00:22:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.821+0000 7fdacd400640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdaac0678c0 0x7fdaac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.825+0000 7fdacd400640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdaac0678c0 0x7fdaac069d80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdac80b1fa0 tx=0x7fdab8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:27.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:27.825+0000 7fdac2000640 1 -- 172.21.15.98:0/696652921 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fdabc0c6030 con 0x7fdac80cbbb0 2024-05-27T00:22:28.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 -- 172.21.15.98:0/696652921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdac8159ae0 con 0x7fdac80cbbb0 2024-05-27T00:22:28.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdac2000640 1 -- 172.21.15.98:0/696652921 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6006 (secure 0 0 0) 0x7fdabc016150 con 0x7fdac80cbbb0 2024-05-27T00:22:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:28.036 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:20.214130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:28.037 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-05-27T00:22:28.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 -- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdaac0678c0 msgr2=0x7fdaac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:28.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdaac0678c0 0x7fdaac069d80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdac80b1fa0 tx=0x7fdab8009040 comp rx=0 tx=0).stop 2024-05-27T00:22:28.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 -- 172.21.15.98:0/696652921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac80cbbb0 msgr2=0x7fdac80b0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:28.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac80cbbb0 0x7fdac80b0080 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0b0040 tx=0x7fdabc01ada0 comp rx=0 tx=0).stop 2024-05-27T00:22:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 -- 172.21.15.98:0/696652921 shutdown_connections 2024-05-27T00:22:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdac8154830 0x7fdac81f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdaac0678c0 0x7fdaac069d80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdac80cc010 0x7fdac815e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 --2- 172.21.15.98:0/696652921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdac80cbbb0 0x7fdac80b0080 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 -- 172.21.15.98:0/696652921 >> 172.21.15.98:0/696652921 conn(0x7fdac8067530 msgr2=0x7fdac81529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 -- 172.21.15.98:0/696652921 shutdown_connections 2024-05-27T00:22:28.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:28.033+0000 7fdad0800640 1 -- 172.21.15.98:0/696652921 wait complete. 2024-05-27T00:22:28.056 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:22:30.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f101a000640 1 Processor -- start 2024-05-27T00:22:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f101a000640 1 -- start start 2024-05-27T00:22:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f101a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 0x7f10140c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f101a000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f10140c6e00 0x7f1014071e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f101a000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10140c1a90 con 0x7f1014154830 2024-05-27T00:22:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f101a000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f101415b8d0 con 0x7f10140c6e00 2024-05-27T00:22:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f101a000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f10140c2b30 con 0x7f101415b470 2024-05-27T00:22:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1013400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1014154830 0x7f10140cbbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37508/0 (socket says 172.21.15.98:37508) 2024-05-27T00:22:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1013400640 1 -- 172.21.15.98:0/1292059703 learned_addr learned my addr 172.21.15.98:0/1292059703 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1012a00640 1 --2- 172.21.15.98:0/1292059703 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 0x7f10140c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1013e00640 1 --2- 172.21.15.98:0/1292059703 >> v2:172.21.15.204:3300/0 conn(0x7f10140c6e00 0x7f1014071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1012000640 1 -- 172.21.15.98:0/1292059703 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2618441644 0 0) 0x7f10140c1a90 con 0x7f1014154830 2024-05-27T00:22:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1012000640 1 -- 172.21.15.98:0/1292059703 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff8003530 con 0x7f1014154830 2024-05-27T00:22:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1012000640 1 -- 172.21.15.98:0/1292059703 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 350125270 0 0) 0x7f0ff8003530 con 0x7f1014154830 2024-05-27T00:22:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1012a00640 1 -- 172.21.15.98:0/1292059703 >> v2:172.21.15.204:3300/0 conn(0x7f10140c6e00 msgr2=0x7f1014071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1012a00640 1 --2- 172.21.15.98:0/1292059703 >> v2:172.21.15.204:3300/0 conn(0x7f10140c6e00 0x7f1014071e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1012a00640 1 -- 172.21.15.98:0/1292059703 >> v1:172.21.15.98:6789/0 conn(0x7f1014154830 legacy=0x7f10140cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.201+0000 7f1012a00640 1 -- 172.21.15.98:0/1292059703 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10140c1d70 con 0x7f101415b470 2024-05-27T00:22:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f1012a00640 1 --2- 172.21.15.98:0/1292059703 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 0x7f10140c68c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f10000097c0 tx=0x7f100001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5612e6740d65586b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f1012000640 1 -- 172.21.15.98:0/1292059703 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10000b0070 con 0x7f101415b470 2024-05-27T00:22:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f1012000640 1 -- 172.21.15.98:0/1292059703 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10000ab400 con 0x7f101415b470 2024-05-27T00:22:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f1012000640 1 -- 172.21.15.98:0/1292059703 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1004001680 con 0x7f101415b470 2024-05-27T00:22:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f101a000640 1 -- 172.21.15.98:0/1292059703 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 msgr2=0x7f10140c68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f101a000640 1 --2- 172.21.15.98:0/1292059703 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 0x7f10140c68c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f10000097c0 tx=0x7f100001c3c0 comp rx=0 tx=0).stop 2024-05-27T00:22:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f101a000640 1 -- 172.21.15.98:0/1292059703 shutdown_connections 2024-05-27T00:22:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f101a000640 1 --2- 172.21.15.98:0/1292059703 >> v2:172.21.15.204:3300/0 conn(0x7f10140c6e00 0x7f1014071e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f101a000640 1 --2- 172.21.15.98:0/1292059703 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 0x7f10140c68c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f101a000640 1 -- 172.21.15.98:0/1292059703 >> 172.21.15.98:0/1292059703 conn(0x7f1014067530 msgr2=0x7f101406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f101a000640 1 -- 172.21.15.98:0/1292059703 shutdown_connections 2024-05-27T00:22:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.205+0000 7f101a000640 1 -- 172.21.15.98:0/1292059703 wait complete. 2024-05-27T00:22:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f101a000640 1 Processor -- start 2024-05-27T00:22:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f101a000640 1 -- start start 2024-05-27T00:22:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f101a000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10140c6e00 0x7f101415d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f101a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1014154830 0x7f10141575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f101a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 0x7f101415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f101a000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1014154c10 con 0x7f10140c6e00 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f101a000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f10140c26a0 con 0x7f1014154830 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f101a000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1014154d90 con 0x7f101415b470 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f1013400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10140c6e00 0x7f101415d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f1013400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10140c6e00 0x7f101415d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52552/0 (socket says 172.21.15.98:52552) 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f1013400640 1 -- 172.21.15.98:0/2160649055 learned_addr learned my addr 172.21.15.98:0/2160649055 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f1013e00640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 0x7f101415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f1012a00640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1014154830 0x7f10141575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f1013400640 1 -- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 msgr2=0x7f101415ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.209+0000 7f1013400640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 0x7f101415ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f1013400640 1 -- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1014154830 msgr2=0x7f10141575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f1013400640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1014154830 0x7f10141575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f1013400640 1 -- 172.21.15.98:0/2160649055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10140c5c30 con 0x7f10140c6e00 2024-05-27T00:22:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f1012a00640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1014154830 0x7f10141575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f1013e00640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 0x7f101415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f1013400640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10140c6e00 0x7f101415d590 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f100400c780 tx=0x7f100400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f100b400640 1 -- 172.21.15.98:0/2160649055 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1004005ae0 con 0x7f10140c6e00 2024-05-27T00:22:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f100b400640 1 -- 172.21.15.98:0/2160649055 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1004005ca0 con 0x7f10140c6e00 2024-05-27T00:22:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f100b400640 1 -- 172.21.15.98:0/2160649055 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10040072b0 con 0x7f10140c6e00 2024-05-27T00:22:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f101a000640 1 -- 172.21.15.98:0/2160649055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101415dc40 con 0x7f10140c6e00 2024-05-27T00:22:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f101a000640 1 -- 172.21.15.98:0/2160649055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10141f3c00 con 0x7f10140c6e00 2024-05-27T00:22:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f101a000640 1 -- 172.21.15.98:0/2160649055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1014075cb0 con 0x7f10140c6e00 2024-05-27T00:22:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.213+0000 7f100b400640 1 -- 172.21.15.98:0/2160649055 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1004007450 con 0x7f10140c6e00 2024-05-27T00:22:30.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.217+0000 7f100b400640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0ff40678f0 0x7f0ff4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:30.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.217+0000 7f100b400640 1 -- 172.21.15.98:0/2160649055 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f1004085e30 con 0x7f10140c6e00 2024-05-27T00:22:30.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.217+0000 7f1012a00640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0ff40678f0 0x7f0ff4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:30.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.217+0000 7f1012a00640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0ff40678f0 0x7f0ff4069db0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f10000ad000 tx=0x7f10000a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.221+0000 7f100b400640 1 -- 172.21.15.98:0/2160649055 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f10140c5c30 con 0x7f10140c6e00 2024-05-27T00:22:30.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.417+0000 7f101a000640 1 -- 172.21.15.98:0/2160649055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10140c7420 con 0x7f10140c6e00 2024-05-27T00:22:30.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.417+0000 7f100b400640 1 -- 172.21.15.98:0/2160649055 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6006 (secure 0 0 0) 0x7f10040566c0 con 0x7f10140c6e00 2024-05-27T00:22:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:30.420 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:20.214130+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:30.420 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 30 2024-05-27T00:22:30.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 -- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0ff40678f0 msgr2=0x7f0ff4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:30.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0ff40678f0 0x7f0ff4069db0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f10000ad000 tx=0x7f10000a6000 comp rx=0 tx=0).stop 2024-05-27T00:22:30.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 -- 172.21.15.98:0/2160649055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10140c6e00 msgr2=0x7f101415d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:30.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10140c6e00 0x7f101415d590 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f100400c780 tx=0x7f100400cc40 comp rx=0 tx=0).stop 2024-05-27T00:22:30.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 -- 172.21.15.98:0/2160649055 shutdown_connections 2024-05-27T00:22:30.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f101415b470 0x7f101415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:30.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0ff40678f0 0x7f0ff4069db0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:30.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1014154830 0x7f10141575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:30.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 --2- 172.21.15.98:0/2160649055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10140c6e00 0x7f101415d590 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:30.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 -- 172.21.15.98:0/2160649055 >> 172.21.15.98:0/2160649055 conn(0x7f1014067530 msgr2=0x7f10140c8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:30.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 -- 172.21.15.98:0/2160649055 shutdown_connections 2024-05-27T00:22:30.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:30.421+0000 7f101a000640 1 -- 172.21.15.98:0/2160649055 wait complete. 2024-05-27T00:22:30.441 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:22:32.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:32.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2806800640 1 Processor -- start 2024-05-27T00:22:32.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2806800640 1 -- start start 2024-05-27T00:22:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2806800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c7e20 0x7f280006f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2806800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f28001511b0 0x7f2800151600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2806800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28000c1150 con 0x7f28000c8200 2024-05-27T00:22:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2806800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f2800151b40 con 0x7f28001511b0 2024-05-27T00:22:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2806800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f280006edd0 con 0x7f28000c7e20 2024-05-27T00:22:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2805e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f28001511b0 0x7f2800151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2805400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c7e20 0x7f280006f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2805e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f28001511b0 0x7f2800151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:32968/0 (socket says 172.21.15.98:32968) 2024-05-27T00:22:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2805e00640 1 -- 172.21.15.98:0/3824819881 learned_addr learned my addr 172.21.15.98:0/3824819881 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2805400640 1 -- 172.21.15.98:0/3824819881 >> v2:172.21.15.204:3300/0 conn(0x7f28001511b0 msgr2=0x7f2800151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2805400640 1 --2- 172.21.15.98:0/3824819881 >> v2:172.21.15.204:3300/0 conn(0x7f28001511b0 0x7f2800151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2805400640 1 -- 172.21.15.98:0/3824819881 >> v1:172.21.15.204:6789/0 conn(0x7f28000c8200 legacy=0x7f280006e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.589+0000 7f2805400640 1 -- 172.21.15.98:0/3824819881 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2800072360 con 0x7f28000c7e20 2024-05-27T00:22:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f2805400640 1 --2- 172.21.15.98:0/3824819881 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c7e20 0x7f280006f260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f27f40097c0 tx=0x7f27f401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23c2e58091c7c0a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f27ffe00640 1 -- 172.21.15.98:0/3824819881 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f40b1070 con 0x7f28000c7e20 2024-05-27T00:22:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f27ffe00640 1 -- 172.21.15.98:0/3824819881 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f40ac410 con 0x7f28000c7e20 2024-05-27T00:22:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f27ffe00640 1 -- 172.21.15.98:0/3824819881 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f40ac7e0 con 0x7f28000c7e20 2024-05-27T00:22:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f2806800640 1 -- 172.21.15.98:0/3824819881 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c7e20 msgr2=0x7f280006f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f2806800640 1 --2- 172.21.15.98:0/3824819881 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c7e20 0x7f280006f260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f27f40097c0 tx=0x7f27f401c4a0 comp rx=0 tx=0).stop 2024-05-27T00:22:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f2806800640 1 -- 172.21.15.98:0/3824819881 shutdown_connections 2024-05-27T00:22:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f2806800640 1 --2- 172.21.15.98:0/3824819881 >> v2:172.21.15.204:3300/0 conn(0x7f28001511b0 0x7f2800151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f2806800640 1 --2- 172.21.15.98:0/3824819881 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c7e20 0x7f280006f260 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f2806800640 1 -- 172.21.15.98:0/3824819881 >> 172.21.15.98:0/3824819881 conn(0x7f28000cbe00 msgr2=0x7f28000cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f2806800640 1 -- 172.21.15.98:0/3824819881 shutdown_connections 2024-05-27T00:22:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f2806800640 1 -- 172.21.15.98:0/3824819881 wait complete. 2024-05-27T00:22:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.593+0000 7f2806800640 1 Processor -- start 2024-05-27T00:22:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2806800640 1 -- start start 2024-05-27T00:22:32.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2806800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28000c7e20 0x7f280013c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:32.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2806800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c8200 0x7f280013d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2806800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28001511b0 0x7f28001f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2806800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f280006b5f0 con 0x7f28001511b0 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2806800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f280006b470 con 0x7f28000c7e20 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2806800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f280006b770 con 0x7f28000c8200 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2805e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28001511b0 0x7f28001f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2805e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28001511b0 0x7f28001f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52562/0 (socket says 172.21.15.98:52562) 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2805e00640 1 -- 172.21.15.98:0/2168449541 learned_addr learned my addr 172.21.15.98:0/2168449541 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2805400640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28000c7e20 0x7f280013c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2804a00640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c8200 0x7f280013d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2805e00640 1 -- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c8200 msgr2=0x7f280013d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2805e00640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c8200 0x7f280013d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2805e00640 1 -- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28000c7e20 msgr2=0x7f280013c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2805e00640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28000c7e20 0x7f280013c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:32.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2805e00640 1 -- 172.21.15.98:0/2168449541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2800141550 con 0x7f28001511b0 2024-05-27T00:22:32.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2804a00640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c8200 0x7f280013d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:32.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2805e00640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28001511b0 0x7f28001f2660 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f27f800bf00 tx=0x7f27f800e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:32.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f27fd600640 1 -- 172.21.15.98:0/2168449541 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f800b310 con 0x7f28001511b0 2024-05-27T00:22:32.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f27fd600640 1 -- 172.21.15.98:0/2168449541 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f800ee50 con 0x7f28001511b0 2024-05-27T00:22:32.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f27fd600640 1 -- 172.21.15.98:0/2168449541 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8003be0 con 0x7f28001511b0 2024-05-27T00:22:32.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2806800640 1 -- 172.21.15.98:0/2168449541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28000cbba0 con 0x7f28001511b0 2024-05-27T00:22:32.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2806800640 1 -- 172.21.15.98:0/2168449541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28000cb8c0 con 0x7f28001511b0 2024-05-27T00:22:32.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.597+0000 7f2806800640 1 -- 172.21.15.98:0/2168449541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2800075c40 con 0x7f28001511b0 2024-05-27T00:22:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.601+0000 7f27fd600640 1 -- 172.21.15.98:0/2168449541 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f27f8014050 con 0x7f28001511b0 2024-05-27T00:22:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.601+0000 7f27fd600640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27e00678c0 0x7f27e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:32.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.601+0000 7f27fd600640 1 -- 172.21.15.98:0/2168449541 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f27f8088620 con 0x7f28001511b0 2024-05-27T00:22:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.601+0000 7f2805400640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27e00678c0 0x7f27e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:32.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.601+0000 7f2805400640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27e00678c0 0x7f27e0069d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f27f40ae000 tx=0x7f27f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.605+0000 7f27fd600640 1 -- 172.21.15.98:0/2168449541 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f27f8024020 con 0x7f28001511b0 2024-05-27T00:22:32.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 -- 172.21.15.98:0/2168449541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28000cac70 con 0x7f28001511b0 2024-05-27T00:22:32.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f27fd600640 1 -- 172.21.15.98:0/2168449541 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6009 (secure 0 0 0) 0x7f2800141550 con 0x7f28001511b0 2024-05-27T00:22:32.796 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:32.796 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:31.446935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:32.798 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 31 2024-05-27T00:22:32.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 -- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27e00678c0 msgr2=0x7f27e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:32.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27e00678c0 0x7f27e0069d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f27f40ae000 tx=0x7f27f4008040 comp rx=0 tx=0).stop 2024-05-27T00:22:32.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 -- 172.21.15.98:0/2168449541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28001511b0 msgr2=0x7f28001f2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:32.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28001511b0 0x7f28001f2660 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f27f800bf00 tx=0x7f27f800e490 comp rx=0 tx=0).stop 2024-05-27T00:22:32.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 -- 172.21.15.98:0/2168449541 shutdown_connections 2024-05-27T00:22:32.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f28001511b0 0x7f28001f2660 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:32.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28000c8200 0x7f280013d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:32.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27e00678c0 0x7f27e0069d80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:32.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 --2- 172.21.15.98:0/2168449541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28000c7e20 0x7f280013c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:32.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 -- 172.21.15.98:0/2168449541 >> 172.21.15.98:0/2168449541 conn(0x7f28000cbe00 msgr2=0x7f280006e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:32.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.793+0000 7f2806800640 1 -- 172.21.15.98:0/2168449541 shutdown_connections 2024-05-27T00:22:32.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:32.797+0000 7f2806800640 1 -- 172.21.15.98:0/2168449541 wait complete. 2024-05-27T00:22:32.817 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 31, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 31, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5343": { "addr": "172.21.15.98:6834/4290077910", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 4290077910, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 4290077910, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5343, "incarnation": 30, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:22:31.446935+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5343 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:22:34.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:34.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.961+0000 7fd996200640 1 Processor -- start 2024-05-27T00:22:34.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd996200640 1 -- start start 2024-05-27T00:22:34.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd996200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd990070fe0 0x7fd990154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:34.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd996200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd9900cb570 0x7fd990154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:34.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd996200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd99015a950 con 0x7fd990070c00 2024-05-27T00:22:34.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd996200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd99015e060 con 0x7fd990070fe0 2024-05-27T00:22:34.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd996200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd99015e340 con 0x7fd9900cb570 2024-05-27T00:22:34.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd986a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd990070fe0 0x7fd990154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:34.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd986a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd990070fe0 0x7fd990154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42282/0 (socket says 172.21.15.98:42282) 2024-05-27T00:22:34.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd986a00640 1 -- 172.21.15.98:0/1381501597 learned_addr learned my addr 172.21.15.98:0/1381501597 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:34.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd98fe00640 1 --2- 172.21.15.98:0/1381501597 >> v2:172.21.15.204:3300/0 conn(0x7fd9900cb570 0x7fd990154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:34.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd986a00640 1 -- 172.21.15.98:0/1381501597 >> v2:172.21.15.204:3300/0 conn(0x7fd9900cb570 msgr2=0x7fd990154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:34.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd986a00640 1 --2- 172.21.15.98:0/1381501597 >> v2:172.21.15.204:3300/0 conn(0x7fd9900cb570 0x7fd990154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:34.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd986a00640 1 -- 172.21.15.98:0/1381501597 >> v1:172.21.15.98:6789/0 conn(0x7fd990070c00 legacy=0x7fd9900c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:34.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd986a00640 1 -- 172.21.15.98:0/1381501597 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99015e620 con 0x7fd990070fe0 2024-05-27T00:22:34.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.965+0000 7fd98ea00640 1 -- 172.21.15.98:0/1381501597 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164121885 0 0) 0x7fd99015a950 con 0x7fd990070c00 2024-05-27T00:22:34.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd986a00640 1 --2- 172.21.15.98:0/1381501597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070fe0 0x7fd990154780 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd99015ef50 tx=0x7fd97801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b920c9f7cad714c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:34.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd98ea00640 1 -- 172.21.15.98:0/1381501597 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9780b5070 con 0x7fd990070fe0 2024-05-27T00:22:34.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd98ea00640 1 -- 172.21.15.98:0/1381501597 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9780a8a10 con 0x7fd990070fe0 2024-05-27T00:22:34.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd98ea00640 1 -- 172.21.15.98:0/1381501597 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9780a8de0 con 0x7fd990070fe0 2024-05-27T00:22:34.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd996200640 1 -- 172.21.15.98:0/1381501597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070fe0 msgr2=0x7fd990154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:34.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd996200640 1 --2- 172.21.15.98:0/1381501597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070fe0 0x7fd990154780 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd99015ef50 tx=0x7fd97801ed10 comp rx=0 tx=0).stop 2024-05-27T00:22:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd996200640 1 -- 172.21.15.98:0/1381501597 shutdown_connections 2024-05-27T00:22:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd996200640 1 --2- 172.21.15.98:0/1381501597 >> v2:172.21.15.204:3300/0 conn(0x7fd9900cb570 0x7fd990154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd996200640 1 --2- 172.21.15.98:0/1381501597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070fe0 0x7fd990154780 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd996200640 1 -- 172.21.15.98:0/1381501597 >> 172.21.15.98:0/1381501597 conn(0x7fd990072f90 msgr2=0x7fd9900733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd996200640 1 -- 172.21.15.98:0/1381501597 shutdown_connections 2024-05-27T00:22:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd996200640 1 -- 172.21.15.98:0/1381501597 wait complete. 2024-05-27T00:22:34.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd996200640 1 Processor -- start 2024-05-27T00:22:34.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.969+0000 7fd996200640 1 -- start start 2024-05-27T00:22:34.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd996200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070c00 0x7fd99014f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:34.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd996200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd990070fe0 0x7fd9900c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:34.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd996200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9900cb570 0x7fd9900c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:34.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd996200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9900c9110 con 0x7fd990070c00 2024-05-27T00:22:34.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd996200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd99015ebe0 con 0x7fd9900cb570 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd996200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd99015ae50 con 0x7fd990070fe0 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd98f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070c00 0x7fd99014f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd98f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070c00 0x7fd99014f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42298/0 (socket says 172.21.15.98:42298) 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd98f400640 1 -- 172.21.15.98:0/1837061890 learned_addr learned my addr 172.21.15.98:0/1837061890 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd986a00640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd990070fe0 0x7fd9900c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd98fe00640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9900cb570 0x7fd9900c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd98f400640 1 -- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd990070fe0 msgr2=0x7fd9900c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd98f400640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd990070fe0 0x7fd9900c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd98f400640 1 -- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9900cb570 msgr2=0x7fd9900c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd98f400640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9900cb570 0x7fd9900c7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd98f400640 1 -- 172.21.15.98:0/1837061890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9900c5de0 con 0x7fd990070c00 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd98f400640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070c00 0x7fd99014f030 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd98000c500 tx=0x7fd98000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd987e00640 1 -- 172.21.15.98:0/1837061890 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980017070 con 0x7fd990070c00 2024-05-27T00:22:34.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd987e00640 1 -- 172.21.15.98:0/1837061890 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd980003600 con 0x7fd990070c00 2024-05-27T00:22:34.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd987e00640 1 -- 172.21.15.98:0/1837061890 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9800039f0 con 0x7fd990070c00 2024-05-27T00:22:34.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd996200640 1 -- 172.21.15.98:0/1837061890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9901f1100 con 0x7fd990070c00 2024-05-27T00:22:34.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd996200640 1 -- 172.21.15.98:0/1837061890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9901f0e40 con 0x7fd990070c00 2024-05-27T00:22:34.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd996200640 1 -- 172.21.15.98:0/1837061890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd990075cb0 con 0x7fd990070c00 2024-05-27T00:22:34.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.973+0000 7fd987e00640 1 -- 172.21.15.98:0/1837061890 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd98001a070 con 0x7fd990070c00 2024-05-27T00:22:34.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.977+0000 7fd987e00640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd96c0678c0 0x7fd96c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:34.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.977+0000 7fd987e00640 1 -- 172.21.15.98:0/1837061890 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7fd980085320 con 0x7fd990070c00 2024-05-27T00:22:34.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.977+0000 7fd986a00640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd96c0678c0 0x7fd96c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:34.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.977+0000 7fd986a00640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd96c0678c0 0x7fd96c069d80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd9780b2000 tx=0x7fd9780aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:34.981+0000 7fd987e00640 1 -- 172.21.15.98:0/1837061890 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd980055cf0 con 0x7fd990070c00 2024-05-27T00:22:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 -- 172.21.15.98:0/1837061890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd99006eb60 con 0x7fd990070c00 2024-05-27T00:22:35.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd987e00640 1 -- 172.21.15.98:0/1837061890 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6009 (secure 0 0 0) 0x7fd9800850e0 con 0x7fd990070c00 2024-05-27T00:22:35.170 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:35.171 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:31.446935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:35.171 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 31 2024-05-27T00:22:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 -- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd96c0678c0 msgr2=0x7fd96c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:35.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd96c0678c0 0x7fd96c069d80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd9780b2000 tx=0x7fd9780aa070 comp rx=0 tx=0).stop 2024-05-27T00:22:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 -- 172.21.15.98:0/1837061890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070c00 msgr2=0x7fd99014f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070c00 0x7fd99014f030 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd98000c500 tx=0x7fd98000c9c0 comp rx=0 tx=0).stop 2024-05-27T00:22:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 -- 172.21.15.98:0/1837061890 shutdown_connections 2024-05-27T00:22:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd96c0678c0 0x7fd96c069d80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9900cb570 0x7fd9900c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd990070fe0 0x7fd9900c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 --2- 172.21.15.98:0/1837061890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd990070c00 0x7fd99014f030 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 -- 172.21.15.98:0/1837061890 >> 172.21.15.98:0/1837061890 conn(0x7fd990072f90 msgr2=0x7fd9900c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 -- 172.21.15.98:0/1837061890 shutdown_connections 2024-05-27T00:22:35.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:35.169+0000 7fd996200640 1 -- 172.21.15.98:0/1837061890 wait complete. 2024-05-27T00:22:35.191 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:22:37.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.337+0000 7f6164600640 1 Processor -- start 2024-05-27T00:22:37.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.337+0000 7f6164600640 1 -- start start 2024-05-27T00:22:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6164600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f615c0735d0 0x7f615c151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6164600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615c0c17d0 con 0x7f615c0731f0 2024-05-27T00:22:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6164600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615c070e10 con 0x7f615c0c79f0 2024-05-27T00:22:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6164600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f615c071990 con 0x7f615c0735d0 2024-05-27T00:22:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6161c00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f615c0731f0 0x7f615c0c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42992/0 (socket says 172.21.15.98:42992) 2024-05-27T00:22:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6161c00640 1 -- 172.21.15.98:0/2294827295 learned_addr learned my addr 172.21.15.98:0/2294827295 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6161200640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f615c0735d0 0x7f615c151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578373878 0 0) 0x7f615c0c17d0 con 0x7f615c0731f0 2024-05-27T00:22:37.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6144005740 con 0x7f615c0731f0 2024-05-27T00:22:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 32470201 0 0) 0x7f6144005740 con 0x7f615c0731f0 2024-05-27T00:22:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f615c0735d0 msgr2=0x7f615c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f615c0735d0 0x7f615c151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 >> v1:172.21.15.204:6789/0 conn(0x7f615c0c79f0 legacy=0x7f615c157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615c0710f0 con 0x7f615c0731f0 2024-05-27T00:22:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f6150001680 con 0x7f615c0731f0 2024-05-27T00:22:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 >> v1:172.21.15.98:6789/0 conn(0x7f615c0731f0 legacy=0x7f615c0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:37.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f615c0735d0 0x7f6144005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6144006180 0x7f6144006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6144006f60 0x7f61440073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f615c071990 con 0x7f6144006180 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f614400dd50 con 0x7f6144006f60 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f614400df50 con 0x7f615c0735d0 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6161200640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6144006180 0x7f6144006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6161c00640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f615c0735d0 0x7f6144005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6162600640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6144006f60 0x7f61440073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6161200640 1 -- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f615c0735d0 msgr2=0x7f6144005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6161200640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f615c0735d0 0x7f6144005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6161200640 1 -- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6144006f60 msgr2=0x7f61440073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6161200640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6144006f60 0x7f61440073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6161200640 1 -- 172.21.15.98:0/2294827295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6144005740 con 0x7f6144006180 2024-05-27T00:22:37.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.341+0000 7f6162600640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6144006f60 0x7f61440073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f6161200640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6144006180 0x7f6144006a20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6148009ab0 tx=0x7f614801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34022bda254ceed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61480b2070 con 0x7f6144006180 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61480a5640 con 0x7f6144006180 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f615be00640 1 -- 172.21.15.98:0/2294827295 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61480096b0 con 0x7f6144006180 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f6164600640 1 -- 172.21.15.98:0/2294827295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6144006180 msgr2=0x7f6144006a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f6164600640 1 --2- 172.21.15.98:0/2294827295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6144006180 0x7f6144006a20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6148009ab0 tx=0x7f614801ca50 comp rx=0 tx=0).stop 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f6162600640 1 -- 172.21.15.98:0/2294827295 reap_dead start 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f6164600640 1 -- 172.21.15.98:0/2294827295 shutdown_connections 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f6164600640 1 -- 172.21.15.98:0/2294827295 >> 172.21.15.98:0/2294827295 conn(0x7f615c067530 msgr2=0x7f615c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f6164600640 1 -- 172.21.15.98:0/2294827295 shutdown_connections 2024-05-27T00:22:37.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.345+0000 7f6164600640 1 -- 172.21.15.98:0/2294827295 wait complete. 2024-05-27T00:22:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 Processor -- start 2024-05-27T00:22:37.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 -- start start 2024-05-27T00:22:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6144006180 0x7f615c149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f615c1599b0 0x7f615c147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615c159e30 0x7f615c147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f615c0c8400 con 0x7f615c1599b0 2024-05-27T00:22:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f615c0716b0 con 0x7f615c159e30 2024-05-27T00:22:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f615c158080 con 0x7f6144006180 2024-05-27T00:22:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6161200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f615c1599b0 0x7f615c147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:37.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6161200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f615c1599b0 0x7f615c147040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42322/0 (socket says 172.21.15.98:42322) 2024-05-27T00:22:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6161200640 1 -- 172.21.15.98:0/2810568666 learned_addr learned my addr 172.21.15.98:0/2810568666 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6162600640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615c159e30 0x7f615c147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6161c00640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6144006180 0x7f615c149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6161200640 1 -- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6144006180 msgr2=0x7f615c149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6161200640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6144006180 0x7f615c149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6161200640 1 -- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615c159e30 msgr2=0x7f615c147580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6161200640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615c159e30 0x7f615c147580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6161200640 1 -- 172.21.15.98:0/2810568666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615c14a6a0 con 0x7f615c1599b0 2024-05-27T00:22:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6162600640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615c159e30 0x7f615c147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:37.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6161200640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f615c1599b0 0x7f615c147040 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6148009ab0 tx=0x7f6144007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:37.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6159600640 1 -- 172.21.15.98:0/2810568666 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61480bc030 con 0x7f615c1599b0 2024-05-27T00:22:37.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6159600640 1 -- 172.21.15.98:0/2810568666 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f614800f000 con 0x7f615c1599b0 2024-05-27T00:22:37.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6159600640 1 -- 172.21.15.98:0/2810568666 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61480b2040 con 0x7f615c1599b0 2024-05-27T00:22:37.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 -- 172.21.15.98:0/2810568666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615c147ac0 con 0x7f615c1599b0 2024-05-27T00:22:37.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 -- 172.21.15.98:0/2810568666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f615c1ec340 con 0x7f615c1599b0 2024-05-27T00:22:37.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.349+0000 7f6164600640 1 -- 172.21.15.98:0/2810568666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f615c075cb0 con 0x7f615c1599b0 2024-05-27T00:22:37.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.353+0000 7f6159600640 1 -- 172.21.15.98:0/2810568666 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f61480bf070 con 0x7f615c1599b0 2024-05-27T00:22:37.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.353+0000 7f6159600640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f613c0678c0 0x7f613c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:37.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.353+0000 7f6159600640 1 -- 172.21.15.98:0/2810568666 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f6148120130 con 0x7f615c1599b0 2024-05-27T00:22:37.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.353+0000 7f6161c00640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f613c0678c0 0x7f613c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:37.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.353+0000 7f6161c00640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f613c0678c0 0x7f613c069d80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6150005a40 tx=0x7f6150009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.357+0000 7f6159600640 1 -- 172.21.15.98:0/2810568666 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f6148016760 con 0x7f615c1599b0 2024-05-27T00:22:37.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.553+0000 7f6164600640 1 -- 172.21.15.98:0/2810568666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f615c0b2010 con 0x7f615c1599b0 2024-05-27T00:22:37.558 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:37.558 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:37.450848+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:37.558 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-05-27T00:22:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6159600640 1 -- 172.21.15.98:0/2810568666 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6007 (secure 0 0 0) 0x7f61480f0a40 con 0x7f615c1599b0 2024-05-27T00:22:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 -- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f613c0678c0 msgr2=0x7f613c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f613c0678c0 0x7f613c069d80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6150005a40 tx=0x7f6150009040 comp rx=0 tx=0).stop 2024-05-27T00:22:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 -- 172.21.15.98:0/2810568666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f615c1599b0 msgr2=0x7f615c147040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f615c1599b0 0x7f615c147040 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6148009ab0 tx=0x7f6144007a50 comp rx=0 tx=0).stop 2024-05-27T00:22:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 -- 172.21.15.98:0/2810568666 shutdown_connections 2024-05-27T00:22:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f613c0678c0 0x7f613c069d80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f615c159e30 0x7f615c147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f615c1599b0 0x7f615c147040 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 --2- 172.21.15.98:0/2810568666 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6144006180 0x7f615c149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 -- 172.21.15.98:0/2810568666 >> 172.21.15.98:0/2810568666 conn(0x7f615c067530 msgr2=0x7f615c14ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 -- 172.21.15.98:0/2810568666 shutdown_connections 2024-05-27T00:22:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:37.557+0000 7f6164600640 1 -- 172.21.15.98:0/2810568666 wait complete. 2024-05-27T00:22:37.911 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:22:39.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:40.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f3549e00640 1 Processor -- start 2024-05-27T00:22:40.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f3549e00640 1 -- start start 2024-05-27T00:22:40.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f3549e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3544154de0 0x7f35440b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:40.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f3549e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f35440d0070 0x7f35440d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:40.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f3549e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35440b3f20 con 0x7f3544154a00 2024-05-27T00:22:40.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f3549e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f35440b4200 con 0x7f3544154de0 2024-05-27T00:22:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f3549e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f35440b44e0 con 0x7f35440d0070 2024-05-27T00:22:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f353a000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3544154de0 0x7f35440b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f353a000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3544154de0 0x7f35440b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42334/0 (socket says 172.21.15.98:42334) 2024-05-27T00:22:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f353a000640 1 -- 172.21.15.98:0/3421225155 learned_addr learned my addr 172.21.15.98:0/3421225155 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.057+0000 7f3543400640 1 --2- 172.21.15.98:0/3421225155 >> v2:172.21.15.204:3300/0 conn(0x7f35440d0070 0x7f35440d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f353a000640 1 -- 172.21.15.98:0/3421225155 >> v2:172.21.15.204:3300/0 conn(0x7f35440d0070 msgr2=0x7f35440d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f353a000640 1 --2- 172.21.15.98:0/3421225155 >> v2:172.21.15.204:3300/0 conn(0x7f35440d0070 0x7f35440d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f353a000640 1 -- 172.21.15.98:0/3421225155 >> v1:172.21.15.204:6789/0 conn(0x7f3544154a00 legacy=0x7f354415b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f353a000640 1 -- 172.21.15.98:0/3421225155 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35440b47c0 con 0x7f3544154de0 2024-05-27T00:22:40.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f353a000640 1 --2- 172.21.15.98:0/3421225155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154de0 0x7f35440b2c50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f35440caef0 tx=0x7f352c01f200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377a9970c5f5e9df server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:40.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f3542000640 1 -- 172.21.15.98:0/3421225155 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c0b4070 con 0x7f3544154de0 2024-05-27T00:22:40.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f3542000640 1 -- 172.21.15.98:0/3421225155 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c0af400 con 0x7f3544154de0 2024-05-27T00:22:40.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f3542000640 1 -- 172.21.15.98:0/3421225155 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c0af7d0 con 0x7f3544154de0 2024-05-27T00:22:40.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f3549e00640 1 -- 172.21.15.98:0/3421225155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154de0 msgr2=0x7f35440b2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:40.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f3549e00640 1 --2- 172.21.15.98:0/3421225155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154de0 0x7f35440b2c50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f35440caef0 tx=0x7f352c01f200 comp rx=0 tx=0).stop 2024-05-27T00:22:40.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f3549e00640 1 -- 172.21.15.98:0/3421225155 shutdown_connections 2024-05-27T00:22:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f3549e00640 1 --2- 172.21.15.98:0/3421225155 >> v2:172.21.15.204:3300/0 conn(0x7f35440d0070 0x7f35440d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f3549e00640 1 --2- 172.21.15.98:0/3421225155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154de0 0x7f35440b2c50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f3549e00640 1 -- 172.21.15.98:0/3421225155 >> 172.21.15.98:0/3421225155 conn(0x7f3544067530 msgr2=0x7f354406c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.061+0000 7f3549e00640 1 -- 172.21.15.98:0/3421225155 shutdown_connections 2024-05-27T00:22:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 -- 172.21.15.98:0/3421225155 wait complete. 2024-05-27T00:22:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 Processor -- start 2024-05-27T00:22:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 -- start start 2024-05-27T00:22:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35440d0070 0x7f354415e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154a00 0x7f35441e4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3544154de0 0x7f35441e4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f35440b37a0 con 0x7f3544154a00 2024-05-27T00:22:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f35440b3620 con 0x7f35440d0070 2024-05-27T00:22:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f35440cc7e0 con 0x7f3544154de0 2024-05-27T00:22:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353a000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154a00 0x7f35441e4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353a000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154a00 0x7f35441e4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42336/0 (socket says 172.21.15.98:42336) 2024-05-27T00:22:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353a000640 1 -- 172.21.15.98:0/3126397617 learned_addr learned my addr 172.21.15.98:0/3126397617 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3542a00640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35440d0070 0x7f354415e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3543400640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3544154de0 0x7f35441e4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353a000640 1 -- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3544154de0 msgr2=0x7f35441e4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353a000640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3544154de0 0x7f35441e4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353a000640 1 -- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35440d0070 msgr2=0x7f354415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353a000640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35440d0070 0x7f354415e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353a000640 1 -- 172.21.15.98:0/3126397617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35441f02c0 con 0x7f3544154a00 2024-05-27T00:22:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353a000640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154a00 0x7f35441e4200 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f352c0b2040 tx=0x7f352c01bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353be00640 1 -- 172.21.15.98:0/3126397617 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c0b4040 con 0x7f3544154a00 2024-05-27T00:22:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353be00640 1 -- 172.21.15.98:0/3126397617 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c0c0070 con 0x7f3544154a00 2024-05-27T00:22:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f353be00640 1 -- 172.21.15.98:0/3126397617 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c009510 con 0x7f3544154a00 2024-05-27T00:22:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 -- 172.21.15.98:0/3126397617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35441f05a0 con 0x7f3544154a00 2024-05-27T00:22:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.065+0000 7f3549e00640 1 -- 172.21.15.98:0/3126397617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35441f0b60 con 0x7f3544154a00 2024-05-27T00:22:40.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.069+0000 7f3549e00640 1 -- 172.21.15.98:0/3126397617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3544075cb0 con 0x7f3544154a00 2024-05-27T00:22:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.069+0000 7f353be00640 1 -- 172.21.15.98:0/3126397617 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f352c0ba660 con 0x7f3544154a00 2024-05-27T00:22:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.069+0000 7f353be00640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35200678c0 0x7f3520069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.069+0000 7f353be00640 1 -- 172.21.15.98:0/3126397617 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f352c124220 con 0x7f3544154a00 2024-05-27T00:22:40.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.069+0000 7f3542a00640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35200678c0 0x7f3520069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.069+0000 7f3542a00640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35200678c0 0x7f3520069d80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3534004e30 tx=0x7f3534009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.073+0000 7f353be00640 1 -- 172.21.15.98:0/3126397617 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f352c018d60 con 0x7f3544154a00 2024-05-27T00:22:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.269+0000 7f3549e00640 1 -- 172.21.15.98:0/3126397617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35440cbc30 con 0x7f3544154a00 2024-05-27T00:22:40.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.269+0000 7f353be00640 1 -- 172.21.15.98:0/3126397617 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6007 (secure 0 0 0) 0x7f352c0f4b60 con 0x7f3544154a00 2024-05-27T00:22:40.271 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:40.272 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:37.450848+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-05-27T00:22:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 -- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35200678c0 msgr2=0x7f3520069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35200678c0 0x7f3520069d80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3534004e30 tx=0x7f3534009040 comp rx=0 tx=0).stop 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 -- 172.21.15.98:0/3126397617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154a00 msgr2=0x7f35441e4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154a00 0x7f35441e4200 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f352c0b2040 tx=0x7f352c01bfa0 comp rx=0 tx=0).stop 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 -- 172.21.15.98:0/3126397617 shutdown_connections 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3544154de0 0x7f35441e4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544154a00 0x7f35441e4200 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35200678c0 0x7f3520069d80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 --2- 172.21.15.98:0/3126397617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35440d0070 0x7f354415e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 -- 172.21.15.98:0/3126397617 >> 172.21.15.98:0/3126397617 conn(0x7f3544067530 msgr2=0x7f35440cdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 -- 172.21.15.98:0/3126397617 shutdown_connections 2024-05-27T00:22:40.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:40.273+0000 7f3549e00640 1 -- 172.21.15.98:0/3126397617 wait complete. 2024-05-27T00:22:40.293 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:22:42.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc46a600640 1 Processor -- start 2024-05-27T00:22:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc46a600640 1 -- start start 2024-05-27T00:22:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc46a600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc464154b00 0x7fc464071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:42.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc46a600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc464154ee0 con 0x7fc464072bc0 2024-05-27T00:22:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc46a600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4640c4e40 con 0x7fc46415b1f0 2024-05-27T00:22:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc46a600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc464071fd0 con 0x7fc464154b00 2024-05-27T00:22:42.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc469200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc464154b00 0x7fc464071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc469200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc464154b00 0x7fc464071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42340/0 (socket says 172.21.15.98:42340) 2024-05-27T00:22:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc469200640 1 -- 172.21.15.98:0/518987223 learned_addr learned my addr 172.21.15.98:0/518987223 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc469c00640 1 --1- 172.21.15.98:0/518987223 >> v1:172.21.15.98:6789/0 conn(0x7fc464072bc0 0x7fc464073010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43006/0 (socket says 172.21.15.98:43006) 2024-05-27T00:22:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc469200640 1 -- 172.21.15.98:0/518987223 >> v1:172.21.15.204:6789/0 conn(0x7fc46415b1f0 legacy=0x7fc4640cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc469200640 1 -- 172.21.15.98:0/518987223 >> v1:172.21.15.98:6789/0 conn(0x7fc464072bc0 legacy=0x7fc464073010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc469200640 1 -- 172.21.15.98:0/518987223 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4640c5120 con 0x7fc464154b00 2024-05-27T00:22:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.441+0000 7fc463400640 1 -- 172.21.15.98:0/518987223 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862828588 0 0) 0x7fc464154ee0 con 0x7fc464072bc0 2024-05-27T00:22:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc469200640 1 --2- 172.21.15.98:0/518987223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc464154b00 0x7fc464071a90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc458009870 tx=0x7fc45801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1873f508e7e8ca56 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc463400640 1 -- 172.21.15.98:0/518987223 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4580b2070 con 0x7fc464154b00 2024-05-27T00:22:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc463400640 1 -- 172.21.15.98:0/518987223 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4580a9070 con 0x7fc464154b00 2024-05-27T00:22:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc463400640 1 -- 172.21.15.98:0/518987223 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4580ad880 con 0x7fc464154b00 2024-05-27T00:22:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc46a600640 1 -- 172.21.15.98:0/518987223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc464154b00 msgr2=0x7fc464071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc46a600640 1 --2- 172.21.15.98:0/518987223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc464154b00 0x7fc464071a90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc458009870 tx=0x7fc45801c560 comp rx=0 tx=0).stop 2024-05-27T00:22:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc46a600640 1 -- 172.21.15.98:0/518987223 shutdown_connections 2024-05-27T00:22:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc46a600640 1 --2- 172.21.15.98:0/518987223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc464154b00 0x7fc464071a90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc46a600640 1 -- 172.21.15.98:0/518987223 >> 172.21.15.98:0/518987223 conn(0x7fc4640d02e0 msgr2=0x7fc4640d06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc46a600640 1 -- 172.21.15.98:0/518987223 shutdown_connections 2024-05-27T00:22:42.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc46a600640 1 -- 172.21.15.98:0/518987223 wait complete. 2024-05-27T00:22:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc46a600640 1 Processor -- start 2024-05-27T00:22:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.445+0000 7fc46a600640 1 -- start start 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc46a600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc464072bc0 0x7fc464151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc46a600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc464154b00 0x7fc464158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc46a600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc46415b1f0 0x7fc4640cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc46a600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4640c5ee0 con 0x7fc46415b1f0 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc46a600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc464168840 con 0x7fc464072bc0 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc46a600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc464152170 con 0x7fc464154b00 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc46415b1f0 0x7fc4640cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc46415b1f0 0x7fc4640cdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42344/0 (socket says 172.21.15.98:42344) 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469c00640 1 -- 172.21.15.98:0/727588760 learned_addr learned my addr 172.21.15.98:0/727588760 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469200640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc464072bc0 0x7fc464151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc463e00640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc464154b00 0x7fc464158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469c00640 1 -- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc464154b00 msgr2=0x7fc464158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469c00640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc464154b00 0x7fc464158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469c00640 1 -- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc464072bc0 msgr2=0x7fc464151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469c00640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc464072bc0 0x7fc464151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469c00640 1 -- 172.21.15.98:0/727588760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc464159400 con 0x7fc46415b1f0 2024-05-27T00:22:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc463e00640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc464154b00 0x7fc464158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469200640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc464072bc0 0x7fc464151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc469c00640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc46415b1f0 0x7fc4640cdcc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00c830 tx=0x7fc45c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc460c00640 1 -- 172.21.15.98:0/727588760 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c00ce20 con 0x7fc46415b1f0 2024-05-27T00:22:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc460c00640 1 -- 172.21.15.98:0/727588760 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc45c00a640 con 0x7fc46415b1f0 2024-05-27T00:22:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc460c00640 1 -- 172.21.15.98:0/727588760 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c00aa30 con 0x7fc46415b1f0 2024-05-27T00:22:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc46a600640 1 -- 172.21.15.98:0/727588760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc464150060 con 0x7fc46415b1f0 2024-05-27T00:22:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc46a600640 1 -- 172.21.15.98:0/727588760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc464150620 con 0x7fc46415b1f0 2024-05-27T00:22:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.449+0000 7fc46a600640 1 -- 172.21.15.98:0/727588760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc464075c40 con 0x7fc46415b1f0 2024-05-27T00:22:42.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.453+0000 7fc460c00640 1 -- 172.21.15.98:0/727588760 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc45c00abd0 con 0x7fc46415b1f0 2024-05-27T00:22:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.453+0000 7fc460c00640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4440678f0 0x7fc444069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.453+0000 7fc460c00640 1 -- 172.21.15.98:0/727588760 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7fc45c085ca0 con 0x7fc46415b1f0 2024-05-27T00:22:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.453+0000 7fc469200640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4440678f0 0x7fc444069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.453+0000 7fc469200640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4440678f0 0x7fc444069db0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc4580af000 tx=0x7fc4580a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.457+0000 7fc460c00640 1 -- 172.21.15.98:0/727588760 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc45c056710 con 0x7fc46415b1f0 2024-05-27T00:22:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.657+0000 7fc46a600640 1 -- 172.21.15.98:0/727588760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4640c68c0 con 0x7fc46415b1f0 2024-05-27T00:22:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.657+0000 7fc460c00640 1 -- 172.21.15.98:0/727588760 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6007 (secure 0 0 0) 0x7fc45c056010 con 0x7fc46415b1f0 2024-05-27T00:22:42.660 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:42.660 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:37.450848+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 32 2024-05-27T00:22:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 -- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4440678f0 msgr2=0x7fc444069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4440678f0 0x7fc444069db0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc4580af000 tx=0x7fc4580a7000 comp rx=0 tx=0).stop 2024-05-27T00:22:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 -- 172.21.15.98:0/727588760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc46415b1f0 msgr2=0x7fc4640cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc46415b1f0 0x7fc4640cdcc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00c830 tx=0x7fc45c00ccf0 comp rx=0 tx=0).stop 2024-05-27T00:22:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 -- 172.21.15.98:0/727588760 shutdown_connections 2024-05-27T00:22:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc46415b1f0 0x7fc4640cdcc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc464154b00 0x7fc464158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4440678f0 0x7fc444069db0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 --2- 172.21.15.98:0/727588760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc464072bc0 0x7fc464151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 -- 172.21.15.98:0/727588760 >> 172.21.15.98:0/727588760 conn(0x7fc4640d02e0 msgr2=0x7fc464152e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 -- 172.21.15.98:0/727588760 shutdown_connections 2024-05-27T00:22:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:42.661+0000 7fc46a600640 1 -- 172.21.15.98:0/727588760 wait complete. 2024-05-27T00:22:42.681 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:22:44.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:44.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598f200640 1 Processor -- start 2024-05-27T00:22:44.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598f200640 1 -- start start 2024-05-27T00:22:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5990157cd0 0x7f59900c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598f200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59900c59d0 con 0x7f59900b3cb0 2024-05-27T00:22:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598f200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59900cabb0 con 0x7f59901580b0 2024-05-27T00:22:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f59900c51e0 con 0x7f5990157cd0 2024-05-27T00:22:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598e800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f59900b3cb0 0x7f59900c9a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33788/0 (socket says 172.21.15.98:33788) 2024-05-27T00:22:44.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598e800640 1 -- 172.21.15.98:0/1796878972 learned_addr learned my addr 172.21.15.98:0/1796878972 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598de00640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5990157cd0 0x7f59900c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2348104711 0 0) 0x7f59900c59d0 con 0x7f59900b3cb0 2024-05-27T00:22:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5974003530 con 0x7f59900b3cb0 2024-05-27T00:22:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2198278302 0 0) 0x7f5974003530 con 0x7f59900b3cb0 2024-05-27T00:22:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5990157cd0 msgr2=0x7f59900c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5990157cd0 0x7f59900c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 >> v1:172.21.15.204:6789/0 conn(0x7f59901580b0 legacy=0x7f599015e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59900cae50 con 0x7f59900b3cb0 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f5988001a00 con 0x7f59900b3cb0 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 >> v1:172.21.15.98:6789/0 conn(0x7f59900b3cb0 legacy=0x7f59900c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 0x7f5974003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59740043a0 0x7f5974004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5990157cd0 0x7f59740050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59900c51e0 con 0x7f59740039f0 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f597400b9d0 con 0x7f59740043a0 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f597400bbd0 con 0x7f5990157cd0 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598de00640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 0x7f5974003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598e800640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5990157cd0 0x7f59740050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f5985400640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59740043a0 0x7f5974004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598de00640 1 -- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5990157cd0 msgr2=0x7f59740050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598de00640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5990157cd0 0x7f59740050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598de00640 1 -- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59740043a0 msgr2=0x7f5974004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598de00640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59740043a0 0x7f5974004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:44.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.829+0000 7f598de00640 1 -- 172.21.15.98:0/1796878972 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5974003530 con 0x7f59740039f0 2024-05-27T00:22:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598de00640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 0x7f5974003dd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5980009ae0 tx=0x7f598001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc9a84b996a9ca9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59800b2070 con 0x7f59740039f0 2024-05-27T00:22:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59800a5660 con 0x7f59740039f0 2024-05-27T00:22:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598d400640 1 -- 172.21.15.98:0/1796878972 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59800096b0 con 0x7f59740039f0 2024-05-27T00:22:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598f200640 1 -- 172.21.15.98:0/1796878972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 msgr2=0x7f5974003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598f200640 1 --2- 172.21.15.98:0/1796878972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 0x7f5974003dd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5980009ae0 tx=0x7f598001ca70 comp rx=0 tx=0).stop 2024-05-27T00:22:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598e800640 1 -- 172.21.15.98:0/1796878972 reap_dead start 2024-05-27T00:22:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598f200640 1 -- 172.21.15.98:0/1796878972 shutdown_connections 2024-05-27T00:22:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598f200640 1 -- 172.21.15.98:0/1796878972 >> 172.21.15.98:0/1796878972 conn(0x7f5990070b30 msgr2=0x7f5990070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598f200640 1 -- 172.21.15.98:0/1796878972 shutdown_connections 2024-05-27T00:22:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.833+0000 7f598f200640 1 -- 172.21.15.98:0/1796878972 wait complete. 2024-05-27T00:22:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598f200640 1 Processor -- start 2024-05-27T00:22:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598f200640 1 -- start start 2024-05-27T00:22:44.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598f200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 0x7f5990154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:44.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59901e5320 0x7f599014dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:44.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f599014a9c0 0x7f599014ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598f200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f599015e120 con 0x7f59740039f0 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f599015de40 con 0x7f599014a9c0 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f59900cb410 con 0x7f59901e5320 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 0x7f5990154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 0x7f5990154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42634/0 (socket says 172.21.15.98:42634) 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598de00640 1 -- 172.21.15.98:0/2302809953 learned_addr learned my addr 172.21.15.98:0/2302809953 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f5985400640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59901e5320 0x7f599014dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598e800640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f599014a9c0 0x7f599014ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598de00640 1 -- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59901e5320 msgr2=0x7f599014dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598de00640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59901e5320 0x7f599014dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598de00640 1 -- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f599014a9c0 msgr2=0x7f599014ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598de00640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f599014a9c0 0x7f599014ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598de00640 1 -- 172.21.15.98:0/2302809953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5990157840 con 0x7f59740039f0 2024-05-27T00:22:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f5985400640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59901e5320 0x7f599014dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:44.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598de00640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 0x7f5990154270 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5980098480 tx=0x7f5974005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:44.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f5986a00640 1 -- 172.21.15.98:0/2302809953 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59800b2040 con 0x7f59740039f0 2024-05-27T00:22:44.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f5986a00640 1 -- 172.21.15.98:0/2302809953 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59800a5e10 con 0x7f59740039f0 2024-05-27T00:22:44.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f5986a00640 1 -- 172.21.15.98:0/2302809953 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980004020 con 0x7f59740039f0 2024-05-27T00:22:44.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598f200640 1 -- 172.21.15.98:0/2302809953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f599006b790 con 0x7f59740039f0 2024-05-27T00:22:44.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.837+0000 7f598f200640 1 -- 172.21.15.98:0/2302809953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59901e5e60 con 0x7f59740039f0 2024-05-27T00:22:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.841+0000 7f598f200640 1 -- 172.21.15.98:0/2302809953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5990075c40 con 0x7f59740039f0 2024-05-27T00:22:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.841+0000 7f5986a00640 1 -- 172.21.15.98:0/2302809953 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f59800c1020 con 0x7f59740039f0 2024-05-27T00:22:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.841+0000 7f5986a00640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f596c0678f0 0x7f596c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.841+0000 7f5986a00640 1 -- 172.21.15.98:0/2302809953 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f598011ff80 con 0x7f59740039f0 2024-05-27T00:22:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.841+0000 7f5985400640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f596c0678f0 0x7f596c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:44.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.841+0000 7f5985400640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f596c0678f0 0x7f596c069db0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f59901edf40 tx=0x7f597800ac60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:44.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:44.845+0000 7f5986a00640 1 -- 172.21.15.98:0/2302809953 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f5980016780 con 0x7f59740039f0 2024-05-27T00:22:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.033+0000 7f598f200640 1 -- 172.21.15.98:0/2302809953 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f599015b330 con 0x7f59740039f0 2024-05-27T00:22:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.033+0000 7f5986a00640 1 -- 172.21.15.98:0/2302809953 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6013 (secure 0 0 0) 0x7f59800f0800 con 0x7f59740039f0 2024-05-27T00:22:45.036 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:45.036 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:42.641140+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:45.036 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 33 2024-05-27T00:22:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.033+0000 7f598f200640 1 -- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f596c0678f0 msgr2=0x7f596c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.033+0000 7f598f200640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f596c0678f0 0x7f596c069db0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f59901edf40 tx=0x7f597800ac60 comp rx=0 tx=0).stop 2024-05-27T00:22:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.037+0000 7f598f200640 1 -- 172.21.15.98:0/2302809953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 msgr2=0x7f5990154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.037+0000 7f598f200640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 0x7f5990154270 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5980098480 tx=0x7f5974005840 comp rx=0 tx=0).stop 2024-05-27T00:22:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.037+0000 7f598f200640 1 -- 172.21.15.98:0/2302809953 shutdown_connections 2024-05-27T00:22:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.037+0000 7f598f200640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f596c0678f0 0x7f596c069db0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.037+0000 7f598f200640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f599014a9c0 0x7f599014ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.037+0000 7f598f200640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59901e5320 0x7f599014dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.037+0000 7f598f200640 1 --2- 172.21.15.98:0/2302809953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59740039f0 0x7f5990154270 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:45.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.037+0000 7f598f200640 1 -- 172.21.15.98:0/2302809953 >> 172.21.15.98:0/2302809953 conn(0x7f5990070b30 msgr2=0x7f5990150160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.037+0000 7f598f200640 1 -- 172.21.15.98:0/2302809953 shutdown_connections 2024-05-27T00:22:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:45.037+0000 7f598f200640 1 -- 172.21.15.98:0/2302809953 wait complete. 2024-05-27T00:22:45.057 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 33, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 33, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5343": { "addr": "172.21.15.98:6834/4290077910", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 4290077910, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 4290077910, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5343, "incarnation": 30, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:clientreplay", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:22:42.641140+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5343 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:22:47.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.201+0000 7f5931a00640 1 Processor -- start 2024-05-27T00:22:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.201+0000 7f5931a00640 1 -- start start 2024-05-27T00:22:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f5931a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f592c1511d0 0x7f592c070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f5931a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f592c0cc8f0 0x7f592c0c1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f5931a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f592c0c1000 con 0x7f592c0c8670 2024-05-27T00:22:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f5931a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f592c0b1dd0 con 0x7f592c0cc8f0 2024-05-27T00:22:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f5931a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f592c1515b0 con 0x7f592c1511d0 2024-05-27T00:22:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f592a000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f592c0cc8f0 0x7f592c0c1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f592a000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f592c0cc8f0 0x7f592c0c1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42636/0 (socket says 172.21.15.98:42636) 2024-05-27T00:22:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f592a000640 1 -- 172.21.15.98:0/2627936540 learned_addr learned my addr 172.21.15.98:0/2627936540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f592aa00640 1 --2- 172.21.15.98:0/2627936540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f592c1511d0 0x7f592c070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f592a000640 1 -- 172.21.15.98:0/2627936540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f592c1511d0 msgr2=0x7f592c070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f592a000640 1 --2- 172.21.15.98:0/2627936540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f592c1511d0 0x7f592c070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f592a000640 1 -- 172.21.15.98:0/2627936540 >> v1:172.21.15.204:6789/0 conn(0x7f592c0c8670 legacy=0x7f592c15a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.205+0000 7f592a000640 1 -- 172.21.15.98:0/2627936540 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592c0c83c0 con 0x7f592c0cc8f0 2024-05-27T00:22:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f592a000640 1 --2- 172.21.15.98:0/2627936540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f592c0cc8f0 0x7f592c0c1310 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5918009870 tx=0x7f591801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f3ab7bf7331a9b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5929600640 1 -- 172.21.15.98:0/2627936540 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59180b0070 con 0x7f592c0cc8f0 2024-05-27T00:22:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5929600640 1 -- 172.21.15.98:0/2627936540 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59180ab600 con 0x7f592c0cc8f0 2024-05-27T00:22:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5929600640 1 -- 172.21.15.98:0/2627936540 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59180ab9d0 con 0x7f592c0cc8f0 2024-05-27T00:22:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 -- 172.21.15.98:0/2627936540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f592c0cc8f0 msgr2=0x7f592c0c1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 --2- 172.21.15.98:0/2627936540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f592c0cc8f0 0x7f592c0c1310 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5918009870 tx=0x7f591801c4d0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 -- 172.21.15.98:0/2627936540 shutdown_connections 2024-05-27T00:22:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 --2- 172.21.15.98:0/2627936540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f592c0cc8f0 0x7f592c0c1310 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 --2- 172.21.15.98:0/2627936540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f592c1511d0 0x7f592c070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 -- 172.21.15.98:0/2627936540 >> 172.21.15.98:0/2627936540 conn(0x7f592c067530 msgr2=0x7f592c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 -- 172.21.15.98:0/2627936540 shutdown_connections 2024-05-27T00:22:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 -- 172.21.15.98:0/2627936540 wait complete. 2024-05-27T00:22:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 Processor -- start 2024-05-27T00:22:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 -- start start 2024-05-27T00:22:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f592c0c8670 0x7f592c153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f592c0cc8f0 0x7f592c1540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f592c1511d0 0x7f592c1f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f592c0c2150 con 0x7f592c1511d0 2024-05-27T00:22:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f592c0c8e90 con 0x7f592c0c8670 2024-05-27T00:22:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.209+0000 7f5931a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f592c0c22d0 con 0x7f592c0cc8f0 2024-05-27T00:22:47.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f592c0c8670 0x7f592c153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:47.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f592c1511d0 0x7f592c1f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:47.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f592c0c8670 0x7f592c153440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:46008/0 (socket says 172.21.15.98:46008) 2024-05-27T00:22:47.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f592c0cc8f0 0x7f592c1540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:47.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592aa00640 1 -- 172.21.15.98:0/3609320851 learned_addr learned my addr 172.21.15.98:0/3609320851 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:47.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592b400640 1 -- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f592c0cc8f0 msgr2=0x7f592c1540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592b400640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f592c0cc8f0 0x7f592c1540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592b400640 1 -- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f592c0c8670 msgr2=0x7f592c153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592b400640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f592c0c8670 0x7f592c153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592b400640 1 -- 172.21.15.98:0/3609320851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592c1f2640 con 0x7f592c1511d0 2024-05-27T00:22:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f592b400640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f592c1511d0 0x7f592c1f4df0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f592400c730 tx=0x7f592400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:47.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f5922a00640 1 -- 172.21.15.98:0/3609320851 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5924003b80 con 0x7f592c1511d0 2024-05-27T00:22:47.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f5922a00640 1 -- 172.21.15.98:0/3609320851 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5924003d20 con 0x7f592c1511d0 2024-05-27T00:22:47.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f5931a00640 1 -- 172.21.15.98:0/3609320851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f592c151a30 con 0x7f592c1511d0 2024-05-27T00:22:47.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f5922a00640 1 -- 172.21.15.98:0/3609320851 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592400aa30 con 0x7f592c1511d0 2024-05-27T00:22:47.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f5931a00640 1 -- 172.21.15.98:0/3609320851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f592c151ff0 con 0x7f592c1511d0 2024-05-27T00:22:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f5931a00640 1 -- 172.21.15.98:0/3609320851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f592c075cb0 con 0x7f592c1511d0 2024-05-27T00:22:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.213+0000 7f5922a00640 1 -- 172.21.15.98:0/3609320851 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f592400abd0 con 0x7f592c1511d0 2024-05-27T00:22:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.217+0000 7f5922a00640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f590c0678c0 0x7f590c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.217+0000 7f5922a00640 1 -- 172.21.15.98:0/3609320851 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f5924085d20 con 0x7f592c1511d0 2024-05-27T00:22:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.217+0000 7f592aa00640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f590c0678c0 0x7f590c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.217+0000 7f592aa00640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f590c0678c0 0x7f590c069d80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f591c007a80 tx=0x7f591c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:47.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.221+0000 7f5922a00640 1 -- 172.21.15.98:0/3609320851 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f59240565a0 con 0x7f592c1511d0 2024-05-27T00:22:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 -- 172.21.15.98:0/3609320851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f592c0713c0 con 0x7f592c1511d0 2024-05-27T00:22:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5922a00640 1 -- 172.21.15.98:0/3609320851 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6013 (secure 0 0 0) 0x7f5924055ea0 con 0x7f592c1511d0 2024-05-27T00:22:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:47.411 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:42.641140+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 33 2024-05-27T00:22:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 -- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f590c0678c0 msgr2=0x7f590c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f590c0678c0 0x7f590c069d80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f591c007a80 tx=0x7f591c008040 comp rx=0 tx=0).stop 2024-05-27T00:22:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 -- 172.21.15.98:0/3609320851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f592c1511d0 msgr2=0x7f592c1f4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f592c1511d0 0x7f592c1f4df0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f592400c730 tx=0x7f592400cbf0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 -- 172.21.15.98:0/3609320851 shutdown_connections 2024-05-27T00:22:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f592c1511d0 0x7f592c1f4df0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f592c0cc8f0 0x7f592c1540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f590c0678c0 0x7f590c069d80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 --2- 172.21.15.98:0/3609320851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f592c0c8670 0x7f592c153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:47.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 -- 172.21.15.98:0/3609320851 >> 172.21.15.98:0/3609320851 conn(0x7f592c067530 msgr2=0x7f592c072620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:47.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 -- 172.21.15.98:0/3609320851 shutdown_connections 2024-05-27T00:22:47.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:47.409+0000 7f5931a00640 1 -- 172.21.15.98:0/3609320851 wait complete. 2024-05-27T00:22:47.432 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:22:47.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:22:47.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:22:49.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:49.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.617+0000 7f1b57800640 1 Processor -- start 2024-05-27T00:22:49.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.617+0000 7f1b57800640 1 -- start start 2024-05-27T00:22:49.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b57800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1b50152520 0x7f1b500c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:49.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b57800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b500c70e0 0x7f1b5006b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:49.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b57800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b5006cc70 con 0x7f1b50073760 2024-05-27T00:22:49.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b57800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1b50072fa0 con 0x7f1b500c70e0 2024-05-27T00:22:49.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b57800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f1b50152900 con 0x7f1b50152520 2024-05-27T00:22:49.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b55800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b500c70e0 0x7f1b5006b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:49.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b4fe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1b50073760 0x7f1b50071e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33792/0 (socket says 172.21.15.98:33792) 2024-05-27T00:22:49.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b54e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1b50152520 0x7f1b500c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:49.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b4fe00640 1 -- 172.21.15.98:0/187319704 learned_addr learned my addr 172.21.15.98:0/187319704 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:49.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b55800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b500c70e0 0x7f1b5006b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42650/0 (socket says 172.21.15.98:42650) 2024-05-27T00:22:49.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b55800640 1 -- 172.21.15.98:0/187319704 >> v2:172.21.15.204:3300/0 conn(0x7f1b50152520 msgr2=0x7f1b500c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:49.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b55800640 1 --2- 172.21.15.98:0/187319704 >> v2:172.21.15.204:3300/0 conn(0x7f1b50152520 0x7f1b500c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b55800640 1 -- 172.21.15.98:0/187319704 >> v1:172.21.15.98:6789/0 conn(0x7f1b50073760 legacy=0x7f1b50071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:49.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.621+0000 7f1b55800640 1 -- 172.21.15.98:0/187319704 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5006c6b0 con 0x7f1b500c70e0 2024-05-27T00:22:49.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b55800640 1 --2- 172.21.15.98:0/187319704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b500c70e0 0x7f1b5006b720 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1b500ca960 tx=0x7f1b4801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4813256121aac191 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:49.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b4f400640 1 -- 172.21.15.98:0/187319704 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b480b2070 con 0x7f1b500c70e0 2024-05-27T00:22:49.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b4f400640 1 -- 172.21.15.98:0/187319704 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b480a58a0 con 0x7f1b500c70e0 2024-05-27T00:22:49.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b4f400640 1 -- 172.21.15.98:0/187319704 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b480a5c70 con 0x7f1b500c70e0 2024-05-27T00:22:49.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b57800640 1 -- 172.21.15.98:0/187319704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b500c70e0 msgr2=0x7f1b5006b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:49.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b57800640 1 --2- 172.21.15.98:0/187319704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b500c70e0 0x7f1b5006b720 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1b500ca960 tx=0x7f1b4801c8f0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b57800640 1 -- 172.21.15.98:0/187319704 shutdown_connections 2024-05-27T00:22:49.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b57800640 1 --2- 172.21.15.98:0/187319704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b500c70e0 0x7f1b5006b720 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b57800640 1 --2- 172.21.15.98:0/187319704 >> v2:172.21.15.204:3300/0 conn(0x7f1b50152520 0x7f1b500c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b57800640 1 -- 172.21.15.98:0/187319704 >> 172.21.15.98:0/187319704 conn(0x7f1b500c9b70 msgr2=0x7f1b500c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:49.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b57800640 1 -- 172.21.15.98:0/187319704 shutdown_connections 2024-05-27T00:22:49.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.625+0000 7f1b57800640 1 -- 172.21.15.98:0/187319704 wait complete. 2024-05-27T00:22:49.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b57800640 1 Processor -- start 2024-05-27T00:22:49.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b57800640 1 -- start start 2024-05-27T00:22:49.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b57800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b50073760 0x7f1b50151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:49.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b57800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b500c70e0 0x7f1b5014d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:49.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b57800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b50152520 0x7f1b501f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:49.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b57800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b50165a30 con 0x7f1b50073760 2024-05-27T00:22:49.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b57800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1b501658b0 con 0x7f1b50152520 2024-05-27T00:22:49.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b57800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1b50165bb0 con 0x7f1b500c70e0 2024-05-27T00:22:49.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b54e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b50073760 0x7f1b50151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:49.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b54e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b50073760 0x7f1b50151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42654/0 (socket says 172.21.15.98:42654) 2024-05-27T00:22:49.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b54e00640 1 -- 172.21.15.98:0/3733067545 learned_addr learned my addr 172.21.15.98:0/3733067545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:49.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b55800640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b50152520 0x7f1b501f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:49.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b4fe00640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b500c70e0 0x7f1b5014d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:49.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b54e00640 1 -- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b500c70e0 msgr2=0x7f1b5014d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:49.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b54e00640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b500c70e0 0x7f1b5014d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b54e00640 1 -- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b50152520 msgr2=0x7f1b501f7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:49.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b54e00640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b50152520 0x7f1b501f7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b54e00640 1 -- 172.21.15.98:0/3733067545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b501505d0 con 0x7f1b50073760 2024-05-27T00:22:49.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b4fe00640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b500c70e0 0x7f1b5014d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:49.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b54e00640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b50073760 0x7f1b50151280 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400e930 tx=0x7f1b4400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:49.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b4cc00640 1 -- 172.21.15.98:0/3733067545 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44017070 con 0x7f1b50073760 2024-05-27T00:22:49.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b4cc00640 1 -- 172.21.15.98:0/3733067545 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b4400c9a0 con 0x7f1b50073760 2024-05-27T00:22:49.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b57800640 1 -- 172.21.15.98:0/3733067545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5014cbf0 con 0x7f1b50073760 2024-05-27T00:22:49.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b4cc00640 1 -- 172.21.15.98:0/3733067545 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4400cd50 con 0x7f1b50073760 2024-05-27T00:22:49.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.629+0000 7f1b57800640 1 -- 172.21.15.98:0/3733067545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b50151950 con 0x7f1b50073760 2024-05-27T00:22:49.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.633+0000 7f1b57800640 1 -- 172.21.15.98:0/3733067545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b50075cb0 con 0x7f1b50073760 2024-05-27T00:22:49.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.633+0000 7f1b4cc00640 1 -- 172.21.15.98:0/3733067545 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1b4401a070 con 0x7f1b50073760 2024-05-27T00:22:49.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.633+0000 7f1b4cc00640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b300678c0 0x7f1b30069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:49.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.633+0000 7f1b4cc00640 1 -- 172.21.15.98:0/3733067545 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f1b44085de0 con 0x7f1b50073760 2024-05-27T00:22:49.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.637+0000 7f1b4fe00640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b300678c0 0x7f1b30069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:49.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.637+0000 7f1b4fe00640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b300678c0 0x7f1b30069d80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c006fd0 tx=0x7f1b3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:49.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.641+0000 7f1b4cc00640 1 -- 172.21.15.98:0/3733067545 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1b44056170 con 0x7f1b50073760 2024-05-27T00:22:49.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.897+0000 7f1b57800640 1 -- 172.21.15.98:0/3733067545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b500c6cb0 con 0x7f1b50073760 2024-05-27T00:22:49.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.897+0000 7f1b4cc00640 1 -- 172.21.15.98:0/3733067545 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6013 (secure 0 0 0) 0x7f1b4405a470 con 0x7f1b50073760 2024-05-27T00:22:49.899 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:49.899 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:42.641140+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:22:49.900 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 33 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 -- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b300678c0 msgr2=0x7f1b30069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b300678c0 0x7f1b30069d80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c006fd0 tx=0x7f1b3c008040 comp rx=0 tx=0).stop 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 -- 172.21.15.98:0/3733067545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b50073760 msgr2=0x7f1b50151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b50073760 0x7f1b50151280 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1b4400e930 tx=0x7f1b4400edf0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 -- 172.21.15.98:0/3733067545 shutdown_connections 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b300678c0 0x7f1b30069d80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b50152520 0x7f1b501f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b500c70e0 0x7f1b5014d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 --2- 172.21.15.98:0/3733067545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b50073760 0x7f1b50151280 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 -- 172.21.15.98:0/3733067545 >> 172.21.15.98:0/3733067545 conn(0x7f1b500c9b70 msgr2=0x7f1b500c9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 -- 172.21.15.98:0/3733067545 shutdown_connections 2024-05-27T00:22:49.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:49.901+0000 7f1b57800640 1 -- 172.21.15.98:0/3733067545 wait complete. 2024-05-27T00:22:49.928 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:22:51.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.073+0000 7f975b200640 1 Processor -- start 2024-05-27T00:22:52.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.073+0000 7f975b200640 1 -- start start 2024-05-27T00:22:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.073+0000 7f975b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97540735d0 0x7f9754151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.073+0000 7f975b200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f97540c79f0 0x7f9754151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f975b200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97540c17d0 con 0x7f97540731f0 2024-05-27T00:22:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f975b200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9754071990 con 0x7f97540c79f0 2024-05-27T00:22:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f975b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9754070e10 con 0x7f97540735d0 2024-05-27T00:22:52.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f97540731f0 0x7f97540c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33804/0 (socket says 172.21.15.98:33804) 2024-05-27T00:22:52.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753e00640 1 -- 172.21.15.98:0/1105216750 learned_addr learned my addr 172.21.15.98:0/1105216750 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f974b400640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97540735d0 0x7f9754151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9759200640 1 --2- 172.21.15.98:0/1105216750 >> v2:172.21.15.204:3300/0 conn(0x7f97540c79f0 0x7f9754151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2384318604 0 0) 0x7f97540c17d0 con 0x7f97540731f0 2024-05-27T00:22:52.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9738003530 con 0x7f97540731f0 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2356227241 0 0) 0x7f9738003530 con 0x7f97540731f0 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97540735d0 msgr2=0x7f9754151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97540735d0 0x7f9754151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 >> v2:172.21.15.204:3300/0 conn(0x7f97540c79f0 msgr2=0x7f9754151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 --2- 172.21.15.98:0/1105216750 >> v2:172.21.15.204:3300/0 conn(0x7f97540c79f0 0x7f9754151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97540710f0 con 0x7f97540731f0 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f9744001a00 con 0x7f97540731f0 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 >> v1:172.21.15.98:6789/0 conn(0x7f97540731f0 legacy=0x7f97540c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380039c0 0x7f9738003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9738004810 0x7f9738004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97540735d0 0x7f97380051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9754071990 con 0x7f97380039c0 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9754070e10 con 0x7f9738004810 2024-05-27T00:22:52.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f973800b9d0 con 0x7f97540735d0 2024-05-27T00:22:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753e00640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380039c0 0x7f9738003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f974b400640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9738004810 0x7f9738004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9759200640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97540735d0 0x7f97380051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753e00640 1 -- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97540735d0 msgr2=0x7f97380051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753e00640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97540735d0 0x7f97380051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753e00640 1 -- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9738004810 msgr2=0x7f9738004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753e00640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9738004810 0x7f9738004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.077+0000 7f9753e00640 1 -- 172.21.15.98:0/1105216750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9738003530 con 0x7f97380039c0 2024-05-27T00:22:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f9753e00640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380039c0 0x7f9738003de0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f974400a920 tx=0x7f974401fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f659e6d5399517 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974400a4c0 con 0x7f97380039c0 2024-05-27T00:22:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97440aa660 con 0x7f97380039c0 2024-05-27T00:22:52.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f9753400640 1 -- 172.21.15.98:0/1105216750 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97440aaa50 con 0x7f97380039c0 2024-05-27T00:22:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f975b200640 1 -- 172.21.15.98:0/1105216750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380039c0 msgr2=0x7f9738003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f975b200640 1 --2- 172.21.15.98:0/1105216750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380039c0 0x7f9738003de0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f974400a920 tx=0x7f974401fba0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f9759200640 1 -- 172.21.15.98:0/1105216750 reap_dead start 2024-05-27T00:22:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f975b200640 1 -- 172.21.15.98:0/1105216750 shutdown_connections 2024-05-27T00:22:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f975b200640 1 -- 172.21.15.98:0/1105216750 >> 172.21.15.98:0/1105216750 conn(0x7f9754067530 msgr2=0x7f975406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f975b200640 1 -- 172.21.15.98:0/1105216750 shutdown_connections 2024-05-27T00:22:52.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f975b200640 1 -- 172.21.15.98:0/1105216750 wait complete. 2024-05-27T00:22:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f975b200640 1 Processor -- start 2024-05-27T00:22:52.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.081+0000 7f975b200640 1 -- start start 2024-05-27T00:22:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f975b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f975413b980 0x7f97541f0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f975b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f975413bd60 0x7f9754156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f975b200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9754140240 0x7f975414c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f975b200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97540c8400 con 0x7f9754140240 2024-05-27T00:22:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f975b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f97540716b0 con 0x7f975413b980 2024-05-27T00:22:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f975b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9754151430 con 0x7f975413bd60 2024-05-27T00:22:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9759200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9754140240 0x7f975414c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9759200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9754140240 0x7f975414c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42678/0 (socket says 172.21.15.98:42678) 2024-05-27T00:22:52.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9759200640 1 -- 172.21.15.98:0/455871113 learned_addr learned my addr 172.21.15.98:0/455871113 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9753e00640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f975413b980 0x7f97541f0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f974b400640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f975413bd60 0x7f9754156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9759200640 1 -- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f975413bd60 msgr2=0x7f9754156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9759200640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f975413bd60 0x7f9754156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9759200640 1 -- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f975413b980 msgr2=0x7f97541f0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9759200640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f975413b980 0x7f97541f0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9759200640 1 -- 172.21.15.98:0/455871113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9754156eb0 con 0x7f9754140240 2024-05-27T00:22:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9759200640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9754140240 0x7f975414c020 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f97540c81f0 tx=0x7f974c011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9750c00640 1 -- 172.21.15.98:0/455871113 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c019020 con 0x7f9754140240 2024-05-27T00:22:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9750c00640 1 -- 172.21.15.98:0/455871113 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f974c003620 con 0x7f9754140240 2024-05-27T00:22:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f975b200640 1 -- 172.21.15.98:0/455871113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f975414c560 con 0x7f9754140240 2024-05-27T00:22:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9750c00640 1 -- 172.21.15.98:0/455871113 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c003a10 con 0x7f9754140240 2024-05-27T00:22:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f975b200640 1 -- 172.21.15.98:0/455871113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f975414d9b0 con 0x7f9754140240 2024-05-27T00:22:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f975b200640 1 -- 172.21.15.98:0/455871113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9754075cb0 con 0x7f9754140240 2024-05-27T00:22:52.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.085+0000 7f9750c00640 1 -- 172.21.15.98:0/455871113 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f974c01c020 con 0x7f9754140240 2024-05-27T00:22:52.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.089+0000 7f9750c00640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97300678f0 0x7f9730069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.089+0000 7f9750c00640 1 -- 172.21.15.98:0/455871113 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f974c0860f0 con 0x7f9754140240 2024-05-27T00:22:52.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.089+0000 7f9753e00640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97300678f0 0x7f9730069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.089+0000 7f9753e00640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97300678f0 0x7f9730069db0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f97440b0000 tx=0x7f97440b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:52.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.093+0000 7f9750c00640 1 -- 172.21.15.98:0/455871113 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f974c023030 con 0x7f9754140240 2024-05-27T00:22:52.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.281+0000 7f975b200640 1 -- 172.21.15.98:0/455871113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9754158510 con 0x7f9754140240 2024-05-27T00:22:52.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.281+0000 7f9750c00640 1 -- 172.21.15.98:0/455871113 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6014 (secure 0 0 0) 0x7f974c0122d0 con 0x7f9754140240 2024-05-27T00:22:52.284 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:52.284 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:50.518361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5343,"qdb_cluster":[5343]},"id":1}]} 2024-05-27T00:22:52.284 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 34 2024-05-27T00:22:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 -- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97300678f0 msgr2=0x7f9730069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97300678f0 0x7f9730069db0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f97440b0000 tx=0x7f97440b5040 comp rx=0 tx=0).stop 2024-05-27T00:22:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 -- 172.21.15.98:0/455871113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9754140240 msgr2=0x7f975414c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9754140240 0x7f975414c020 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f97540c81f0 tx=0x7f974c011920 comp rx=0 tx=0).stop 2024-05-27T00:22:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 -- 172.21.15.98:0/455871113 shutdown_connections 2024-05-27T00:22:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9754140240 0x7f975414c020 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f975413bd60 0x7f9754156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97300678f0 0x7f9730069db0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 --2- 172.21.15.98:0/455871113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f975413b980 0x7f97541f0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 -- 172.21.15.98:0/455871113 >> 172.21.15.98:0/455871113 conn(0x7f9754067530 msgr2=0x7f97540c71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:52.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 -- 172.21.15.98:0/455871113 shutdown_connections 2024-05-27T00:22:52.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.285+0000 7f975b200640 1 -- 172.21.15.98:0/455871113 wait complete. 2024-05-27T00:22:52.305 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5202 2024-05-27T00:22:52.305 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2024-05-27T00:22:52.305 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-05-27T00:22:52.306 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-27T00:22:52.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-27T00:22:52.309 INFO:tasks.ceph.mds.c:Started 2024-05-27T00:22:52.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:22:52.394 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2024-05-27T00:22:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.445+0000 7f6bd1e00640 1 Processor -- start 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.445+0000 7f6bd1e00640 1 -- start start 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bd1e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc0cc1f0 0x7f6bcc1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bd1e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f6bcc073570 0x7f6bcc1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bd1e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bcc0c1f60 con 0x7f6bcc0cbe10 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bd1e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f6bcc159810 con 0x7f6bcc073570 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bd1e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6bcc0c6dd0 con 0x7f6bcc0cc1f0 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bcaa00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6bcc0cbe10 0x7f6bcc0c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33818/0 (socket says 172.21.15.98:33818) 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bcaa00640 1 -- 172.21.15.98:0/2918495004 learned_addr learned my addr 172.21.15.98:0/2918495004 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bc2000640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc0cc1f0 0x7f6bcc1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bcb400640 1 --2- 172.21.15.98:0/2918495004 >> v2:172.21.15.204:3300/0 conn(0x7f6bcc073570 0x7f6bcc1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2888892337 0 0) 0x7f6bcc0c1f60 con 0x7f6bcc0cbe10 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bb0003530 con 0x7f6bcc0cbe10 2024-05-27T00:22:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3674413725 0 0) 0x7f6bb0003530 con 0x7f6bcc0cbe10 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc0cc1f0 msgr2=0x7f6bcc1592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc0cc1f0 0x7f6bcc1592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 >> v2:172.21.15.204:3300/0 conn(0x7f6bcc073570 msgr2=0x7f6bcc1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 --2- 172.21.15.98:0/2918495004 >> v2:172.21.15.204:3300/0 conn(0x7f6bcc073570 0x7f6bcc1559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bcc0c70b0 con 0x7f6bcc0cbe10 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f6bbc001680 con 0x7f6bcc0cbe10 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 >> v1:172.21.15.98:6789/0 conn(0x7f6bcc0cbe10 legacy=0x7f6bcc0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 0x7f6bb0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bb0004810 0x7f6bb0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc0cc1f0 0x7f6bb00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc159810 con 0x7f6bb00039c0 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc0c6dd0 con 0x7f6bb0004810 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6bb000ba00 con 0x7f6bcc0cc1f0 2024-05-27T00:22:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bcaa00640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 0x7f6bb0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bc2000640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bb0004810 0x7f6bb0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bcb400640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc0cc1f0 0x7f6bb00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bcaa00640 1 -- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc0cc1f0 msgr2=0x7f6bb00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bcaa00640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc0cc1f0 0x7f6bb00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bcaa00640 1 -- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bb0004810 msgr2=0x7f6bb0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:22:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bcaa00640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bb0004810 0x7f6bb0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bcaa00640 1 -- 172.21.15.98:0/2918495004 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb0003530 con 0x7f6bb00039c0 2024-05-27T00:22:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.449+0000 7f6bc2000640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bb0004810 0x7f6bb0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:22:52.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bcaa00640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 0x7f6bb0003de0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00c2c0 tx=0x7f6bbc01f4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae4e19245994437 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:52.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc007d60 con 0x7f6bb00039c0 2024-05-27T00:22:52.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bbc005b00 con 0x7f6bb00039c0 2024-05-27T00:22:52.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bca000640 1 -- 172.21.15.98:0/2918495004 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc007070 con 0x7f6bb00039c0 2024-05-27T00:22:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2918495004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 msgr2=0x7f6bb0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bd1e00640 1 --2- 172.21.15.98:0/2918495004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 0x7f6bb0003de0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00c2c0 tx=0x7f6bbc01f4b0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bcb400640 1 -- 172.21.15.98:0/2918495004 reap_dead start 2024-05-27T00:22:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2918495004 shutdown_connections 2024-05-27T00:22:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2918495004 >> 172.21.15.98:0/2918495004 conn(0x7f6bcc067530 msgr2=0x7f6bcc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2918495004 shutdown_connections 2024-05-27T00:22:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2918495004 wait complete. 2024-05-27T00:22:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bd1e00640 1 Processor -- start 2024-05-27T00:22:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.453+0000 7f6bd1e00640 1 -- start start 2024-05-27T00:22:52.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bd1e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 0x7f6bcc153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bd1e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc15c840 0x7f6bcc157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bd1e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bcc14fa70 0x7f6bcc157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bd1e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc0c1330 con 0x7f6bb00039c0 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bd1e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc0c1050 con 0x7f6bcc14fa70 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bd1e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6bcc0c79d0 con 0x7f6bcc15c840 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bcaa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 0x7f6bcc153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bcaa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 0x7f6bcc153f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42696/0 (socket says 172.21.15.98:42696) 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bcaa00640 1 -- 172.21.15.98:0/2670482414 learned_addr learned my addr 172.21.15.98:0/2670482414 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bc2000640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc15c840 0x7f6bcc157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bcb400640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bcc14fa70 0x7f6bcc157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bcaa00640 1 -- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc15c840 msgr2=0x7f6bcc157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bcaa00640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc15c840 0x7f6bcc157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bcaa00640 1 -- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bcc14fa70 msgr2=0x7f6bcc157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bcaa00640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bcc14fa70 0x7f6bcc157db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bcaa00640 1 -- 172.21.15.98:0/2670482414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bcc1568a0 con 0x7f6bb00039c0 2024-05-27T00:22:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bcaa00640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 0x7f6bcc153f00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00c2c0 tx=0x7f6bb0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bc3e00640 1 -- 172.21.15.98:0/2670482414 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc005cb0 con 0x7f6bb00039c0 2024-05-27T00:22:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bc3e00640 1 -- 172.21.15.98:0/2670482414 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bbc005e50 con 0x7f6bb00039c0 2024-05-27T00:22:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2670482414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bcc1ee080 con 0x7f6bb00039c0 2024-05-27T00:22:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bc3e00640 1 -- 172.21.15.98:0/2670482414 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc0049f0 con 0x7f6bb00039c0 2024-05-27T00:22:52.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2670482414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bcc1ee640 con 0x7f6bb00039c0 2024-05-27T00:22:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2670482414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bcc075cb0 con 0x7f6bb00039c0 2024-05-27T00:22:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.457+0000 7f6bc3e00640 1 -- 172.21.15.98:0/2670482414 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6bbc0ba030 con 0x7f6bb00039c0 2024-05-27T00:22:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.461+0000 7f6bc3e00640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6ba80678f0 0x7f6ba8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:22:52.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.461+0000 7f6bc3e00640 1 -- 172.21.15.98:0/2670482414 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f6bbc122a60 con 0x7f6bb00039c0 2024-05-27T00:22:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.461+0000 7f6bc2000640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6ba80678f0 0x7f6ba8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:22:52.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.461+0000 7f6bc2000640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6ba80678f0 0x7f6ba8069db0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0c8720 tx=0x7f6bb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:22:52.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.465+0000 7f6bc3e00640 1 -- 172.21.15.98:0/2670482414 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f6bbc0191c0 con 0x7f6bb00039c0 2024-05-27T00:22:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2670482414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bcc1574d0 con 0x7f6bb00039c0 2024-05-27T00:22:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bc3e00640 1 -- 172.21.15.98:0/2670482414 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6836 (secure 0 0 0) 0x7f6bbc0f3250 con 0x7f6bb00039c0 2024-05-27T00:22:52.662 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:22:52.662 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:50.518361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5343,"qdb_cluster":[5343]},"id":1}]} 2024-05-27T00:22:52.662 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-27T00:22:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6ba80678f0 msgr2=0x7f6ba8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6ba80678f0 0x7f6ba8069db0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0c8720 tx=0x7f6bb4008040 comp rx=0 tx=0).stop 2024-05-27T00:22:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2670482414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 msgr2=0x7f6bcc153f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:22:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 0x7f6bcc153f00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00c2c0 tx=0x7f6bb0005840 comp rx=0 tx=0).stop 2024-05-27T00:22:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2670482414 shutdown_connections 2024-05-27T00:22:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6ba80678f0 0x7f6ba8069db0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bcc14fa70 0x7f6bcc157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bcc15c840 0x7f6bcc157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 --2- 172.21.15.98:0/2670482414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bb00039c0 0x7f6bcc153f00 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:22:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2670482414 >> 172.21.15.98:0/2670482414 conn(0x7f6bcc067530 msgr2=0x7f6bcc0c5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:22:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2670482414 shutdown_connections 2024-05-27T00:22:52.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:22:52.661+0000 7f6bd1e00640 1 -- 172.21.15.98:0/2670482414 wait complete. 2024-05-27T00:22:52.682 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-05-27T00:22:52.682 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 30 secs before thrashing 2024-05-27T00:22:58.353 INFO:tasks.ceph.mds.a.smithi098.stderr:2024-05-27T00:22:58.353+0000 7f1c1b200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:23:00.692 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 87% (199/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 94% (215/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2024-05-27T00:23:00.880 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at a4c752df713 v0.6 2024-05-27T00:23:00.883 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.6 2024-05-27T00:23:00.897 INFO:tasks.workunit.client.0.smithi098.stdout:v0.7 2024-05-27T00:23:00.898 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 7 -eq 48 ] 2024-05-27T00:23:00.898 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.7 2024-05-27T00:23:00.898 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.7 2024-05-27T00:23:00.898 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.7 2024-05-27T00:23:01.932 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2024-05-27T00:23:01.937 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.7 2024-05-27T00:23:02.126 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 8 -eq 48 ] 2024-05-27T00:23:02.126 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.8 2024-05-27T00:23:02.126 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.8 2024-05-27T00:23:02.127 INFO:tasks.workunit.client.0.smithi098.stdout:v0.8 2024-05-27T00:23:02.127 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.8 2024-05-27T00:23:03.019 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2024-05-27T00:23:03.024 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.8 2024-05-27T00:23:03.140 INFO:tasks.workunit.client.0.smithi098.stdout:v0.9 2024-05-27T00:23:03.141 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 9 -eq 48 ] 2024-05-27T00:23:03.141 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.9 2024-05-27T00:23:03.141 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.9 2024-05-27T00:23:03.141 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.9 2024-05-27T00:23:03.863 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2024-05-27T00:23:03.868 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.9 2024-05-27T00:23:03.966 INFO:tasks.workunit.client.0.smithi098.stdout:v0.10 2024-05-27T00:23:03.967 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 10 -eq 48 ] 2024-05-27T00:23:03.967 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.10 2024-05-27T00:23:03.967 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.10 2024-05-27T00:23:03.967 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.10 2024-05-27T00:23:12.302 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 14% (24/164) Updating files: 15% (25/164) Updating files: 16% (27/164) Updating files: 17% (28/164) Updating files: 18% (30/164) Updating files: 19% (32/164) Updating files: 20% (33/164) Updating files: 21% (35/164) Updating files: 22% (37/164) Updating files: 23% (38/164) Updating files: 24% (40/164) Updating files: 25% (41/164) Updating files: 26% (43/164) Updating files: 27% (45/164) Updating files: 28% (46/164) Updating files: 29% (48/164) Updating files: 30% (50/164) Updating files: 31% (51/164) Updating files: 32% (53/164) Updating files: 33% (55/164) Updating files: 34% (56/164) Updating files: 35% (58/164) Updating files: 36% (60/164) Updating files: 37% (61/164) Updating files: 38% (63/164) Updating files: 39% (64/164) Updating files: 40% (66/164) Updating files: 41% (68/164) Updating files: 42% (69/164) Updating files: 43% (71/164) Updating files: 44% (73/164) Updating files: 45% (74/164) Updating files: 46% (76/164) Updating files: 47% (78/164) Updating files: 48% (79/164) Updating files: 49% (81/164) Updating files: 50% (82/164) Updating files: 51% (84/164) Updating files: 52% (86/164) Updating files: 53% (87/164) Updating files: 54% (89/164) Updating files: 55% (91/164) Updating files: 56% (92/164) Updating files: 57% (94/164) Updating files: 58% (96/164) Updating files: 59% (97/164) Updating files: 60% (99/164) Updating files: 61% (101/164) Updating files: 62% (102/164) Updating files: 63% (104/164) Updating files: 64% (105/164) Updating files: 65% (107/164) Updating files: 66% (109/164) Updating files: 67% (110/164) Updating files: 68% (112/164) Updating files: 69% (114/164) Updating files: 70% (115/164) Updating files: 71% (117/164) Updating files: 72% (119/164) Updating files: 73% (120/164) Updating files: 74% (122/164) Updating files: 75% (123/164) Updating files: 76% (125/164) Updating files: 77% (127/164) Updating files: 78% (128/164) Updating files: 79% (130/164) Updating files: 80% (132/164) Updating files: 81% (133/164) Updating files: 82% (135/164) Updating files: 83% (137/164) Updating files: 84% (138/164) Updating files: 85% (140/164) Updating files: 86% (142/164) Updating files: 87% (143/164) Updating files: 88% (145/164) Updating files: 89% (146/164) Updating files: 90% (148/164) Updating files: 91% (150/164) Updating files: 92% (151/164) Updating files: 93% (153/164) Updating files: 94% (155/164) Updating files: 95% (156/164) Updating files: 96% (158/164) Updating files: 97% (160/164) Updating files: 98% (161/164) Updating files: 99% (163/164) Updating files: 100% (164/164) Updating files: 100% (164/164), done. 2024-05-27T00:23:12.580 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 007313fe844 v0.10 2024-05-27T00:23:12.586 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.10 2024-05-27T00:23:12.613 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 11 -eq 48 ] 2024-05-27T00:23:12.613 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.11 2024-05-27T00:23:12.614 INFO:tasks.workunit.client.0.smithi098.stdout:v0.11 2024-05-27T00:23:12.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.11 2024-05-27T00:23:12.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.11 2024-05-27T00:23:13.047 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2024-05-27T00:23:13.050 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.11 2024-05-27T00:23:13.160 INFO:tasks.workunit.client.0.smithi098.stdout:v0.12 2024-05-27T00:23:13.160 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 12 -eq 48 ] 2024-05-27T00:23:13.160 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.12 2024-05-27T00:23:13.160 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.12 2024-05-27T00:23:13.160 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.12 2024-05-27T00:23:13.582 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at f524150822a v0.12 2024-05-27T00:23:13.586 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.12 2024-05-27T00:23:13.710 INFO:tasks.workunit.client.0.smithi098.stdout:v0.13 2024-05-27T00:23:13.710 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 13 -eq 48 ] 2024-05-27T00:23:13.710 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.13 2024-05-27T00:23:13.711 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.13 2024-05-27T00:23:13.711 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.13 2024-05-27T00:23:14.313 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 2b32297267d v0.13 2024-05-27T00:23:14.318 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.13 2024-05-27T00:23:14.374 INFO:tasks.workunit.client.0.smithi098.stdout:v0.14 2024-05-27T00:23:14.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 14 -eq 48 ] 2024-05-27T00:23:14.374 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.14 2024-05-27T00:23:14.375 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.14 2024-05-27T00:23:14.375 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.14 2024-05-27T00:23:15.011 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2024-05-27T00:23:15.016 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.14 2024-05-27T00:23:15.131 INFO:tasks.workunit.client.0.smithi098.stdout:v0.15 2024-05-27T00:23:15.131 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 15 -eq 48 ] 2024-05-27T00:23:15.131 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.15 2024-05-27T00:23:15.131 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.15 2024-05-27T00:23:15.131 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.15 2024-05-27T00:23:15.670 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at ab3c421eb95 v0.15 2024-05-27T00:23:15.674 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.15 2024-05-27T00:23:15.770 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 16 -eq 48 ] 2024-05-27T00:23:15.770 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.16 2024-05-27T00:23:15.770 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.16 2024-05-27T00:23:15.771 INFO:tasks.workunit.client.0.smithi098.stdout:v0.16 2024-05-27T00:23:15.771 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.16 2024-05-27T00:23:16.267 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 38dbaa5175c v0.16 2024-05-27T00:23:16.271 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.16 2024-05-27T00:23:16.417 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 17 -eq 48 ] 2024-05-27T00:23:16.417 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.17 2024-05-27T00:23:16.417 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.17 2024-05-27T00:23:16.417 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.17 2024-05-27T00:23:16.417 INFO:tasks.workunit.client.0.smithi098.stdout:v0.17 2024-05-27T00:23:17.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:23:17.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:23:22.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:22.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789a00640 1 Processor -- start 2024-05-27T00:23:22.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789a00640 1 -- start start 2024-05-27T00:23:22.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8784154690 0x7f878415b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:22.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f87840ccda0 0x7f87840cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:22.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87840cb190 con 0x7f87840cc290 2024-05-27T00:23:22.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f878415b650 con 0x7f87840ccda0 2024-05-27T00:23:22.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789a00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f87840caf40 con 0x7f8784154690 2024-05-27T00:23:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f87840ccda0 0x7f87840cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8783400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f87840cc290 0x7f87840c4f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53998/0 (socket says 172.21.15.98:53998) 2024-05-27T00:23:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8783400640 1 -- 172.21.15.98:0/2322465493 learned_addr learned my addr 172.21.15.98:0/2322465493 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:23:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8783e00640 1 --2- 172.21.15.98:0/2322465493 >> v2:172.21.15.204:3300/0 conn(0x7f8784154690 0x7f878415b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789000640 1 -- 172.21.15.98:0/2322465493 >> v2:172.21.15.204:3300/0 conn(0x7f8784154690 msgr2=0x7f878415b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789000640 1 --2- 172.21.15.98:0/2322465493 >> v2:172.21.15.204:3300/0 conn(0x7f8784154690 0x7f878415b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789000640 1 -- 172.21.15.98:0/2322465493 >> v1:172.21.15.98:6789/0 conn(0x7f87840cc290 legacy=0x7f87840c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.877+0000 7f8789000640 1 -- 172.21.15.98:0/2322465493 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878415dd20 con 0x7f87840ccda0 2024-05-27T00:23:22.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.881+0000 7f8789000640 1 --2- 172.21.15.98:0/2322465493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87840ccda0 0x7f87840cd1b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f87840cb740 tx=0x7f877c01f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce93679724aad66 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:22.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.881+0000 7f8782a00640 1 -- 172.21.15.98:0/2322465493 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c0b4070 con 0x7f87840ccda0 2024-05-27T00:23:22.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.881+0000 7f8782a00640 1 -- 172.21.15.98:0/2322465493 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877c0af600 con 0x7f87840ccda0 2024-05-27T00:23:22.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.881+0000 7f8782a00640 1 -- 172.21.15.98:0/2322465493 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c0af9d0 con 0x7f87840ccda0 2024-05-27T00:23:22.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.881+0000 7f8789a00640 1 -- 172.21.15.98:0/2322465493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87840ccda0 msgr2=0x7f87840cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:22.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.881+0000 7f8789a00640 1 --2- 172.21.15.98:0/2322465493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87840ccda0 0x7f87840cd1b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f87840cb740 tx=0x7f877c01f110 comp rx=0 tx=0).stop 2024-05-27T00:23:22.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.881+0000 7f8789a00640 1 -- 172.21.15.98:0/2322465493 shutdown_connections 2024-05-27T00:23:22.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.881+0000 7f8789a00640 1 --2- 172.21.15.98:0/2322465493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87840ccda0 0x7f87840cd1b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:22.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.881+0000 7f8789a00640 1 --2- 172.21.15.98:0/2322465493 >> v2:172.21.15.204:3300/0 conn(0x7f8784154690 0x7f878415b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:22.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.881+0000 7f8789a00640 1 -- 172.21.15.98:0/2322465493 >> 172.21.15.98:0/2322465493 conn(0x7f878406c740 msgr2=0x7f878406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:22.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789a00640 1 -- 172.21.15.98:0/2322465493 shutdown_connections 2024-05-27T00:23:22.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789a00640 1 -- 172.21.15.98:0/2322465493 wait complete. 2024-05-27T00:23:22.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789a00640 1 Processor -- start 2024-05-27T00:23:22.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789a00640 1 -- start start 2024-05-27T00:23:22.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87840cc290 0x7f878415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:22.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87840ccda0 0x7f87841f0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:22.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784154690 0x7f87840c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:22.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f878415e5c0 con 0x7f8784154690 2024-05-27T00:23:22.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f878415e2e0 con 0x7f87840cc290 2024-05-27T00:23:22.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f87840cb7e0 con 0x7f87840ccda0 2024-05-27T00:23:22.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784154690 0x7f87840c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:22.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784154690 0x7f87840c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34306/0 (socket says 172.21.15.98:34306) 2024-05-27T00:23:22.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789000640 1 -- 172.21.15.98:0/1173894406 learned_addr learned my addr 172.21.15.98:0/1173894406 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:22.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8783e00640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87840cc290 0x7f878415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:22.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8783400640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87840ccda0 0x7f87841f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:22.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789000640 1 -- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87840ccda0 msgr2=0x7f87841f0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:22.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789000640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87840ccda0 0x7f87841f0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:22.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789000640 1 -- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87840cc290 msgr2=0x7f878415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:22.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789000640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87840cc290 0x7f878415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:22.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8789000640 1 -- 172.21.15.98:0/1173894406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87840c91c0 con 0x7f8784154690 2024-05-27T00:23:22.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.885+0000 7f8783400640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87840ccda0 0x7f87841f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:23:22.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.889+0000 7f8789000640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784154690 0x7f87840c9860 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f877c01bd20 tx=0x7f877c01beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:22.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.889+0000 7f877be00640 1 -- 172.21.15.98:0/1173894406 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c0b4040 con 0x7f8784154690 2024-05-27T00:23:22.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.889+0000 7f877be00640 1 -- 172.21.15.98:0/1173894406 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877c0c1070 con 0x7f8784154690 2024-05-27T00:23:22.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.889+0000 7f877be00640 1 -- 172.21.15.98:0/1173894406 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c009450 con 0x7f8784154690 2024-05-27T00:23:22.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.889+0000 7f8789a00640 1 -- 172.21.15.98:0/1173894406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87841f01d0 con 0x7f8784154690 2024-05-27T00:23:22.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.889+0000 7f8789a00640 1 -- 172.21.15.98:0/1173894406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87840c7ca0 con 0x7f8784154690 2024-05-27T00:23:22.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.889+0000 7f8789a00640 1 -- 172.21.15.98:0/1173894406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8784075c40 con 0x7f8784154690 2024-05-27T00:23:22.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.889+0000 7f877be00640 1 -- 172.21.15.98:0/1173894406 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f877c0ba6b0 con 0x7f8784154690 2024-05-27T00:23:22.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.893+0000 7f877be00640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87640678f0 0x7f8764069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:22.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.893+0000 7f877be00640 1 -- 172.21.15.98:0/1173894406 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f877c1249d0 con 0x7f8784154690 2024-05-27T00:23:22.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.893+0000 7f8783e00640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87640678f0 0x7f8764069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:22.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.893+0000 7f8783e00640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87640678f0 0x7f8764069db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8774009ac0 tx=0x7f8774009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:22.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:22.897+0000 7f877be00640 1 -- 172.21.15.98:0/1173894406 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f877c018c70 con 0x7f8784154690 2024-05-27T00:23:23.151 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2024-05-27T00:23:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.201+0000 7f8789a00640 1 -- 172.21.15.98:0/1173894406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87840cba80 con 0x7f8784154690 2024-05-27T00:23:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.201+0000 7f877be00640 1 -- 172.21.15.98:0/1173894406 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6836 (secure 0 0 0) 0x7f877c0f52e0 con 0x7f8784154690 2024-05-27T00:23:23.204 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:23.204 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:50.518361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5343,"qdb_cluster":[5343]},"id":1}]} 2024-05-27T00:23:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-27T00:23:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 -- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87640678f0 msgr2=0x7f8764069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87640678f0 0x7f8764069db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8774009ac0 tx=0x7f8774009190 comp rx=0 tx=0).stop 2024-05-27T00:23:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 -- 172.21.15.98:0/1173894406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784154690 msgr2=0x7f87840c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784154690 0x7f87840c9860 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f877c01bd20 tx=0x7f877c01beb0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 -- 172.21.15.98:0/1173894406 shutdown_connections 2024-05-27T00:23:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8784154690 0x7f87840c9860 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f87840ccda0 0x7f87841f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87640678f0 0x7f8764069db0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 --2- 172.21.15.98:0/1173894406 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87840cc290 0x7f878415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 -- 172.21.15.98:0/1173894406 >> 172.21.15.98:0/1173894406 conn(0x7f878406c740 msgr2=0x7f87840ce670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 -- 172.21.15.98:0/1173894406 shutdown_connections 2024-05-27T00:23:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.205+0000 7f8789a00640 1 -- 172.21.15.98:0/1173894406 wait complete. 2024-05-27T00:23:23.223 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at dac3a8c35fb v0.17 2024-05-27T00:23:23.226 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.17 2024-05-27T00:23:23.229 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2024-05-27T00:23:23.229 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-27T00:23:23.229 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:23:23.251 INFO:tasks.ceph.mds.a:Stopped 2024-05-27T00:23:23.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:23.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbed4a00640 1 Processor -- start 2024-05-27T00:23:23.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbed4a00640 1 -- start start 2024-05-27T00:23:23.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbed4a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbed00b3260 0x7fbed00c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:23.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbed4a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbed0070aa0 0x7fbed00c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:23.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbed4a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed0071010 con 0x7fbed0070660 2024-05-27T00:23:23.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbed4a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbed00c3890 con 0x7fbed0070aa0 2024-05-27T00:23:23.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbed4a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbed00c88c0 con 0x7fbed00b3260 2024-05-27T00:23:23.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbecfe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbed0070aa0 0x7fbed00c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:23.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbecfe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbed0070aa0 0x7fbed00c9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50622/0 (socket says 172.21.15.98:50622) 2024-05-27T00:23:23.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbecfe00640 1 -- 172.21.15.98:0/898315648 learned_addr learned my addr 172.21.15.98:0/898315648 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:23.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbecf400640 1 --2- 172.21.15.98:0/898315648 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbed00b3260 0x7fbed00c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:23.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbecfe00640 1 -- 172.21.15.98:0/898315648 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbed00b3260 msgr2=0x7fbed00c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:23.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbecfe00640 1 --2- 172.21.15.98:0/898315648 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbed00b3260 0x7fbed00c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbecfe00640 1 -- 172.21.15.98:0/898315648 >> v1:172.21.15.204:6789/0 conn(0x7fbed0070660 legacy=0x7fbed00c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:23.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.429+0000 7fbecfe00640 1 -- 172.21.15.98:0/898315648 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed00c8b80 con 0x7fbed0070aa0 2024-05-27T00:23:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbecfe00640 1 --2- 172.21.15.98:0/898315648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070aa0 0x7fbed00c9ef0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009e50 tx=0x7fbec801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc0c120128be474 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbece000640 1 -- 172.21.15.98:0/898315648 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec80b4070 con 0x7fbed0070aa0 2024-05-27T00:23:23.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbece000640 1 -- 172.21.15.98:0/898315648 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec80af400 con 0x7fbed0070aa0 2024-05-27T00:23:23.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbece000640 1 -- 172.21.15.98:0/898315648 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec80af7d0 con 0x7fbed0070aa0 2024-05-27T00:23:23.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbed4a00640 1 -- 172.21.15.98:0/898315648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070aa0 msgr2=0x7fbed00c9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:23.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbed4a00640 1 --2- 172.21.15.98:0/898315648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070aa0 0x7fbed00c9ef0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009e50 tx=0x7fbec801f210 comp rx=0 tx=0).stop 2024-05-27T00:23:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbed4a00640 1 -- 172.21.15.98:0/898315648 shutdown_connections 2024-05-27T00:23:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbed4a00640 1 --2- 172.21.15.98:0/898315648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070aa0 0x7fbed00c9ef0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbed4a00640 1 --2- 172.21.15.98:0/898315648 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbed00b3260 0x7fbed00c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbed4a00640 1 -- 172.21.15.98:0/898315648 >> 172.21.15.98:0/898315648 conn(0x7fbed00cdeb0 msgr2=0x7fbed00ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbed4a00640 1 -- 172.21.15.98:0/898315648 shutdown_connections 2024-05-27T00:23:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbed4a00640 1 -- 172.21.15.98:0/898315648 wait complete. 2024-05-27T00:23:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbed4a00640 1 Processor -- start 2024-05-27T00:23:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.433+0000 7fbed4a00640 1 -- start start 2024-05-27T00:23:23.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbed4a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070660 0x7fbed01570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:23.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbed4a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbed0070aa0 0x7fbed01f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:23.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbed4a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbed0156ac0 0x7fbed015db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:23.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbed4a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbed00c9420 con 0x7fbed0070660 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbed4a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbed00c9140 con 0x7fbed0070aa0 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbed4a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbed00c8200 con 0x7fbed0156ac0 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecf400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070660 0x7fbed01570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecf400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070660 0x7fbed01570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50630/0 (socket says 172.21.15.98:50630) 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecf400640 1 -- 172.21.15.98:0/1802666838 learned_addr learned my addr 172.21.15.98:0/1802666838 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecfe00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbed0156ac0 0x7fbed015db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecea00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbed0070aa0 0x7fbed01f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecf400640 1 -- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbed0156ac0 msgr2=0x7fbed015db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecf400640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbed0156ac0 0x7fbed015db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecf400640 1 -- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbed0070aa0 msgr2=0x7fbed01f0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecf400640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbed0070aa0 0x7fbed01f0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecf400640 1 -- 172.21.15.98:0/1802666838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed015caf0 con 0x7fbed0070660 2024-05-27T00:23:23.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecea00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbed0070aa0 0x7fbed01f0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:23:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbecf400640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070660 0x7fbed01570e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbec000d620 tx=0x7fbec000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbec7400640 1 -- 172.21.15.98:0/1802666838 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec000b620 con 0x7fbed0070660 2024-05-27T00:23:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbec7400640 1 -- 172.21.15.98:0/1802666838 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec0019070 con 0x7fbed0070660 2024-05-27T00:23:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbec7400640 1 -- 172.21.15.98:0/1802666838 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec000b9e0 con 0x7fbed0070660 2024-05-27T00:23:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbed4a00640 1 -- 172.21.15.98:0/1802666838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed015cdb0 con 0x7fbed0070660 2024-05-27T00:23:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbed4a00640 1 -- 172.21.15.98:0/1802666838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed01f03d0 con 0x7fbed0070660 2024-05-27T00:23:23.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.437+0000 7fbed4a00640 1 -- 172.21.15.98:0/1802666838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed0075c40 con 0x7fbed0070660 2024-05-27T00:23:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.441+0000 7fbec7400640 1 -- 172.21.15.98:0/1802666838 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbec000bb80 con 0x7fbed0070660 2024-05-27T00:23:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.441+0000 7fbec7400640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbeb00678f0 0x7fbeb0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.441+0000 7fbec7400640 1 -- 172.21.15.98:0/1802666838 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7fbec0086300 con 0x7fbed0070660 2024-05-27T00:23:23.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.441+0000 7fbecea00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbeb00678f0 0x7fbeb0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:23.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.441+0000 7fbecea00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbeb00678f0 0x7fbeb0069db0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbed0158460 tx=0x7fbebc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:23.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.445+0000 7fbec7400640 1 -- 172.21.15.98:0/1802666838 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fbec0056b80 con 0x7fbed0070660 2024-05-27T00:23:23.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 -- 172.21.15.98:0/1802666838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbed00cc950 con 0x7fbed0070660 2024-05-27T00:23:23.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbec7400640 1 -- 172.21.15.98:0/1802666838 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6836 (secure 0 0 0) 0x7fbec0056480 con 0x7fbed0070660 2024-05-27T00:23:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:23.636 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:50.518361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5343,"qdb_cluster":[5343]},"id":1}]} 2024-05-27T00:23:23.636 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-27T00:23:23.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 -- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbeb00678f0 msgr2=0x7fbeb0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:23.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbeb00678f0 0x7fbeb0069db0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbed0158460 tx=0x7fbebc009040 comp rx=0 tx=0).stop 2024-05-27T00:23:23.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 -- 172.21.15.98:0/1802666838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070660 msgr2=0x7fbed01570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:23.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070660 0x7fbed01570e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbec000d620 tx=0x7fbec000dae0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 -- 172.21.15.98:0/1802666838 shutdown_connections 2024-05-27T00:23:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbed0156ac0 0x7fbed015db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbeb00678f0 0x7fbeb0069db0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbed0070aa0 0x7fbed01f0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 --2- 172.21.15.98:0/1802666838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed0070660 0x7fbed01570e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.633+0000 7fbed4a00640 1 -- 172.21.15.98:0/1802666838 >> 172.21.15.98:0/1802666838 conn(0x7fbed00cdeb0 msgr2=0x7fbed00c6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.637+0000 7fbed4a00640 1 -- 172.21.15.98:0/1802666838 shutdown_connections 2024-05-27T00:23:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:23.637+0000 7fbed4a00640 1 -- 172.21.15.98:0/1802666838 wait complete. 2024-05-27T00:23:23.657 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:23:25.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:25.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1263e00640 1 Processor -- start 2024-05-27T00:23:25.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1263e00640 1 -- start start 2024-05-27T00:23:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1263e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f125c0cbe10 0x7f125c1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1263e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f125c0cc1f0 0x7f125c0c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1263e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125c0c6dd0 con 0x7f125c073570 2024-05-27T00:23:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1263e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f125c159810 con 0x7f125c0cbe10 2024-05-27T00:23:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1263e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f125c0c70b0 con 0x7f125c0cc1f0 2024-05-27T00:23:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1261400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f125c0cbe10 0x7f125c1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1261400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f125c0cbe10 0x7f125c1592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50642/0 (socket says 172.21.15.98:50642) 2024-05-27T00:23:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1261400640 1 -- 172.21.15.98:0/1988500097 learned_addr learned my addr 172.21.15.98:0/1988500097 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.801+0000 7f1260a00640 1 --2- 172.21.15.98:0/1988500097 >> v2:172.21.15.204:3300/0 conn(0x7f125c0cc1f0 0x7f125c0c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1261400640 1 -- 172.21.15.98:0/1988500097 >> v2:172.21.15.204:3300/0 conn(0x7f125c0cc1f0 msgr2=0x7f125c0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1261400640 1 --2- 172.21.15.98:0/1988500097 >> v2:172.21.15.204:3300/0 conn(0x7f125c0cc1f0 0x7f125c0c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1261400640 1 -- 172.21.15.98:0/1988500097 >> v1:172.21.15.204:6789/0 conn(0x7f125c073570 legacy=0x7f125c1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1261400640 1 -- 172.21.15.98:0/1988500097 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125c0c7390 con 0x7f125c0cbe10 2024-05-27T00:23:25.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1261400640 1 --2- 172.21.15.98:0/1988500097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c0cbe10 0x7f125c1592d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f125c0cc6d0 tx=0x7f125001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c522d937513d4999 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:25.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f125be00640 1 -- 172.21.15.98:0/1988500097 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12500b5070 con 0x7f125c0cbe10 2024-05-27T00:23:25.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f125be00640 1 -- 172.21.15.98:0/1988500097 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12500a8a10 con 0x7f125c0cbe10 2024-05-27T00:23:25.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f125be00640 1 -- 172.21.15.98:0/1988500097 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12500a8de0 con 0x7f125c0cbe10 2024-05-27T00:23:25.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1263e00640 1 -- 172.21.15.98:0/1988500097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c0cbe10 msgr2=0x7f125c1592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:25.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1263e00640 1 --2- 172.21.15.98:0/1988500097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c0cbe10 0x7f125c1592d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f125c0cc6d0 tx=0x7f125001ed10 comp rx=0 tx=0).stop 2024-05-27T00:23:25.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1263e00640 1 -- 172.21.15.98:0/1988500097 shutdown_connections 2024-05-27T00:23:25.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1263e00640 1 --2- 172.21.15.98:0/1988500097 >> v2:172.21.15.204:3300/0 conn(0x7f125c0cc1f0 0x7f125c0c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:25.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1263e00640 1 --2- 172.21.15.98:0/1988500097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c0cbe10 0x7f125c1592d0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:25.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.805+0000 7f1263e00640 1 -- 172.21.15.98:0/1988500097 >> 172.21.15.98:0/1988500097 conn(0x7f125c067530 msgr2=0x7f125c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:25.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1263e00640 1 -- 172.21.15.98:0/1988500097 shutdown_connections 2024-05-27T00:23:25.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1263e00640 1 -- 172.21.15.98:0/1988500097 wait complete. 2024-05-27T00:23:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1263e00640 1 Processor -- start 2024-05-27T00:23:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1263e00640 1 -- start start 2024-05-27T00:23:25.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1263e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f125c073570 0x7f125c1f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1263e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125c0cbe10 0x7f125c1f0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1263e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c0cc1f0 0x7f125c157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1263e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f125c0c79d0 con 0x7f125c0cc1f0 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1263e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f125c0c1330 con 0x7f125c073570 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1263e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f125c0c6990 con 0x7f125c0cbe10 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1261400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f125c073570 0x7f125c1f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1261e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c0cc1f0 0x7f125c157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1261400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f125c073570 0x7f125c1f4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:48436/0 (socket says 172.21.15.98:48436) 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1261400640 1 -- 172.21.15.98:0/4128508550 learned_addr learned my addr 172.21.15.98:0/4128508550 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1260a00640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125c0cbe10 0x7f125c1f0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1261e00640 1 -- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125c0cbe10 msgr2=0x7f125c1f0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1261e00640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125c0cbe10 0x7f125c1f0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:25.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1261e00640 1 -- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f125c073570 msgr2=0x7f125c1f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:25.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1261e00640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f125c073570 0x7f125c1f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:25.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1261e00640 1 -- 172.21.15.98:0/4128508550 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125c1f01a0 con 0x7f125c0cc1f0 2024-05-27T00:23:25.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.809+0000 7f1261e00640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c0cc1f0 0x7f125c157b80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f125400c730 tx=0x7f125400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:25.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.813+0000 7f1259600640 1 -- 172.21.15.98:0/4128508550 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254003b80 con 0x7f125c0cc1f0 2024-05-27T00:23:25.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.813+0000 7f1259600640 1 -- 172.21.15.98:0/4128508550 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1254003d20 con 0x7f125c0cc1f0 2024-05-27T00:23:25.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.813+0000 7f1259600640 1 -- 172.21.15.98:0/4128508550 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254005310 con 0x7f125c0cc1f0 2024-05-27T00:23:25.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.813+0000 7f1263e00640 1 -- 172.21.15.98:0/4128508550 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125c0af640 con 0x7f125c0cc1f0 2024-05-27T00:23:25.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.813+0000 7f1263e00640 1 -- 172.21.15.98:0/4128508550 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125c0afc00 con 0x7f125c0cc1f0 2024-05-27T00:23:25.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.813+0000 7f1263e00640 1 -- 172.21.15.98:0/4128508550 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f125c075cb0 con 0x7f125c0cc1f0 2024-05-27T00:23:25.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.813+0000 7f1259600640 1 -- 172.21.15.98:0/4128508550 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f12540056c0 con 0x7f125c0cc1f0 2024-05-27T00:23:25.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.813+0000 7f1259600640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f123c0678f0 0x7f123c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:25.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.813+0000 7f1259600640 1 -- 172.21.15.98:0/4128508550 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f1254085db0 con 0x7f125c0cc1f0 2024-05-27T00:23:25.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.813+0000 7f1261400640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f123c0678f0 0x7f123c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:25.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.817+0000 7f1261400640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f123c0678f0 0x7f123c069db0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f125001ed50 tx=0x7f1250009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:25.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:25.817+0000 7f1259600640 1 -- 172.21.15.98:0/4128508550 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f125408a070 con 0x7f125c0cc1f0 2024-05-27T00:23:26.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.005+0000 7f1263e00640 1 -- 172.21.15.98:0/4128508550 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f125c0afdd0 con 0x7f125c0cc1f0 2024-05-27T00:23:26.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.005+0000 7f1259600640 1 -- 172.21.15.98:0/4128508550 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6836 (secure 0 0 0) 0x7f125c0afdd0 con 0x7f125c0cc1f0 2024-05-27T00:23:26.009 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:26.009 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:50.518361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5343,"qdb_cluster":[5343]},"id":1}]} 2024-05-27T00:23:26.010 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-27T00:23:26.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 -- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f123c0678f0 msgr2=0x7f123c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:26.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f123c0678f0 0x7f123c069db0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f125001ed50 tx=0x7f1250009040 comp rx=0 tx=0).stop 2024-05-27T00:23:26.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 -- 172.21.15.98:0/4128508550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c0cc1f0 msgr2=0x7f125c157b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:26.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c0cc1f0 0x7f125c157b80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f125400c730 tx=0x7f125400cbf0 comp rx=0 tx=0).stop 2024-05-27T00:23:26.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 -- 172.21.15.98:0/4128508550 shutdown_connections 2024-05-27T00:23:26.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c0cc1f0 0x7f125c157b80 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:26.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125c0cbe10 0x7f125c1f0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:26.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f123c0678f0 0x7f123c069db0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:26.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 --2- 172.21.15.98:0/4128508550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f125c073570 0x7f125c1f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:26.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 -- 172.21.15.98:0/4128508550 >> 172.21.15.98:0/4128508550 conn(0x7f125c067530 msgr2=0x7f125c06bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:26.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 -- 172.21.15.98:0/4128508550 shutdown_connections 2024-05-27T00:23:26.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:26.009+0000 7f1263e00640 1 -- 172.21.15.98:0/4128508550 wait complete. 2024-05-27T00:23:26.031 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:23:28.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5be6200640 1 Processor -- start 2024-05-27T00:23:28.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5be6200640 1 -- start start 2024-05-27T00:23:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5be6200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5be00cc1f0 0x7f5be01592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5be6200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5be0073570 0x7f5be01559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5be6200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5be00c1f60 con 0x7f5be00cbe10 2024-05-27T00:23:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5be6200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5be0159810 con 0x7f5be00cc1f0 2024-05-27T00:23:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5be6200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f5be00c6dd0 con 0x7f5be0073570 2024-05-27T00:23:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5bdea00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5be00cc1f0 0x7f5be01592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5bdea00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5be00cc1f0 0x7f5be01592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50672/0 (socket says 172.21.15.98:50672) 2024-05-27T00:23:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5bdea00640 1 -- 172.21.15.98:0/3448119223 learned_addr learned my addr 172.21.15.98:0/3448119223 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:28.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5bdfe00640 1 --2- 172.21.15.98:0/3448119223 >> v2:172.21.15.204:3300/0 conn(0x7f5be0073570 0x7f5be01559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.177+0000 7f5bdea00640 1 -- 172.21.15.98:0/3448119223 >> v2:172.21.15.204:3300/0 conn(0x7f5be0073570 msgr2=0x7f5be01559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5bdea00640 1 --2- 172.21.15.98:0/3448119223 >> v2:172.21.15.204:3300/0 conn(0x7f5be0073570 0x7f5be01559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5bdea00640 1 -- 172.21.15.98:0/3448119223 >> v1:172.21.15.204:6789/0 conn(0x7f5be00cbe10 legacy=0x7f5be00c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:28.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5bdea00640 1 -- 172.21.15.98:0/3448119223 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be00c70b0 con 0x7f5be00cc1f0 2024-05-27T00:23:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5bdea00640 1 --2- 172.21.15.98:0/3448119223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 0x7f5be01592d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5be006e230 tx=0x7f5bcc01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6774a3c49a74ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5bde000640 1 -- 172.21.15.98:0/3448119223 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc0b5070 con 0x7f5be00cc1f0 2024-05-27T00:23:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5bde000640 1 -- 172.21.15.98:0/3448119223 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bcc0a8a10 con 0x7f5be00cc1f0 2024-05-27T00:23:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5bde000640 1 -- 172.21.15.98:0/3448119223 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc0a8de0 con 0x7f5be00cc1f0 2024-05-27T00:23:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5be6200640 1 -- 172.21.15.98:0/3448119223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 msgr2=0x7f5be01592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5be6200640 1 --2- 172.21.15.98:0/3448119223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 0x7f5be01592d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5be006e230 tx=0x7f5bcc01ed10 comp rx=0 tx=0).stop 2024-05-27T00:23:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5be6200640 1 -- 172.21.15.98:0/3448119223 shutdown_connections 2024-05-27T00:23:28.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5be6200640 1 --2- 172.21.15.98:0/3448119223 >> v2:172.21.15.204:3300/0 conn(0x7f5be0073570 0x7f5be01559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5be6200640 1 --2- 172.21.15.98:0/3448119223 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 0x7f5be01592d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5be6200640 1 -- 172.21.15.98:0/3448119223 >> 172.21.15.98:0/3448119223 conn(0x7f5be0067530 msgr2=0x7f5be006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5be6200640 1 -- 172.21.15.98:0/3448119223 shutdown_connections 2024-05-27T00:23:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.181+0000 7f5be6200640 1 -- 172.21.15.98:0/3448119223 wait complete. 2024-05-27T00:23:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5be6200640 1 Processor -- start 2024-05-27T00:23:28.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5be6200640 1 -- start start 2024-05-27T00:23:28.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5be6200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5be0073570 0x7f5be01deae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:28.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5be6200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be00cbe10 0x7f5be014e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:28.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5be6200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 0x7f5be01f1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:28.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5be6200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5be00c1330 con 0x7f5be00cc1f0 2024-05-27T00:23:28.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5be6200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5be00c1050 con 0x7f5be0073570 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5be6200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5be00c79d0 con 0x7f5be00cbe10 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdfe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 0x7f5be01f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdfe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 0x7f5be01f1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50682/0 (socket says 172.21.15.98:50682) 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdfe00640 1 -- 172.21.15.98:0/2946599837 learned_addr learned my addr 172.21.15.98:0/2946599837 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdea00640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be00cbe10 0x7f5be014e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdf400640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5be0073570 0x7f5be01deae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdfe00640 1 -- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be00cbe10 msgr2=0x7f5be014e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdfe00640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be00cbe10 0x7f5be014e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdfe00640 1 -- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5be0073570 msgr2=0x7f5be01deae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdfe00640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5be0073570 0x7f5be01deae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:28.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdfe00640 1 -- 172.21.15.98:0/2946599837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be00c69d0 con 0x7f5be00cc1f0 2024-05-27T00:23:28.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bdfe00640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 0x7f5be01f1890 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800efc0 tx=0x7f5bd800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:28.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bd7400640 1 -- 172.21.15.98:0/2946599837 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8018020 con 0x7f5be00cc1f0 2024-05-27T00:23:28.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bd7400640 1 -- 172.21.15.98:0/2946599837 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bd8014480 con 0x7f5be00cc1f0 2024-05-27T00:23:28.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5bd7400640 1 -- 172.21.15.98:0/2946599837 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8014870 con 0x7f5be00cc1f0 2024-05-27T00:23:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.185+0000 7f5be6200640 1 -- 172.21.15.98:0/2946599837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be01ef280 con 0x7f5be00cc1f0 2024-05-27T00:23:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.189+0000 7f5be6200640 1 -- 172.21.15.98:0/2946599837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be01df100 con 0x7f5be00cc1f0 2024-05-27T00:23:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.189+0000 7f5be6200640 1 -- 172.21.15.98:0/2946599837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5be0075cb0 con 0x7f5be00cc1f0 2024-05-27T00:23:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.189+0000 7f5bd7400640 1 -- 172.21.15.98:0/2946599837 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5bd801b020 con 0x7f5be00cc1f0 2024-05-27T00:23:28.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.189+0000 7f5bd7400640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5bc00678f0 0x7f5bc0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:28.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.189+0000 7f5bd7400640 1 -- 172.21.15.98:0/2946599837 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f5bd80869c0 con 0x7f5be00cc1f0 2024-05-27T00:23:28.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.189+0000 7f5bdf400640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5bc00678f0 0x7f5bc0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:28.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.189+0000 7f5bdf400640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5bc00678f0 0x7f5bc0069db0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0001650 tx=0x7f5bd0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.193+0000 7f5bd7400640 1 -- 172.21.15.98:0/2946599837 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f5bd8022070 con 0x7f5be00cc1f0 2024-05-27T00:23:28.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.381+0000 7f5be6200640 1 -- 172.21.15.98:0/2946599837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5be014ee80 con 0x7f5be00cc1f0 2024-05-27T00:23:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.381+0000 7f5bd7400640 1 -- 172.21.15.98:0/2946599837 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6836 (secure 0 0 0) 0x7f5bd8010040 con 0x7f5be00cc1f0 2024-05-27T00:23:28.384 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:28.384 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:50.518361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5343,"qdb_cluster":[5343]},"id":1}]} 2024-05-27T00:23:28.384 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-27T00:23:28.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 -- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5bc00678f0 msgr2=0x7f5bc0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:28.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5bc00678f0 0x7f5bc0069db0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0001650 tx=0x7f5bd0009040 comp rx=0 tx=0).stop 2024-05-27T00:23:28.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 -- 172.21.15.98:0/2946599837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 msgr2=0x7f5be01f1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:28.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 0x7f5be01f1890 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800efc0 tx=0x7f5bd800c490 comp rx=0 tx=0).stop 2024-05-27T00:23:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 -- 172.21.15.98:0/2946599837 shutdown_connections 2024-05-27T00:23:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5be00cc1f0 0x7f5be01f1890 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be00cbe10 0x7f5be014e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5bc00678f0 0x7f5bc0069db0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 --2- 172.21.15.98:0/2946599837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5be0073570 0x7f5be01deae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 -- 172.21.15.98:0/2946599837 >> 172.21.15.98:0/2946599837 conn(0x7f5be0067530 msgr2=0x7f5be006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 -- 172.21.15.98:0/2946599837 shutdown_connections 2024-05-27T00:23:28.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:28.385+0000 7f5be6200640 1 -- 172.21.15.98:0/2946599837 wait complete. 2024-05-27T00:23:28.405 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:23:30.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:30.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.553+0000 7f172fe00640 1 Processor -- start 2024-05-27T00:23:30.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.553+0000 7f172fe00640 1 -- start start 2024-05-27T00:23:30.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172fe00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1730152810 0x7f173015c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17300c6d10 0x7f17300cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17300cc860 0x7f17300c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172fe00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f17300c7c20 con 0x7f17300cc860 2024-05-27T00:23:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172fe00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f173015ca10 con 0x7f1730152810 2024-05-27T00:23:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f17300c7660 con 0x7f17300c6d10 2024-05-27T00:23:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17300cc860 0x7f17300c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17300cc860 0x7f17300c3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50686/0 (socket says 172.21.15.98:50686) 2024-05-27T00:23:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172f400640 1 -- 172.21.15.98:0/108693632 learned_addr learned my addr 172.21.15.98:0/108693632 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172ea00640 1 --2- 172.21.15.98:0/108693632 >> v2:172.21.15.204:3300/0 conn(0x7f1730152810 0x7f173015c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172e000640 1 --2- 172.21.15.98:0/108693632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17300c6d10 0x7f17300cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172f400640 1 -- 172.21.15.98:0/108693632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17300c6d10 msgr2=0x7f17300cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172f400640 1 --2- 172.21.15.98:0/108693632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17300c6d10 0x7f17300cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172f400640 1 -- 172.21.15.98:0/108693632 >> v2:172.21.15.204:3300/0 conn(0x7f1730152810 msgr2=0x7f173015c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172f400640 1 --2- 172.21.15.98:0/108693632 >> v2:172.21.15.204:3300/0 conn(0x7f1730152810 0x7f173015c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.557+0000 7f172f400640 1 -- 172.21.15.98:0/108693632 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17300c7940 con 0x7f17300cc860 2024-05-27T00:23:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172f400640 1 --2- 172.21.15.98:0/108693632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 0x7f17300c3b50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f17300c9fb0 tx=0x7f172801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a3975f2bc2179b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172d600640 1 -- 172.21.15.98:0/108693632 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17280b1070 con 0x7f17300cc860 2024-05-27T00:23:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172d600640 1 -- 172.21.15.98:0/108693632 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17280ac630 con 0x7f17300cc860 2024-05-27T00:23:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172d600640 1 -- 172.21.15.98:0/108693632 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17280a8050 con 0x7f17300cc860 2024-05-27T00:23:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 -- 172.21.15.98:0/108693632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 msgr2=0x7f17300c3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 --2- 172.21.15.98:0/108693632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 0x7f17300c3b50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f17300c9fb0 tx=0x7f172801c520 comp rx=0 tx=0).stop 2024-05-27T00:23:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 -- 172.21.15.98:0/108693632 shutdown_connections 2024-05-27T00:23:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 --2- 172.21.15.98:0/108693632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 0x7f17300c3b50 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 --2- 172.21.15.98:0/108693632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17300c6d10 0x7f17300cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 --2- 172.21.15.98:0/108693632 >> v2:172.21.15.204:3300/0 conn(0x7f1730152810 0x7f173015c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 -- 172.21.15.98:0/108693632 >> 172.21.15.98:0/108693632 conn(0x7f173006b470 msgr2=0x7f173006b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 -- 172.21.15.98:0/108693632 shutdown_connections 2024-05-27T00:23:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 -- 172.21.15.98:0/108693632 wait complete. 2024-05-27T00:23:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 Processor -- start 2024-05-27T00:23:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.561+0000 7f172fe00640 1 -- start start 2024-05-27T00:23:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17300c6d10 0x7f17301f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 0x7f1730073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1730152810 0x7f173014bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172fe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1730152bf0 con 0x7f17300cc860 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17300c44a0 con 0x7f17300c6d10 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1730062ce0 con 0x7f1730152810 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172e000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 0x7f1730073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172e000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 0x7f1730073060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50700/0 (socket says 172.21.15.98:50700) 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172e000640 1 -- 172.21.15.98:0/2158305225 learned_addr learned my addr 172.21.15.98:0/2158305225 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172ea00640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17300c6d10 0x7f17301f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172f400640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1730152810 0x7f173014bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172e000640 1 -- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1730152810 msgr2=0x7f173014bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172e000640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1730152810 0x7f173014bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172e000640 1 -- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17300c6d10 msgr2=0x7f17301f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172e000640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17300c6d10 0x7f17301f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172e000640 1 -- 172.21.15.98:0/2158305225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17300c9020 con 0x7f17300cc860 2024-05-27T00:23:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172e000640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 0x7f1730073060 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f171c00e8a0 tx=0x7f171c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f1726a00640 1 -- 172.21.15.98:0/2158305225 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c017070 con 0x7f17300cc860 2024-05-27T00:23:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f1726a00640 1 -- 172.21.15.98:0/2158305225 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f171c00ee90 con 0x7f17300cc860 2024-05-27T00:23:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172fe00640 1 -- 172.21.15.98:0/2158305225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17300cc5a0 con 0x7f17300cc860 2024-05-27T00:23:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f1726a00640 1 -- 172.21.15.98:0/2158305225 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c00cb80 con 0x7f17300cc860 2024-05-27T00:23:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172fe00640 1 -- 172.21.15.98:0/2158305225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1730157a20 con 0x7f17300cc860 2024-05-27T00:23:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f172fe00640 1 -- 172.21.15.98:0/2158305225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1730075c40 con 0x7f17300cc860 2024-05-27T00:23:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.565+0000 7f1726a00640 1 -- 172.21.15.98:0/2158305225 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f171c01a070 con 0x7f17300cc860 2024-05-27T00:23:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.569+0000 7f1726a00640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17100678c0 0x7f1710069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.569+0000 7f1726a00640 1 -- 172.21.15.98:0/2158305225 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f171c0860c0 con 0x7f17300cc860 2024-05-27T00:23:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.569+0000 7f172ea00640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17100678c0 0x7f1710069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.569+0000 7f172ea00640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17100678c0 0x7f1710069d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f17200113e0 tx=0x7f172000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.573+0000 7f1726a00640 1 -- 172.21.15.98:0/2158305225 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f171c0562f0 con 0x7f17300cc860 2024-05-27T00:23:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.761+0000 7f172fe00640 1 -- 172.21.15.98:0/2158305225 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17300c82f0 con 0x7f17300cc860 2024-05-27T00:23:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.761+0000 7f1726a00640 1 -- 172.21.15.98:0/2158305225 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6836 (secure 0 0 0) 0x7f171c05a5f0 con 0x7f17300cc860 2024-05-27T00:23:30.765 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:30.766 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:50.518361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5343,"qdb_cluster":[5343]},"id":1}]} 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 -- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17100678c0 msgr2=0x7f1710069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17100678c0 0x7f1710069d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f17200113e0 tx=0x7f172000b040 comp rx=0 tx=0).stop 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 -- 172.21.15.98:0/2158305225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 msgr2=0x7f1730073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 0x7f1730073060 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f171c00e8a0 tx=0x7f171c00ed60 comp rx=0 tx=0).stop 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 -- 172.21.15.98:0/2158305225 shutdown_connections 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1730152810 0x7f173014bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17300cc860 0x7f1730073060 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f17100678c0 0x7f1710069d80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 --2- 172.21.15.98:0/2158305225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17300c6d10 0x7f17301f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 -- 172.21.15.98:0/2158305225 >> 172.21.15.98:0/2158305225 conn(0x7f173006b470 msgr2=0x7f173006b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:30.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 -- 172.21.15.98:0/2158305225 shutdown_connections 2024-05-27T00:23:30.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:30.765+0000 7f172fe00640 1 -- 172.21.15.98:0/2158305225 wait complete. 2024-05-27T00:23:30.785 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:23:32.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:32.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.929+0000 7f45f1200640 1 Processor -- start 2024-05-27T00:23:32.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.929+0000 7f45f1200640 1 -- start start 2024-05-27T00:23:32.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.933+0000 7f45f1200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f45ec0735d0 0x7f45ec151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:32.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.933+0000 7f45f1200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45ec0c17d0 con 0x7f45ec0731f0 2024-05-27T00:23:32.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.933+0000 7f45f1200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45ec070e10 con 0x7f45ec0c79f0 2024-05-27T00:23:32.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.933+0000 7f45f1200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f45ec071990 con 0x7f45ec0735d0 2024-05-27T00:23:32.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.933+0000 7f45e9e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f45ec0731f0 0x7f45ec0c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38364/0 (socket says 172.21.15.98:38364) 2024-05-27T00:23:32.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.933+0000 7f45e9e00640 1 -- 172.21.15.98:0/1597274446 learned_addr learned my addr 172.21.15.98:0/1597274446 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:23:32.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.933+0000 7f45e9400640 1 --2- 172.21.15.98:0/1597274446 >> v2:172.21.15.98:3300/0 conn(0x7f45ec0735d0 0x7f45ec151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:32.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.933+0000 7f45e9400640 1 -- 172.21.15.98:0/1597274446 >> v1:172.21.15.204:6789/0 conn(0x7f45ec0c79f0 legacy=0x7f45ec157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:32.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.933+0000 7f45e9400640 1 -- 172.21.15.98:0/1597274446 >> v1:172.21.15.98:6789/0 conn(0x7f45ec0731f0 legacy=0x7f45ec0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:32.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.933+0000 7f45e9400640 1 -- 172.21.15.98:0/1597274446 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ec0710f0 con 0x7f45ec0735d0 2024-05-27T00:23:32.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45e9400640 1 --2- 172.21.15.98:0/1597274446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0735d0 0x7f45ec151430 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f45ec071b10 tx=0x7f45d801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fa71a0ec8bed6a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:32.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45e8a00640 1 -- 172.21.15.98:0/1597274446 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d80b0070 con 0x7f45ec0735d0 2024-05-27T00:23:32.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45e8a00640 1 -- 172.21.15.98:0/1597274446 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45d80ab600 con 0x7f45ec0735d0 2024-05-27T00:23:32.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45e8a00640 1 -- 172.21.15.98:0/1597274446 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d80ab9d0 con 0x7f45ec0735d0 2024-05-27T00:23:32.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 -- 172.21.15.98:0/1597274446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0735d0 msgr2=0x7f45ec151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:32.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 --2- 172.21.15.98:0/1597274446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0735d0 0x7f45ec151430 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f45ec071b10 tx=0x7f45d801c330 comp rx=0 tx=0).stop 2024-05-27T00:23:32.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 -- 172.21.15.98:0/1597274446 shutdown_connections 2024-05-27T00:23:32.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 --2- 172.21.15.98:0/1597274446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0735d0 0x7f45ec151430 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:32.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 -- 172.21.15.98:0/1597274446 >> 172.21.15.98:0/1597274446 conn(0x7f45ec067530 msgr2=0x7f45ec06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:32.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 -- 172.21.15.98:0/1597274446 shutdown_connections 2024-05-27T00:23:32.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 -- 172.21.15.98:0/1597274446 wait complete. 2024-05-27T00:23:32.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 Processor -- start 2024-05-27T00:23:32.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 -- start start 2024-05-27T00:23:32.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45ec0731f0 0x7f45ec156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45ec0735d0 0x7f45ec1561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0c79f0 0x7f45ec1566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45ec0c8400 con 0x7f45ec0c79f0 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f45ec0716b0 con 0x7f45ec0735d0 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.937+0000 7f45f1200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f45ec158080 con 0x7f45ec0731f0 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45ea800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0c79f0 0x7f45ec1566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45ea800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0c79f0 0x7f45ec1566f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50730/0 (socket says 172.21.15.98:50730) 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45ea800640 1 -- 172.21.15.98:0/3919691731 learned_addr learned my addr 172.21.15.98:0/3919691731 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45e9400640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45ec0735d0 0x7f45ec1561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45e9e00640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45ec0731f0 0x7f45ec156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45ea800640 1 -- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45ec0731f0 msgr2=0x7f45ec156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:32.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45ea800640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45ec0731f0 0x7f45ec156f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:32.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45ea800640 1 -- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45ec0735d0 msgr2=0x7f45ec1561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:32.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45ea800640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45ec0735d0 0x7f45ec1561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:32.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45ea800640 1 -- 172.21.15.98:0/3919691731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ec14c950 con 0x7f45ec0c79f0 2024-05-27T00:23:32.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45e9400640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45ec0735d0 0x7f45ec1561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:23:32.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45ea800640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0c79f0 0x7f45ec1566f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f45e400c370 tx=0x7f45e400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:32.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45e2000640 1 -- 172.21.15.98:0/3919691731 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4005220 con 0x7f45ec0c79f0 2024-05-27T00:23:32.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45e2000640 1 -- 172.21.15.98:0/3919691731 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45e40053c0 con 0x7f45ec0c79f0 2024-05-27T00:23:32.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45f1200640 1 -- 172.21.15.98:0/3919691731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ec14cc30 con 0x7f45ec0c79f0 2024-05-27T00:23:32.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45e2000640 1 -- 172.21.15.98:0/3919691731 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4007270 con 0x7f45ec0c79f0 2024-05-27T00:23:32.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45f1200640 1 -- 172.21.15.98:0/3919691731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ec1556d0 con 0x7f45ec0c79f0 2024-05-27T00:23:32.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45f1200640 1 -- 172.21.15.98:0/3919691731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45ec075cb0 con 0x7f45ec0c79f0 2024-05-27T00:23:32.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45e2000640 1 -- 172.21.15.98:0/3919691731 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f45e4019020 con 0x7f45ec0c79f0 2024-05-27T00:23:32.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.941+0000 7f45e2000640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45cc0678f0 0x7f45cc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:32.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.945+0000 7f45e2000640 1 -- 172.21.15.98:0/3919691731 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f45e4085220 con 0x7f45ec0c79f0 2024-05-27T00:23:32.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.945+0000 7f45e9e00640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45cc0678f0 0x7f45cc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:32.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.945+0000 7f45e9e00640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45cc0678f0 0x7f45cc069db0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0012f0 tx=0x7f45dc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:32.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:32.945+0000 7f45e2000640 1 -- 172.21.15.98:0/3919691731 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f45e4055aa0 con 0x7f45ec0c79f0 2024-05-27T00:23:33.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.133+0000 7f45f1200640 1 -- 172.21.15.98:0/3919691731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45ec158510 con 0x7f45ec0c79f0 2024-05-27T00:23:33.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.133+0000 7f45e2000640 1 -- 172.21.15.98:0/3919691731 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6836 (secure 0 0 0) 0x7f45e40553a0 con 0x7f45ec0c79f0 2024-05-27T00:23:33.137 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:33.137 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:50.518361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5343,"qdb_cluster":[5343]},"id":1}]} 2024-05-27T00:23:33.137 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-27T00:23:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 -- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45cc0678f0 msgr2=0x7f45cc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45cc0678f0 0x7f45cc069db0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0012f0 tx=0x7f45dc00b040 comp rx=0 tx=0).stop 2024-05-27T00:23:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 -- 172.21.15.98:0/3919691731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0c79f0 msgr2=0x7f45ec1566f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:33.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0c79f0 0x7f45ec1566f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f45e400c370 tx=0x7f45e400c830 comp rx=0 tx=0).stop 2024-05-27T00:23:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 -- 172.21.15.98:0/3919691731 shutdown_connections 2024-05-27T00:23:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45ec0c79f0 0x7f45ec1566f0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f45cc0678f0 0x7f45cc069db0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45ec0735d0 0x7f45ec1561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 --2- 172.21.15.98:0/3919691731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45ec0731f0 0x7f45ec156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 -- 172.21.15.98:0/3919691731 >> 172.21.15.98:0/3919691731 conn(0x7f45ec067530 msgr2=0x7f45ec0c6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 -- 172.21.15.98:0/3919691731 shutdown_connections 2024-05-27T00:23:33.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:33.137+0000 7f45f1200640 1 -- 172.21.15.98:0/3919691731 wait complete. 2024-05-27T00:23:33.159 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:23:35.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:35.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b7e00640 1 Processor -- start 2024-05-27T00:23:35.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b7e00640 1 -- start start 2024-05-27T00:23:35.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa0b00cbe10 0x7fa0b01592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:35.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b7e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa0b00cc1f0 0x7fa0b00c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b7e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b00c6dd0 con 0x7fa0b0073570 2024-05-27T00:23:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b7e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b0159810 con 0x7fa0b00cbe10 2024-05-27T00:23:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b7e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b00c70b0 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b5400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa0b00cbe10 0x7fa0b01592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b5400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa0b00cbe10 0x7fa0b01592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39060/0 (socket says 172.21.15.98:39060) 2024-05-27T00:23:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b5400640 1 -- 172.21.15.98:0/2703377901 learned_addr learned my addr 172.21.15.98:0/2703377901 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:35.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b5e00640 1 --1- 172.21.15.98:0/2703377901 >> v1:172.21.15.98:6789/0 conn(0x7fa0b0073570 0x7fa0b01559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52470/0 (socket says 172.21.15.98:52470) 2024-05-27T00:23:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b4a00640 1 --2- 172.21.15.98:0/2703377901 >> v2:172.21.15.204:3300/0 conn(0x7fa0b00cc1f0 0x7fa0b00c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b5400640 1 -- 172.21.15.98:0/2703377901 >> v2:172.21.15.204:3300/0 conn(0x7fa0b00cc1f0 msgr2=0x7fa0b00c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b5400640 1 --2- 172.21.15.98:0/2703377901 >> v2:172.21.15.204:3300/0 conn(0x7fa0b00cc1f0 0x7fa0b00c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.305+0000 7fa0b5400640 1 -- 172.21.15.98:0/2703377901 >> v1:172.21.15.98:6789/0 conn(0x7fa0b0073570 legacy=0x7fa0b01559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0b5400640 1 -- 172.21.15.98:0/2703377901 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b00c7390 con 0x7fa0b00cbe10 2024-05-27T00:23:35.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0afe00640 1 -- 172.21.15.98:0/2703377901 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1237698447 0 0) 0x7fa0b00c6dd0 con 0x7fa0b0073570 2024-05-27T00:23:35.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0b5400640 1 --2- 172.21.15.98:0/2703377901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cbe10 0x7fa0b01592d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00cc6d0 tx=0x7fa0a401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e09d71c37ed2acc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:35.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0afe00640 1 -- 172.21.15.98:0/2703377901 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a40b4070 con 0x7fa0b00cbe10 2024-05-27T00:23:35.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0afe00640 1 -- 172.21.15.98:0/2703377901 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a40af400 con 0x7fa0b00cbe10 2024-05-27T00:23:35.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0afe00640 1 -- 172.21.15.98:0/2703377901 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a40af7d0 con 0x7fa0b00cbe10 2024-05-27T00:23:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0b7e00640 1 -- 172.21.15.98:0/2703377901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cbe10 msgr2=0x7fa0b01592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0b7e00640 1 --2- 172.21.15.98:0/2703377901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cbe10 0x7fa0b01592d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00cc6d0 tx=0x7fa0a401f1e0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0b7e00640 1 -- 172.21.15.98:0/2703377901 shutdown_connections 2024-05-27T00:23:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0b7e00640 1 --2- 172.21.15.98:0/2703377901 >> v2:172.21.15.204:3300/0 conn(0x7fa0b00cc1f0 0x7fa0b00c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0b7e00640 1 --2- 172.21.15.98:0/2703377901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cbe10 0x7fa0b01592d0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0b7e00640 1 -- 172.21.15.98:0/2703377901 >> 172.21.15.98:0/2703377901 conn(0x7fa0b0067530 msgr2=0x7fa0b006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0b7e00640 1 -- 172.21.15.98:0/2703377901 shutdown_connections 2024-05-27T00:23:35.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.309+0000 7fa0b7e00640 1 -- 172.21.15.98:0/2703377901 wait complete. 2024-05-27T00:23:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b7e00640 1 Processor -- start 2024-05-27T00:23:35.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b7e00640 1 -- start start 2024-05-27T00:23:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0b0073570 0x7fa0b01e0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0b00cbe10 0x7fa0b01edc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cc1f0 0x7fa0b0156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b7e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b00c79d0 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b7e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b00c1330 con 0x7fa0b00cbe10 2024-05-27T00:23:35.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b7e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b00c6990 con 0x7fa0b0073570 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cc1f0 0x7fa0b0156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cc1f0 0x7fa0b0156900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39070/0 (socket says 172.21.15.98:39070) 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5e00640 1 -- 172.21.15.98:0/1263534485 learned_addr learned my addr 172.21.15.98:0/1263534485 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5400640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0b0073570 0x7fa0b01e0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b4a00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0b00cbe10 0x7fa0b01edc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5e00640 1 -- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0b0073570 msgr2=0x7fa0b01e0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5e00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0b0073570 0x7fa0b01e0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5e00640 1 -- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0b00cbe10 msgr2=0x7fa0b01edc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5e00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0b00cbe10 0x7fa0b01edc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5e00640 1 -- 172.21.15.98:0/1263534485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b01deb80 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5400640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0b0073570 0x7fa0b01e0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b4a00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0b00cbe10 0x7fa0b01edc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:23:35.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b5e00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cc1f0 0x7fa0b0156900 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800c410 tx=0x7fa0a800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:35.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0ad600640 1 -- 172.21.15.98:0/1263534485 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a80074e0 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0ad600640 1 -- 172.21.15.98:0/1263534485 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a80047f0 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b7e00640 1 -- 172.21.15.98:0/1263534485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b01dee60 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0ad600640 1 -- 172.21.15.98:0/1263534485 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8007680 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.313+0000 7fa0b7e00640 1 -- 172.21.15.98:0/1263534485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b01df420 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.317+0000 7fa0b7e00640 1 -- 172.21.15.98:0/1263534485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b0075cb0 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.317+0000 7fa0ad600640 1 -- 172.21.15.98:0/1263534485 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa0a8011070 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.317+0000 7fa0ad600640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0900678c0 0x7fa090069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:35.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.317+0000 7fa0ad600640 1 -- 172.21.15.98:0/1263534485 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7fa0a8085a50 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.317+0000 7fa0b5400640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0900678c0 0x7fa090069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.317+0000 7fa0b5400640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0900678c0 0x7fa090069d80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40b1000 tx=0x7fa0a40aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:35.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.321+0000 7fa0ad600640 1 -- 172.21.15.98:0/1263534485 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa0a8056360 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.509+0000 7fa0b7e00640 1 -- 172.21.15.98:0/1263534485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0b006ec60 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.509+0000 7fa0ad600640 1 -- 172.21.15.98:0/1263534485 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6836 (secure 0 0 0) 0x7fa0a8055c60 con 0x7fa0b00cc1f0 2024-05-27T00:23:35.513 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:35.513 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:50.518361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5343,"qdb_cluster":[5343]},"id":1}]} 2024-05-27T00:23:35.513 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-27T00:23:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 -- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0900678c0 msgr2=0x7fa090069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0900678c0 0x7fa090069d80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40b1000 tx=0x7fa0a40aa000 comp rx=0 tx=0).stop 2024-05-27T00:23:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 -- 172.21.15.98:0/1263534485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cc1f0 msgr2=0x7fa0b0156900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cc1f0 0x7fa0b0156900 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800c410 tx=0x7fa0a800c8d0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 -- 172.21.15.98:0/1263534485 shutdown_connections 2024-05-27T00:23:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0b00cc1f0 0x7fa0b0156900 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0900678c0 0x7fa090069d80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0b00cbe10 0x7fa0b01edc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 --2- 172.21.15.98:0/1263534485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0b0073570 0x7fa0b01e0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 -- 172.21.15.98:0/1263534485 >> 172.21.15.98:0/1263534485 conn(0x7fa0b0067530 msgr2=0x7fa0b0158110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 -- 172.21.15.98:0/1263534485 shutdown_connections 2024-05-27T00:23:35.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:35.513+0000 7fa0b7e00640 1 -- 172.21.15.98:0/1263534485 wait complete. 2024-05-27T00:23:35.534 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:23:37.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfe1600640 1 Processor -- start 2024-05-27T00:23:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfe1600640 1 -- start start 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfe1600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcfdc15b880 0x7fcfdc0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfe1600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fcfdc06b470 0x7fcfdc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfe1600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfdc067fe0 con 0x7fcfdc0c7960 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfe1600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcfdc06e5c0 con 0x7fcfdc15b880 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfe1600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fcfdc0c6c30 con 0x7fcfdc06b470 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfdaa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcfdc15b880 0x7fcfdc0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfdaa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcfdc15b880 0x7fcfdc0c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39078/0 (socket says 172.21.15.98:39078) 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfdaa00640 1 -- 172.21.15.98:0/363644057 learned_addr learned my addr 172.21.15.98:0/363644057 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfd2000640 1 --2- 172.21.15.98:0/363644057 >> v2:172.21.15.204:3300/0 conn(0x7fcfdc06b470 0x7fcfdc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfdaa00640 1 -- 172.21.15.98:0/363644057 >> v2:172.21.15.204:3300/0 conn(0x7fcfdc06b470 msgr2=0x7fcfdc06e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfdaa00640 1 --2- 172.21.15.98:0/363644057 >> v2:172.21.15.204:3300/0 conn(0x7fcfdc06b470 0x7fcfdc06e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfdaa00640 1 -- 172.21.15.98:0/363644057 >> v1:172.21.15.204:6789/0 conn(0x7fcfdc0c7960 legacy=0x7fcfdc067130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:37.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.681+0000 7fcfdaa00640 1 -- 172.21.15.98:0/363644057 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc0c6e80 con 0x7fcfdc15b880 2024-05-27T00:23:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfdaa00640 1 --2- 172.21.15.98:0/363644057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc15b880 0x7fcfdc0c93d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc072d80 tx=0x7fcfcc01c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d7eb78f9eabb0c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfda000640 1 -- 172.21.15.98:0/363644057 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0b0070 con 0x7fcfdc15b880 2024-05-27T00:23:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfda000640 1 -- 172.21.15.98:0/363644057 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfcc0ab600 con 0x7fcfdc15b880 2024-05-27T00:23:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfda000640 1 -- 172.21.15.98:0/363644057 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0ab9d0 con 0x7fcfdc15b880 2024-05-27T00:23:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfe1600640 1 -- 172.21.15.98:0/363644057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc15b880 msgr2=0x7fcfdc0c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfe1600640 1 --2- 172.21.15.98:0/363644057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc15b880 0x7fcfdc0c93d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc072d80 tx=0x7fcfcc01c270 comp rx=0 tx=0).stop 2024-05-27T00:23:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfe1600640 1 -- 172.21.15.98:0/363644057 shutdown_connections 2024-05-27T00:23:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfe1600640 1 --2- 172.21.15.98:0/363644057 >> v2:172.21.15.204:3300/0 conn(0x7fcfdc06b470 0x7fcfdc06e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfe1600640 1 --2- 172.21.15.98:0/363644057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc15b880 0x7fcfdc0c93d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfe1600640 1 -- 172.21.15.98:0/363644057 >> 172.21.15.98:0/363644057 conn(0x7fcfdc0cd060 msgr2=0x7fcfdc0cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfe1600640 1 -- 172.21.15.98:0/363644057 shutdown_connections 2024-05-27T00:23:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfe1600640 1 -- 172.21.15.98:0/363644057 wait complete. 2024-05-27T00:23:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.685+0000 7fcfe1600640 1 Processor -- start 2024-05-27T00:23:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfe1600640 1 -- start start 2024-05-27T00:23:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfe1600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc06b470 0x7fcfdc15e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfe1600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcfdc0c7960 0x7fcfdc157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfe1600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcfdc15b880 0x7fcfdc151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfe1600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc0738f0 con 0x7fcfdc06b470 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfe1600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc073770 con 0x7fcfdc0c7960 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfe1600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcfdc067840 con 0x7fcfdc15b880 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfdaa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc06b470 0x7fcfdc15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfdaa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc06b470 0x7fcfdc15e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39090/0 (socket says 172.21.15.98:39090) 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfdaa00640 1 -- 172.21.15.98:0/591244167 learned_addr learned my addr 172.21.15.98:0/591244167 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfdb400640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcfdc15b880 0x7fcfdc151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfd2000640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcfdc0c7960 0x7fcfdc157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfdaa00640 1 -- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcfdc15b880 msgr2=0x7fcfdc151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfdaa00640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcfdc15b880 0x7fcfdc151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfdaa00640 1 -- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcfdc0c7960 msgr2=0x7fcfdc157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfdaa00640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcfdc0c7960 0x7fcfdc157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfdaa00640 1 -- 172.21.15.98:0/591244167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc15a100 con 0x7fcfdc06b470 2024-05-27T00:23:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfdaa00640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc06b470 0x7fcfdc15e590 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc01a920 tx=0x7fcfcc019010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfd3400640 1 -- 172.21.15.98:0/591244167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0b0040 con 0x7fcfdc06b470 2024-05-27T00:23:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfd3400640 1 -- 172.21.15.98:0/591244167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfcc0b4420 con 0x7fcfdc06b470 2024-05-27T00:23:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfd3400640 1 -- 172.21.15.98:0/591244167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0b4810 con 0x7fcfdc06b470 2024-05-27T00:23:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfe1600640 1 -- 172.21.15.98:0/591244167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc1580e0 con 0x7fcfdc06b470 2024-05-27T00:23:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfe1600640 1 -- 172.21.15.98:0/591244167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfdc1f3f80 con 0x7fcfdc06b470 2024-05-27T00:23:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.689+0000 7fcfe1600640 1 -- 172.21.15.98:0/591244167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfdc075cb0 con 0x7fcfdc06b470 2024-05-27T00:23:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.693+0000 7fcfd3400640 1 -- 172.21.15.98:0/591244167 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcfcc0b49b0 con 0x7fcfdc06b470 2024-05-27T00:23:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.693+0000 7fcfd3400640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcfb80678f0 0x7fcfb8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.693+0000 7fcfd3400640 1 -- 172.21.15.98:0/591244167 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7fcfcc11fb20 con 0x7fcfdc06b470 2024-05-27T00:23:37.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.693+0000 7fcfd2000640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcfb80678f0 0x7fcfb8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:37.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.693+0000 7fcfd2000640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcfb80678f0 0x7fcfb8069db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc1529b0 tx=0x7fcfc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:37.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.697+0000 7fcfd3400640 1 -- 172.21.15.98:0/591244167 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fcfdc067fe0 con 0x7fcfdc06b470 2024-05-27T00:23:37.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 -- 172.21.15.98:0/591244167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfdc0b1dc0 con 0x7fcfdc06b470 2024-05-27T00:23:37.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfd3400640 1 -- 172.21.15.98:0/591244167 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6836 (secure 0 0 0) 0x7fcfcc11f070 con 0x7fcfdc06b470 2024-05-27T00:23:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:37.887 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:22:50.518361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5343":{"gid":5343,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6834/4290077910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":4290077910},{"type":"v1","addr":"172.21.15.98:6834","nonce":4290077910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5343,"qdb_cluster":[5343]},"id":1}]} 2024-05-27T00:23:37.887 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 35 2024-05-27T00:23:37.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 -- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcfb80678f0 msgr2=0x7fcfb8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcfb80678f0 0x7fcfb8069db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc1529b0 tx=0x7fcfc4008040 comp rx=0 tx=0).stop 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 -- 172.21.15.98:0/591244167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc06b470 msgr2=0x7fcfdc15e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc06b470 0x7fcfdc15e590 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc01a920 tx=0x7fcfcc019010 comp rx=0 tx=0).stop 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 -- 172.21.15.98:0/591244167 shutdown_connections 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcfdc15b880 0x7fcfdc151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcfb80678f0 0x7fcfb8069db0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcfdc0c7960 0x7fcfdc157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 --2- 172.21.15.98:0/591244167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfdc06b470 0x7fcfdc15e590 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 -- 172.21.15.98:0/591244167 >> 172.21.15.98:0/591244167 conn(0x7fcfdc0cd060 msgr2=0x7fcfdc0c6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 -- 172.21.15.98:0/591244167 shutdown_connections 2024-05-27T00:23:37.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:37.885+0000 7fcfe1600640 1 -- 172.21.15.98:0/591244167 wait complete. 2024-05-27T00:23:37.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:23:39.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.053+0000 7f5978200640 1 Processor -- start 2024-05-27T00:23:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.053+0000 7f5978200640 1 -- start start 2024-05-27T00:23:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.053+0000 7f5978200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f59700cbe10 0x7f59701592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.053+0000 7f5978200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59700cc1f0 0x7f59700c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.053+0000 7f5978200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59700c6dd0 con 0x7f5970073570 2024-05-27T00:23:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.053+0000 7f5978200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f5970159810 con 0x7f59700cbe10 2024-05-27T00:23:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.053+0000 7f5978200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f59700c70b0 con 0x7f59700cc1f0 2024-05-27T00:23:40.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.057+0000 7f5976200640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5970073570 0x7f59701559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52480/0 (socket says 172.21.15.98:52480) 2024-05-27T00:23:40.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.057+0000 7f5976200640 1 -- 172.21.15.98:0/2360970399 learned_addr learned my addr 172.21.15.98:0/2360970399 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:23:40.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.057+0000 7f5974e00640 1 --2- 172.21.15.98:0/2360970399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59700cc1f0 0x7f59700c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.057+0000 7f5975800640 1 --2- 172.21.15.98:0/2360970399 >> v2:172.21.15.204:3300/0 conn(0x7f59700cbe10 0x7f59701592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.057+0000 7f596fe00640 1 -- 172.21.15.98:0/2360970399 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 996557083 0 0) 0x7f59700c6dd0 con 0x7f5970073570 2024-05-27T00:23:40.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.057+0000 7f596fe00640 1 -- 172.21.15.98:0/2360970399 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5958003530 con 0x7f5970073570 2024-05-27T00:23:40.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.057+0000 7f5974e00640 1 -- 172.21.15.98:0/2360970399 >> v2:172.21.15.204:3300/0 conn(0x7f59700cbe10 msgr2=0x7f59701592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:40.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.057+0000 7f5974e00640 1 --2- 172.21.15.98:0/2360970399 >> v2:172.21.15.204:3300/0 conn(0x7f59700cbe10 0x7f59701592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.057+0000 7f5974e00640 1 -- 172.21.15.98:0/2360970399 >> v1:172.21.15.98:6789/0 conn(0x7f5970073570 legacy=0x7f59701559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.057+0000 7f5974e00640 1 -- 172.21.15.98:0/2360970399 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59700c7390 con 0x7f59700cc1f0 2024-05-27T00:23:40.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5974e00640 1 --2- 172.21.15.98:0/2360970399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59700cc1f0 0x7f59700c1850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f595c0097c0 tx=0x7f595c01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aafe03f26974f82f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:40.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f596fe00640 1 -- 172.21.15.98:0/2360970399 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c0b5070 con 0x7f59700cc1f0 2024-05-27T00:23:40.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f596fe00640 1 -- 172.21.15.98:0/2360970399 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f595c0a8990 con 0x7f59700cc1f0 2024-05-27T00:23:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f596fe00640 1 -- 172.21.15.98:0/2360970399 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c0a8d40 con 0x7f59700cc1f0 2024-05-27T00:23:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 -- 172.21.15.98:0/2360970399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59700cc1f0 msgr2=0x7f59700c1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 --2- 172.21.15.98:0/2360970399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59700cc1f0 0x7f59700c1850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f595c0097c0 tx=0x7f595c01ec90 comp rx=0 tx=0).stop 2024-05-27T00:23:40.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 -- 172.21.15.98:0/2360970399 shutdown_connections 2024-05-27T00:23:40.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 --2- 172.21.15.98:0/2360970399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59700cc1f0 0x7f59700c1850 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 --2- 172.21.15.98:0/2360970399 >> v2:172.21.15.204:3300/0 conn(0x7f59700cbe10 0x7f59701592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 -- 172.21.15.98:0/2360970399 >> 172.21.15.98:0/2360970399 conn(0x7f5970067530 msgr2=0x7f597006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:40.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 -- 172.21.15.98:0/2360970399 shutdown_connections 2024-05-27T00:23:40.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 -- 172.21.15.98:0/2360970399 wait complete. 2024-05-27T00:23:40.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 Processor -- start 2024-05-27T00:23:40.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 -- start start 2024-05-27T00:23:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5970073570 0x7f5970158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59700cbe10 0x7f59701f1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59700cc1f0 0x7f5970159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59700c79d0 con 0x7f59700cbe10 2024-05-27T00:23:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f59700c1330 con 0x7f59700cc1f0 2024-05-27T00:23:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.061+0000 7f5978200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f59700c6990 con 0x7f5970073570 2024-05-27T00:23:40.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5974e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59700cbe10 0x7f59701f1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5974e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59700cbe10 0x7f59701f1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39104/0 (socket says 172.21.15.98:39104) 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5974e00640 1 -- 172.21.15.98:0/1541190208 learned_addr learned my addr 172.21.15.98:0/1541190208 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5975800640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5970073570 0x7f5970158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5976200640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59700cc1f0 0x7f5970159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5974e00640 1 -- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5970073570 msgr2=0x7f5970158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5974e00640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5970073570 0x7f5970158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5974e00640 1 -- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59700cc1f0 msgr2=0x7f5970159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5974e00640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59700cc1f0 0x7f5970159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5974e00640 1 -- 172.21.15.98:0/1541190208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59701f17a0 con 0x7f59700cbe10 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5974e00640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59700cbe10 0x7f59701f1260 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f595c0b3040 tx=0x7f595c01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f596d600640 1 -- 172.21.15.98:0/1541190208 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c0c1070 con 0x7f59700cbe10 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f596d600640 1 -- 172.21.15.98:0/1541190208 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f595c004240 con 0x7f59700cbe10 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5978200640 1 -- 172.21.15.98:0/1541190208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59701584c0 con 0x7f59700cbe10 2024-05-27T00:23:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f596d600640 1 -- 172.21.15.98:0/1541190208 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c0b5040 con 0x7f59700cbe10 2024-05-27T00:23:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5978200640 1 -- 172.21.15.98:0/1541190208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5970158a80 con 0x7f59700cbe10 2024-05-27T00:23:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f5978200640 1 -- 172.21.15.98:0/1541190208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5970159350 con 0x7f59700cbe10 2024-05-27T00:23:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f596d600640 1 -- 172.21.15.98:0/1541190208 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f595c0c4070 con 0x7f59700cbe10 2024-05-27T00:23:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f596d600640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59500678c0 0x7f5950069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.065+0000 7f596d600640 1 -- 172.21.15.98:0/1541190208 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6255+0+0 (secure 0 0 0) 0x7f595c122f70 con 0x7f59700cbe10 2024-05-27T00:23:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.069+0000 7f5975800640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59500678c0 0x7f5950069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.069+0000 7f5975800640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59500678c0 0x7f5950069d80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5964009af0 tx=0x7f5964008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.073+0000 7f596d600640 1 -- 172.21.15.98:0/1541190208 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f5970159350 con 0x7f59700cbe10 2024-05-27T00:23:40.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.269+0000 7f5978200640 1 -- 172.21.15.98:0/1541190208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5970075cb0 con 0x7f59700cbe10 2024-05-27T00:23:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.269+0000 7f596d600640 1 -- 172.21.15.98:0/1541190208 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6003 (secure 0 0 0) 0x7f595c0187f0 con 0x7f59700cbe10 2024-05-27T00:23:40.273 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:40.273 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:40.226922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:23:40.273 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 36 2024-05-27T00:23:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 -- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59500678c0 msgr2=0x7f5950069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59500678c0 0x7f5950069d80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5964009af0 tx=0x7f5964008040 comp rx=0 tx=0).stop 2024-05-27T00:23:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 -- 172.21.15.98:0/1541190208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59700cbe10 msgr2=0x7f59701f1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59700cbe10 0x7f59701f1260 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f595c0b3040 tx=0x7f595c01ba30 comp rx=0 tx=0).stop 2024-05-27T00:23:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 -- 172.21.15.98:0/1541190208 shutdown_connections 2024-05-27T00:23:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f59500678c0 0x7f5950069d80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59700cc1f0 0x7f5970159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59700cbe10 0x7f59701f1260 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 --2- 172.21.15.98:0/1541190208 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5970073570 0x7f5970158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 -- 172.21.15.98:0/1541190208 >> 172.21.15.98:0/1541190208 conn(0x7f5970067530 msgr2=0x7f597006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 -- 172.21.15.98:0/1541190208 shutdown_connections 2024-05-27T00:23:40.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.273+0000 7f5978200640 1 -- 172.21.15.98:0/1541190208 wait complete. 2024-05-27T00:23:40.295 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-05-27T00:23:40.295 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:23:40.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.429+0000 7f86d1000640 1 Processor -- start 2024-05-27T00:23:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.429+0000 7f86d1000640 1 -- start start 2024-05-27T00:23:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.429+0000 7f86d1000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc062720 0x7f86cc072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.429+0000 7f86d1000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f86cc15e6b0 0x7f86cc06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.429+0000 7f86d1000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86cc0700d0 con 0x7f86cc154a80 2024-05-27T00:23:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.429+0000 7f86d1000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f86cc0726a0 con 0x7f86cc15e6b0 2024-05-27T00:23:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.429+0000 7f86d1000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f86cc071a90 con 0x7f86cc062720 2024-05-27T00:23:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.433+0000 7f86c9400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc062720 0x7f86cc072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.433+0000 7f86ca800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f86cc15e6b0 0x7f86cc06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.433+0000 7f86c9e00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f86cc154a80 0x7f86cc0c6f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:43064/0 (socket says 172.21.15.98:43064) 2024-05-27T00:23:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.433+0000 7f86c9e00640 1 -- 172.21.15.98:0/1701147710 learned_addr learned my addr 172.21.15.98:0/1701147710 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:23:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.433+0000 7f86c9400640 1 -- 172.21.15.98:0/1701147710 >> v2:172.21.15.204:3300/0 conn(0x7f86cc15e6b0 msgr2=0x7f86cc06ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.433+0000 7f86c9400640 1 --2- 172.21.15.98:0/1701147710 >> v2:172.21.15.204:3300/0 conn(0x7f86cc15e6b0 0x7f86cc06ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.433+0000 7f86c9400640 1 -- 172.21.15.98:0/1701147710 >> v1:172.21.15.204:6789/0 conn(0x7f86cc154a80 legacy=0x7f86cc0c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.433+0000 7f86c9400640 1 -- 172.21.15.98:0/1701147710 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86cc071d70 con 0x7f86cc062720 2024-05-27T00:23:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.433+0000 7f86c9400640 1 --2- 172.21.15.98:0/1701147710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc062720 0x7f86cc072b30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f86b80097c0 tx=0x7f86b801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99051c90d91add6b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86c8a00640 1 -- 172.21.15.98:0/1701147710 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b80b0070 con 0x7f86cc062720 2024-05-27T00:23:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86c8a00640 1 -- 172.21.15.98:0/1701147710 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86b80ab600 con 0x7f86cc062720 2024-05-27T00:23:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86c8a00640 1 -- 172.21.15.98:0/1701147710 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b80ab9d0 con 0x7f86cc062720 2024-05-27T00:23:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 -- 172.21.15.98:0/1701147710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc062720 msgr2=0x7f86cc072b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 --2- 172.21.15.98:0/1701147710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc062720 0x7f86cc072b30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f86b80097c0 tx=0x7f86b801c410 comp rx=0 tx=0).stop 2024-05-27T00:23:40.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 -- 172.21.15.98:0/1701147710 shutdown_connections 2024-05-27T00:23:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 --2- 172.21.15.98:0/1701147710 >> v2:172.21.15.204:3300/0 conn(0x7f86cc15e6b0 0x7f86cc06ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 --2- 172.21.15.98:0/1701147710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc062720 0x7f86cc072b30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 -- 172.21.15.98:0/1701147710 >> 172.21.15.98:0/1701147710 conn(0x7f86cc0cc560 msgr2=0x7f86cc0cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 -- 172.21.15.98:0/1701147710 shutdown_connections 2024-05-27T00:23:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 -- 172.21.15.98:0/1701147710 wait complete. 2024-05-27T00:23:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 Processor -- start 2024-05-27T00:23:40.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 -- start start 2024-05-27T00:23:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc062720 0x7f86cc15b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86cc154a80 0x7f86cc15c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc15e6b0 0x7f86cc0cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86cc167210 con 0x7f86cc154a80 2024-05-27T00:23:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f86cc167090 con 0x7f86cc062720 2024-05-27T00:23:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86d1000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f86cc167390 con 0x7f86cc15e6b0 2024-05-27T00:23:40.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86c9400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86cc154a80 0x7f86cc15c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86c9400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86cc154a80 0x7f86cc15c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39120/0 (socket says 172.21.15.98:39120) 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86c9400640 1 -- 172.21.15.98:0/1482052720 learned_addr learned my addr 172.21.15.98:0/1482052720 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86c9e00640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc062720 0x7f86cc15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.437+0000 7f86ca800640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc15e6b0 0x7f86cc0cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c9400640 1 -- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc15e6b0 msgr2=0x7f86cc0cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c9400640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc15e6b0 0x7f86cc0cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c9400640 1 -- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc062720 msgr2=0x7f86cc15b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c9400640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc062720 0x7f86cc15b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c9400640 1 -- 172.21.15.98:0/1482052720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86cc15cb30 con 0x7f86cc154a80 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c9e00640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc062720 0x7f86cc15b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86ca800640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc15e6b0 0x7f86cc0cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c9400640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86cc154a80 0x7f86cc15c5f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f86b80097c0 tx=0x7f86b80191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c2000640 1 -- 172.21.15.98:0/1482052720 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b80ba070 con 0x7f86cc154a80 2024-05-27T00:23:40.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c2000640 1 -- 172.21.15.98:0/1482052720 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86b80b43f0 con 0x7f86cc154a80 2024-05-27T00:23:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c2000640 1 -- 172.21.15.98:0/1482052720 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b80b0040 con 0x7f86cc154a80 2024-05-27T00:23:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86d1000640 1 -- 172.21.15.98:0/1482052720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86cc1f4360 con 0x7f86cc154a80 2024-05-27T00:23:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86d1000640 1 -- 172.21.15.98:0/1482052720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86cc15bb10 con 0x7f86cc154a80 2024-05-27T00:23:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86d1000640 1 -- 172.21.15.98:0/1482052720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86cc075cb0 con 0x7f86cc154a80 2024-05-27T00:23:40.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c2000640 1 -- 172.21.15.98:0/1482052720 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f86b80b47a0 con 0x7f86cc154a80 2024-05-27T00:23:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.441+0000 7f86c2000640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f86ac0678c0 0x7f86ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.445+0000 7f86c2000640 1 -- 172.21.15.98:0/1482052720 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f86b811f1a0 con 0x7f86cc154a80 2024-05-27T00:23:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.445+0000 7f86c9e00640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f86ac0678c0 0x7f86ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:40.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.445+0000 7f86c9e00640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f86ac0678c0 0x7f86ac069d80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0012f0 tx=0x7f86bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:40.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.449+0000 7f86c2000640 1 -- 172.21.15.98:0/1482052720 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f86b8016090 con 0x7f86cc154a80 2024-05-27T00:23:40.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.645+0000 7f86d1000640 1 -- 172.21.15.98:0/1482052720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86cc0c3750 con 0x7f86cc154a80 2024-05-27T00:23:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.645+0000 7f86c2000640 1 -- 172.21.15.98:0/1482052720 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6003 (secure 0 0 0) 0x7f86b80ef930 con 0x7f86cc154a80 2024-05-27T00:23:40.648 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:40.648 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:40.226922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:23:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 36 2024-05-27T00:23:40.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 -- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f86ac0678c0 msgr2=0x7f86ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f86ac0678c0 0x7f86ac069d80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0012f0 tx=0x7f86bc009040 comp rx=0 tx=0).stop 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 -- 172.21.15.98:0/1482052720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86cc154a80 msgr2=0x7f86cc15c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86cc154a80 0x7f86cc15c5f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f86b80097c0 tx=0x7f86b80191b0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 -- 172.21.15.98:0/1482052720 shutdown_connections 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc15e6b0 0x7f86cc0cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86cc154a80 0x7f86cc15c5f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f86ac0678c0 0x7f86ac069d80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 --2- 172.21.15.98:0/1482052720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc062720 0x7f86cc15b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 -- 172.21.15.98:0/1482052720 >> 172.21.15.98:0/1482052720 conn(0x7f86cc0cc560 msgr2=0x7f86cc0cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 -- 172.21.15.98:0/1482052720 shutdown_connections 2024-05-27T00:23:40.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:40.649+0000 7f86d1000640 1 -- 172.21.15.98:0/1482052720 wait complete. 2024-05-27T00:23:40.669 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 36, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 36, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5502": { "addr": "172.21.15.98:6837/518550803", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 518550803, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 518550803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5502, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:23:40.226922+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5502 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:23:42.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:42.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f895f000640 1 Processor -- start 2024-05-27T00:23:42.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f895f000640 1 -- start start 2024-05-27T00:23:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f895f000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f89580cc1f0 0x7f89581592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f895f000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8958073570 0x7f89581559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f895f000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89580c1f60 con 0x7f89580cbe10 2024-05-27T00:23:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f895f000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8958159810 con 0x7f89580cc1f0 2024-05-27T00:23:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f895f000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f89580c6dd0 con 0x7f8958073570 2024-05-27T00:23:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f894f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f89580cc1f0 0x7f89581592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:42.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f8957e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f89580cbe10 0x7f89580c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52488/0 (socket says 172.21.15.98:52488) 2024-05-27T00:23:42.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f8957e00640 1 -- 172.21.15.98:0/3527276958 learned_addr learned my addr 172.21.15.98:0/3527276958 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:23:42.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f895d000640 1 --2- 172.21.15.98:0/3527276958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8958073570 0x7f89581559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:42.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f894f400640 1 -- 172.21.15.98:0/3527276958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8958073570 msgr2=0x7f89581559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:42.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f894f400640 1 --2- 172.21.15.98:0/3527276958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8958073570 0x7f89581559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:42.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f894f400640 1 -- 172.21.15.98:0/3527276958 >> v1:172.21.15.98:6789/0 conn(0x7f89580cbe10 legacy=0x7f89580c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:42.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f894f400640 1 -- 172.21.15.98:0/3527276958 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89580c70b0 con 0x7f89580cc1f0 2024-05-27T00:23:42.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.817+0000 7f8957400640 1 -- 172.21.15.98:0/3527276958 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 977742032 0 0) 0x7f89580c1f60 con 0x7f89580cbe10 2024-05-27T00:23:42.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f894f400640 1 --2- 172.21.15.98:0/3527276958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cc1f0 0x7f89581592d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8940009d90 tx=0x7f894001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc5a15bd2b61b21 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:42.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f8957400640 1 -- 172.21.15.98:0/3527276958 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89400b4070 con 0x7f89580cc1f0 2024-05-27T00:23:42.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f8957400640 1 -- 172.21.15.98:0/3527276958 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89400af600 con 0x7f89580cc1f0 2024-05-27T00:23:42.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f8957400640 1 -- 172.21.15.98:0/3527276958 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89400af9d0 con 0x7f89580cc1f0 2024-05-27T00:23:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f895f000640 1 -- 172.21.15.98:0/3527276958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cc1f0 msgr2=0x7f89581592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f895f000640 1 --2- 172.21.15.98:0/3527276958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cc1f0 0x7f89581592d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8940009d90 tx=0x7f894001f150 comp rx=0 tx=0).stop 2024-05-27T00:23:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f895f000640 1 -- 172.21.15.98:0/3527276958 shutdown_connections 2024-05-27T00:23:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f895f000640 1 --2- 172.21.15.98:0/3527276958 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8958073570 0x7f89581559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f895f000640 1 --2- 172.21.15.98:0/3527276958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cc1f0 0x7f89581592d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f895f000640 1 -- 172.21.15.98:0/3527276958 >> 172.21.15.98:0/3527276958 conn(0x7f8958067530 msgr2=0x7f895806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f895f000640 1 -- 172.21.15.98:0/3527276958 shutdown_connections 2024-05-27T00:23:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.821+0000 7f895f000640 1 -- 172.21.15.98:0/3527276958 wait complete. 2024-05-27T00:23:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 Processor -- start 2024-05-27T00:23:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 -- start start 2024-05-27T00:23:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8958073570 0x7f89581f0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cbe10 0x7f89580c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89580cc1f0 0x7f895814d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89580c1330 con 0x7f89580cbe10 2024-05-27T00:23:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f89580c1050 con 0x7f8958073570 2024-05-27T00:23:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f89580c79d0 con 0x7f89580cc1f0 2024-05-27T00:23:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f894f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cbe10 0x7f89580c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f894f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cbe10 0x7f89580c4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39140/0 (socket says 172.21.15.98:39140) 2024-05-27T00:23:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f894f400640 1 -- 172.21.15.98:0/115729445 learned_addr learned my addr 172.21.15.98:0/115729445 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895d000640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89580cc1f0 0x7f895814d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f8957e00640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8958073570 0x7f89581f0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f894f400640 1 -- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89580cc1f0 msgr2=0x7f895814d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f894f400640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89580cc1f0 0x7f895814d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f894f400640 1 -- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8958073570 msgr2=0x7f89581f0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f894f400640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8958073570 0x7f89581f0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f894f400640 1 -- 172.21.15.98:0/115729445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89581f1200 con 0x7f89580cbe10 2024-05-27T00:23:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f894f400640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cbe10 0x7f89580c4af0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f89400b2040 tx=0x7f894001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f8954c00640 1 -- 172.21.15.98:0/115729445 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89400b4040 con 0x7f89580cbe10 2024-05-27T00:23:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f8954c00640 1 -- 172.21.15.98:0/115729445 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89400c1070 con 0x7f89580cbe10 2024-05-27T00:23:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f8954c00640 1 -- 172.21.15.98:0/115729445 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940009450 con 0x7f89580cbe10 2024-05-27T00:23:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 -- 172.21.15.98:0/115729445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8958154790 con 0x7f89580cbe10 2024-05-27T00:23:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 -- 172.21.15.98:0/115729445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895814cd90 con 0x7f89580cbe10 2024-05-27T00:23:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.825+0000 7f895f000640 1 -- 172.21.15.98:0/115729445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8958075cb0 con 0x7f89580cbe10 2024-05-27T00:23:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.829+0000 7f8954c00640 1 -- 172.21.15.98:0/115729445 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f89400ba690 con 0x7f89580cbe10 2024-05-27T00:23:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.829+0000 7f8954c00640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f89340678f0 0x7f8934069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.829+0000 7f8954c00640 1 -- 172.21.15.98:0/115729445 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f89401249a0 con 0x7f89580cbe10 2024-05-27T00:23:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.829+0000 7f8957e00640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f89340678f0 0x7f8934069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.829+0000 7f8957e00640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f89340678f0 0x7f8934069db0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8948000f70 tx=0x7f8948007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:42.833+0000 7f8954c00640 1 -- 172.21.15.98:0/115729445 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8940018cb0 con 0x7f89580cbe10 2024-05-27T00:23:43.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.021+0000 7f895f000640 1 -- 172.21.15.98:0/115729445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8958153670 con 0x7f89580cbe10 2024-05-27T00:23:43.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.021+0000 7f8954c00640 1 -- 172.21.15.98:0/115729445 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6003 (secure 0 0 0) 0x7f89400f51c0 con 0x7f89580cbe10 2024-05-27T00:23:43.025 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:43.025 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:40.226922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:23:43.025 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 36 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 -- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f89340678f0 msgr2=0x7f8934069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f89340678f0 0x7f8934069db0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8948000f70 tx=0x7f8948007b60 comp rx=0 tx=0).stop 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 -- 172.21.15.98:0/115729445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cbe10 msgr2=0x7f89580c4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cbe10 0x7f89580c4af0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f89400b2040 tx=0x7f894001bef0 comp rx=0 tx=0).stop 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 -- 172.21.15.98:0/115729445 shutdown_connections 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89580cc1f0 0x7f895814d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89580cbe10 0x7f89580c4af0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f89340678f0 0x7f8934069db0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 --2- 172.21.15.98:0/115729445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8958073570 0x7f89581f0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 -- 172.21.15.98:0/115729445 >> 172.21.15.98:0/115729445 conn(0x7f8958067530 msgr2=0x7f8958152b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:43.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 -- 172.21.15.98:0/115729445 shutdown_connections 2024-05-27T00:23:43.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:43.025+0000 7f895f000640 1 -- 172.21.15.98:0/115729445 wait complete. 2024-05-27T00:23:43.046 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:23:45.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3873400640 1 Processor -- start 2024-05-27T00:23:45.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3873400640 1 -- start start 2024-05-27T00:23:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3873400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f386c154830 0x7f386c15b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3873400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f386c0cc010 0x7f386c071e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3873400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f386c0c1a90 con 0x7f386c0cbbb0 2024-05-27T00:23:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3873400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f386c0c2b30 con 0x7f386c0cc010 2024-05-27T00:23:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3873400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f386c0c26a0 con 0x7f386c154830 2024-05-27T00:23:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3871400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f386c0cc010 0x7f386c071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3871400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f386c0cc010 0x7f386c071e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55794/0 (socket says 172.21.15.98:55794) 2024-05-27T00:23:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3871400640 1 -- 172.21.15.98:0/2226698737 learned_addr learned my addr 172.21.15.98:0/2226698737 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:45.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3870a00640 1 --2- 172.21.15.98:0/2226698737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f386c154830 0x7f386c15b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3871400640 1 -- 172.21.15.98:0/2226698737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f386c154830 msgr2=0x7f386c15b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3871400640 1 --2- 172.21.15.98:0/2226698737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f386c154830 0x7f386c15b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3871400640 1 -- 172.21.15.98:0/2226698737 >> v1:172.21.15.204:6789/0 conn(0x7f386c0cbbb0 legacy=0x7f386c0c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:45.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.193+0000 7f3871400640 1 -- 172.21.15.98:0/2226698737 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386c0c1d70 con 0x7f386c0cc010 2024-05-27T00:23:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f3871400640 1 --2- 172.21.15.98:0/2226698737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cc010 0x7f386c071e90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3864009870 tx=0x7f386401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa88945a4f744d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:45.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f386b400640 1 -- 172.21.15.98:0/2226698737 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38640b2070 con 0x7f386c0cc010 2024-05-27T00:23:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f386b400640 1 -- 172.21.15.98:0/2226698737 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38640a9070 con 0x7f386c0cc010 2024-05-27T00:23:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f386b400640 1 -- 172.21.15.98:0/2226698737 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38640ad880 con 0x7f386c0cc010 2024-05-27T00:23:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f3873400640 1 -- 172.21.15.98:0/2226698737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cc010 msgr2=0x7f386c071e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f3873400640 1 --2- 172.21.15.98:0/2226698737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cc010 0x7f386c071e90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3864009870 tx=0x7f386401c560 comp rx=0 tx=0).stop 2024-05-27T00:23:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f3873400640 1 -- 172.21.15.98:0/2226698737 shutdown_connections 2024-05-27T00:23:45.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f3873400640 1 --2- 172.21.15.98:0/2226698737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cc010 0x7f386c071e90 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f3873400640 1 --2- 172.21.15.98:0/2226698737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f386c154830 0x7f386c15b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f3873400640 1 -- 172.21.15.98:0/2226698737 >> 172.21.15.98:0/2226698737 conn(0x7f386c067530 msgr2=0x7f386c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f3873400640 1 -- 172.21.15.98:0/2226698737 shutdown_connections 2024-05-27T00:23:45.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.197+0000 7f3873400640 1 -- 172.21.15.98:0/2226698737 wait complete. 2024-05-27T00:23:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 Processor -- start 2024-05-27T00:23:45.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 -- start start 2024-05-27T00:23:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cbbb0 0x7f386c155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f386c0cc010 0x7f386c0af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f386c154830 0x7f386c0b0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f386c0c7150 con 0x7f386c0cbbb0 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f386c0c6fd0 con 0x7f386c154830 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f386c167a70 con 0x7f386c0cc010 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3870a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cbbb0 0x7f386c155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3870a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cbbb0 0x7f386c155420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55810/0 (socket says 172.21.15.98:55810) 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3870a00640 1 -- 172.21.15.98:0/1999920805 learned_addr learned my addr 172.21.15.98:0/1999920805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f386be00640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f386c0cc010 0x7f386c0af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3871400640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f386c154830 0x7f386c0b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3870a00640 1 -- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f386c0cc010 msgr2=0x7f386c0af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3870a00640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f386c0cc010 0x7f386c0af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3870a00640 1 -- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f386c154830 msgr2=0x7f386c0b0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3870a00640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f386c154830 0x7f386c0b0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3870a00640 1 -- 172.21.15.98:0/1999920805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386c0aeb20 con 0x7f386c0cbbb0 2024-05-27T00:23:45.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3871400640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f386c154830 0x7f386c0b0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:23:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3870a00640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cbbb0 0x7f386c155420 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f385c00efc0 tx=0x7f385c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3868c00640 1 -- 172.21.15.98:0/1999920805 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c018020 con 0x7f386c0cbbb0 2024-05-27T00:23:45.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3868c00640 1 -- 172.21.15.98:0/1999920805 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f385c014480 con 0x7f386c0cbbb0 2024-05-27T00:23:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 -- 172.21.15.98:0/1999920805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f386c15d700 con 0x7f386c0cbbb0 2024-05-27T00:23:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3868c00640 1 -- 172.21.15.98:0/1999920805 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c014850 con 0x7f386c0cbbb0 2024-05-27T00:23:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 -- 172.21.15.98:0/1999920805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f386c15dca0 con 0x7f386c0cbbb0 2024-05-27T00:23:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.201+0000 7f3873400640 1 -- 172.21.15.98:0/1999920805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f386c0c1300 con 0x7f386c0cbbb0 2024-05-27T00:23:45.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.205+0000 7f3868c00640 1 -- 172.21.15.98:0/1999920805 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f385c01b020 con 0x7f386c0cbbb0 2024-05-27T00:23:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.205+0000 7f3868c00640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f384c0678c0 0x7f384c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.205+0000 7f3868c00640 1 -- 172.21.15.98:0/1999920805 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f385c086db0 con 0x7f386c0cbbb0 2024-05-27T00:23:45.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.205+0000 7f386be00640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f384c0678c0 0x7f384c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:45.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.205+0000 7f386be00640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f384c0678c0 0x7f384c069d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f386c153640 tx=0x7f3858009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:45.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.209+0000 7f3868c00640 1 -- 172.21.15.98:0/1999920805 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f385c08b070 con 0x7f386c0cbbb0 2024-05-27T00:23:45.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.445+0000 7f3873400640 1 -- 172.21.15.98:0/1999920805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f386c0c7420 con 0x7f386c0cbbb0 2024-05-27T00:23:45.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.445+0000 7f3868c00640 1 -- 172.21.15.98:0/1999920805 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6003 (secure 0 0 0) 0x7f385c022030 con 0x7f386c0cbbb0 2024-05-27T00:23:45.448 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:45.448 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:40.226922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:23:45.449 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 36 2024-05-27T00:23:45.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 -- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f384c0678c0 msgr2=0x7f384c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:45.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f384c0678c0 0x7f384c069d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f386c153640 tx=0x7f3858009040 comp rx=0 tx=0).stop 2024-05-27T00:23:45.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 -- 172.21.15.98:0/1999920805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cbbb0 msgr2=0x7f386c155420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:45.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cbbb0 0x7f386c155420 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f385c00efc0 tx=0x7f385c00c490 comp rx=0 tx=0).stop 2024-05-27T00:23:45.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 -- 172.21.15.98:0/1999920805 shutdown_connections 2024-05-27T00:23:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f384c0678c0 0x7f384c069d80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f386c154830 0x7f386c0b0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f386c0cc010 0x7f386c0af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 --2- 172.21.15.98:0/1999920805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f386c0cbbb0 0x7f386c155420 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 -- 172.21.15.98:0/1999920805 >> 172.21.15.98:0/1999920805 conn(0x7f386c067530 msgr2=0x7f386c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 -- 172.21.15.98:0/1999920805 shutdown_connections 2024-05-27T00:23:45.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:45.449+0000 7f3873400640 1 -- 172.21.15.98:0/1999920805 wait complete. 2024-05-27T00:23:45.481 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:23:47.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:47.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.621+0000 7fd9fe400640 1 Processor -- start 2024-05-27T00:23:47.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9fe400640 1 -- start start 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9fe400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd9f80cbe10 0x7fd9f81592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9fe400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9f80cc1f0 0x7fd9f80c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9fe400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9f8073570 0x7fd9f81559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9fe400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd9f8159810 con 0x7fd9f8073570 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9fe400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd9f80c6dd0 con 0x7fd9f80cbe10 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9fe400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f80c70b0 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9f7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9f8073570 0x7fd9f81559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9f7400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd9f80cbe10 0x7fd9f81592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9f7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd9f8073570 0x7fd9f81559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55814/0 (socket says 172.21.15.98:55814) 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9f7e00640 1 -- 172.21.15.98:0/2691407857 learned_addr learned my addr 172.21.15.98:0/2691407857 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:47.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9eea00640 1 --2- 172.21.15.98:0/2691407857 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9f80cc1f0 0x7fd9f80c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:47.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9f7e00640 1 -- 172.21.15.98:0/2691407857 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9f80cc1f0 msgr2=0x7fd9f80c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:47.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9f7e00640 1 --2- 172.21.15.98:0/2691407857 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9f80cc1f0 0x7fd9f80c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9f7e00640 1 -- 172.21.15.98:0/2691407857 >> v2:172.21.15.204:3300/0 conn(0x7fd9f80cbe10 msgr2=0x7fd9f81592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:47.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9f7e00640 1 --2- 172.21.15.98:0/2691407857 >> v2:172.21.15.204:3300/0 conn(0x7fd9f80cbe10 0x7fd9f81592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.625+0000 7fd9f7e00640 1 -- 172.21.15.98:0/2691407857 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f80c7390 con 0x7fd9f8073570 2024-05-27T00:23:47.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.629+0000 7fd9f7e00640 1 --2- 172.21.15.98:0/2691407857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f8073570 0x7fd9f81559e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd9f80cc760 tx=0x7fd9f001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e58fc174ecf53e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:47.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.629+0000 7fd9f6a00640 1 -- 172.21.15.98:0/2691407857 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f00b5070 con 0x7fd9f8073570 2024-05-27T00:23:47.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.629+0000 7fd9f6a00640 1 -- 172.21.15.98:0/2691407857 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9f00a89d0 con 0x7fd9f8073570 2024-05-27T00:23:47.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.629+0000 7fd9f6a00640 1 -- 172.21.15.98:0/2691407857 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f00a8da0 con 0x7fd9f8073570 2024-05-27T00:23:47.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.629+0000 7fd9fe400640 1 -- 172.21.15.98:0/2691407857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f8073570 msgr2=0x7fd9f81559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.629+0000 7fd9fe400640 1 --2- 172.21.15.98:0/2691407857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f8073570 0x7fd9f81559e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd9f80cc760 tx=0x7fd9f001ed10 comp rx=0 tx=0).stop 2024-05-27T00:23:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 -- 172.21.15.98:0/2691407857 shutdown_connections 2024-05-27T00:23:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 --2- 172.21.15.98:0/2691407857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f8073570 0x7fd9f81559e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 --2- 172.21.15.98:0/2691407857 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9f80cc1f0 0x7fd9f80c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 --2- 172.21.15.98:0/2691407857 >> v2:172.21.15.204:3300/0 conn(0x7fd9f80cbe10 0x7fd9f81592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 -- 172.21.15.98:0/2691407857 >> 172.21.15.98:0/2691407857 conn(0x7fd9f8067530 msgr2=0x7fd9f806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 -- 172.21.15.98:0/2691407857 shutdown_connections 2024-05-27T00:23:47.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 -- 172.21.15.98:0/2691407857 wait complete. 2024-05-27T00:23:47.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 Processor -- start 2024-05-27T00:23:47.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 -- start start 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9f8073570 0x7fd9f81569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f80cbe10 0x7fd9f8072ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f80cc1f0 0x7fd9f8151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f80c79d0 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f80c1330 con 0x7fd9f80cbe10 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9fe400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f80c6990 con 0x7fd9f8073570 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9f7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f80cc1f0 0x7fd9f8151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9f7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f80cc1f0 0x7fd9f8151c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55820/0 (socket says 172.21.15.98:55820) 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9f7e00640 1 -- 172.21.15.98:0/2986453318 learned_addr learned my addr 172.21.15.98:0/2986453318 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9eea00640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f80cbe10 0x7fd9f8072ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9f7400640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9f8073570 0x7fd9f81569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9f7e00640 1 -- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9f8073570 msgr2=0x7fd9f81569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:47.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9f7e00640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9f8073570 0x7fd9f81569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9f7e00640 1 -- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f80cbe10 msgr2=0x7fd9f8072ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:47.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9f7e00640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f80cbe10 0x7fd9f8072ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9f7e00640 1 -- 172.21.15.98:0/2986453318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f8156fd0 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.633+0000 7fd9f7e00640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f80cc1f0 0x7fd9f8151c50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd9f001d3c0 tx=0x7fd9f001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:47.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.637+0000 7fd9efe00640 1 -- 172.21.15.98:0/2986453318 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f00b5040 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.637+0000 7fd9efe00640 1 -- 172.21.15.98:0/2986453318 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9f00be070 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.637+0000 7fd9fe400640 1 -- 172.21.15.98:0/2986453318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f81f04a0 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.637+0000 7fd9efe00640 1 -- 172.21.15.98:0/2986453318 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f00b9840 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.637+0000 7fd9fe400640 1 -- 172.21.15.98:0/2986453318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f80b0b20 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.637+0000 7fd9fe400640 1 -- 172.21.15.98:0/2986453318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f81f0670 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.637+0000 7fd9efe00640 1 -- 172.21.15.98:0/2986453318 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd9f00b99e0 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.637+0000 7fd9efe00640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9d40678c0 0x7fd9d4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:47.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.637+0000 7fd9efe00640 1 -- 172.21.15.98:0/2986453318 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7fd9f01232c0 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.641+0000 7fd9f7400640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9d40678c0 0x7fd9d4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:47.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.641+0000 7fd9f7400640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9d40678c0 0x7fd9d4069d80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd9e80074a0 tx=0x7fd9e800d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:47.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.645+0000 7fd9efe00640 1 -- 172.21.15.98:0/2986453318 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd9f0018870 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 -- 172.21.15.98:0/2986453318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9f80c8c80 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9efe00640 1 -- 172.21.15.98:0/2986453318 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6003 (secure 0 0 0) 0x7fd9f00f3b80 con 0x7fd9f80cc1f0 2024-05-27T00:23:47.850 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:47.850 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:40.226922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:23:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 36 2024-05-27T00:23:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 -- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9d40678c0 msgr2=0x7fd9d4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9d40678c0 0x7fd9d4069d80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd9e80074a0 tx=0x7fd9e800d040 comp rx=0 tx=0).stop 2024-05-27T00:23:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 -- 172.21.15.98:0/2986453318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f80cc1f0 msgr2=0x7fd9f8151c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f80cc1f0 0x7fd9f8151c50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd9f001d3c0 tx=0x7fd9f001bab0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 -- 172.21.15.98:0/2986453318 shutdown_connections 2024-05-27T00:23:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9f80cc1f0 0x7fd9f8151c50 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd9d40678c0 0x7fd9d4069d80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f80cbe10 0x7fd9f8072ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 --2- 172.21.15.98:0/2986453318 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9f8073570 0x7fd9f81569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 -- 172.21.15.98:0/2986453318 >> 172.21.15.98:0/2986453318 conn(0x7fd9f8067530 msgr2=0x7fd9f81533c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.845+0000 7fd9fe400640 1 -- 172.21.15.98:0/2986453318 shutdown_connections 2024-05-27T00:23:47.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:47.849+0000 7fd9fe400640 1 -- 172.21.15.98:0/2986453318 wait complete. 2024-05-27T00:23:47.876 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:23:48.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:23:48.319 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:23:49.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:50.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.017+0000 7fc03de00640 1 Processor -- start 2024-05-27T00:23:50.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.017+0000 7fc03de00640 1 -- start start 2024-05-27T00:23:50.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc03de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806f760 0x7fc0380c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc03de00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0380cb8b0 con 0x7fc0380c8240 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc03de00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0380c8bb0 con 0x7fc0380cdf90 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc03de00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc038158b90 con 0x7fc03806f760 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc02e000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc0380c8240 0x7fc0380cb1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41622/0 (socket says 172.21.15.98:41622) 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc02e000640 1 -- 172.21.15.98:0/2366246574 learned_addr learned my addr 172.21.15.98:0/2366246574 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc037400640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806f760 0x7fc0380c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2369715582 0 0) 0x7fc0380cb8b0 con 0x7fc0380c8240 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc01c003530 con 0x7fc0380c8240 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 221522059 0 0) 0x7fc01c003530 con 0x7fc0380c8240 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806f760 msgr2=0x7fc0380c3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806f760 0x7fc0380c3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 >> v1:172.21.15.204:6789/0 conn(0x7fc0380cdf90 legacy=0x7fc0380caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc038158560 con 0x7fc0380c8240 2024-05-27T00:23:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fc020001680 con 0x7fc0380c8240 2024-05-27T00:23:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 >> v1:172.21.15.98:6789/0 conn(0x7fc0380c8240 legacy=0x7fc0380cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c0039f0 0x7fc01c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806f760 0x7fc01c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc01c004d50 0x7fc01c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc038158b90 con 0x7fc01c0039f0 2024-05-27T00:23:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c00b9d0 con 0x7fc01c004d50 2024-05-27T00:23:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc01c00bbd0 con 0x7fc03806f760 2024-05-27T00:23:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036a00640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c0039f0 0x7fc01c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc037400640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc01c004d50 0x7fc01c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036a00640 1 -- 172.21.15.98:0/2366246574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806f760 msgr2=0x7fc01c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036a00640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806f760 0x7fc01c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036a00640 1 -- 172.21.15.98:0/2366246574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc01c004d50 msgr2=0x7fc01c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036a00640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc01c004d50 0x7fc01c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.021+0000 7fc036a00640 1 -- 172.21.15.98:0/2366246574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c003530 con 0x7fc01c0039f0 2024-05-27T00:23:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc036a00640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c0039f0 0x7fc01c003dd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc028001860 tx=0x7fc02801ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d013699694a7eeed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0280b5070 con 0x7fc01c0039f0 2024-05-27T00:23:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0280a87f0 con 0x7fc01c0039f0 2024-05-27T00:23:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc036000640 1 -- 172.21.15.98:0/2366246574 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028001460 con 0x7fc01c0039f0 2024-05-27T00:23:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc03de00640 1 -- 172.21.15.98:0/2366246574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c0039f0 msgr2=0x7fc01c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc03de00640 1 --2- 172.21.15.98:0/2366246574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc01c0039f0 0x7fc01c003dd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc028001860 tx=0x7fc02801ebd0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc037400640 1 -- 172.21.15.98:0/2366246574 reap_dead start 2024-05-27T00:23:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc03de00640 1 -- 172.21.15.98:0/2366246574 shutdown_connections 2024-05-27T00:23:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc03de00640 1 -- 172.21.15.98:0/2366246574 >> 172.21.15.98:0/2366246574 conn(0x7fc03806d680 msgr2=0x7fc03806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc03de00640 1 -- 172.21.15.98:0/2366246574 shutdown_connections 2024-05-27T00:23:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.025+0000 7fc03de00640 1 -- 172.21.15.98:0/2366246574 wait complete. 2024-05-27T00:23:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc03de00640 1 Processor -- start 2024-05-27T00:23:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc03de00640 1 -- start start 2024-05-27T00:23:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc03de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc01c0039f0 0x7fc0380b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc03de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806c160 0x7fc03806c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc03de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038150960 0x7fc038150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc03de00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0380c4fc0 con 0x7fc038150960 2024-05-27T00:23:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc03de00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc0380c3640 con 0x7fc01c0039f0 2024-05-27T00:23:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc03de00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc038158820 con 0x7fc03806c160 2024-05-27T00:23:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc037400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038150960 0x7fc038150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc037400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038150960 0x7fc038150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55844/0 (socket says 172.21.15.98:55844) 2024-05-27T00:23:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc037400640 1 -- 172.21.15.98:0/2736928154 learned_addr learned my addr 172.21.15.98:0/2736928154 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc036a00640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc01c0039f0 0x7fc0380b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc02e000640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806c160 0x7fc03806c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc037400640 1 -- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806c160 msgr2=0x7fc03806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc037400640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806c160 0x7fc03806c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc037400640 1 -- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc01c0039f0 msgr2=0x7fc0380b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc037400640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc01c0039f0 0x7fc0380b3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc037400640 1 -- 172.21.15.98:0/2736928154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03806cb00 con 0x7fc038150960 2024-05-27T00:23:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc02e000640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806c160 0x7fc03806c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:23:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc037400640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038150960 0x7fc038150de0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc030009ed0 tx=0x7fc0300115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc02fe00640 1 -- 172.21.15.98:0/2736928154 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030011de0 con 0x7fc038150960 2024-05-27T00:23:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc02fe00640 1 -- 172.21.15.98:0/2736928154 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc03001b070 con 0x7fc038150960 2024-05-27T00:23:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc02fe00640 1 -- 172.21.15.98:0/2736928154 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0300039a0 con 0x7fc038150960 2024-05-27T00:23:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc03de00640 1 -- 172.21.15.98:0/2736928154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc038151320 con 0x7fc038150960 2024-05-27T00:23:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.029+0000 7fc03de00640 1 -- 172.21.15.98:0/2736928154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0380ab3b0 con 0x7fc038150960 2024-05-27T00:23:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.033+0000 7fc03de00640 1 -- 172.21.15.98:0/2736928154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc038075cb0 con 0x7fc038150960 2024-05-27T00:23:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.033+0000 7fc02fe00640 1 -- 172.21.15.98:0/2736928154 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc030003b40 con 0x7fc038150960 2024-05-27T00:23:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.033+0000 7fc02fe00640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0140678c0 0x7fc014069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.033+0000 7fc02fe00640 1 -- 172.21.15.98:0/2736928154 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7fc030086650 con 0x7fc038150960 2024-05-27T00:23:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.033+0000 7fc036a00640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0140678c0 0x7fc014069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.037+0000 7fc036a00640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0140678c0 0x7fc014069d80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc0280b2000 tx=0x7fc0280aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.041+0000 7fc02fe00640 1 -- 172.21.15.98:0/2736928154 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc030025020 con 0x7fc038150960 2024-05-27T00:23:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.241+0000 7fc03de00640 1 -- 172.21.15.98:0/2736928154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0380c8db0 con 0x7fc038150960 2024-05-27T00:23:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.241+0000 7fc02fe00640 1 -- 172.21.15.98:0/2736928154 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6003 (secure 0 0 0) 0x7fc030056f10 con 0x7fc038150960 2024-05-27T00:23:50.244 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:50.244 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:40.226922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:23:50.244 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 36 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 -- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0140678c0 msgr2=0x7fc014069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0140678c0 0x7fc014069d80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc0280b2000 tx=0x7fc0280aa070 comp rx=0 tx=0).stop 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 -- 172.21.15.98:0/2736928154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038150960 msgr2=0x7fc038150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038150960 0x7fc038150de0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc030009ed0 tx=0x7fc0300115a0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 -- 172.21.15.98:0/2736928154 shutdown_connections 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038150960 0x7fc038150de0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc03806c160 0x7fc03806c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0140678c0 0x7fc014069d80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 --2- 172.21.15.98:0/2736928154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc01c0039f0 0x7fc0380b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 -- 172.21.15.98:0/2736928154 >> 172.21.15.98:0/2736928154 conn(0x7fc03806d680 msgr2=0x7fc03815bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 -- 172.21.15.98:0/2736928154 shutdown_connections 2024-05-27T00:23:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:50.245+0000 7fc03de00640 1 -- 172.21.15.98:0/2736928154 wait complete. 2024-05-27T00:23:50.266 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:23:52.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:52.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f5354200640 1 Processor -- start 2024-05-27T00:23:52.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f5354200640 1 -- start start 2024-05-27T00:23:52.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f5354200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c067130 0x7f534c1591d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:52.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f5354200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f534c0677a0 con 0x7f534c06c080 2024-05-27T00:23:52.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f5354200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f534c067a80 con 0x7f534c152be0 2024-05-27T00:23:52.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f5354200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f534c159710 con 0x7f534c067130 2024-05-27T00:23:52.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f5352200640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f534c06c080 0x7f534c06c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41624/0 (socket says 172.21.15.98:41624) 2024-05-27T00:23:52.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f5352200640 1 -- 172.21.15.98:0/2323740726 learned_addr learned my addr 172.21.15.98:0/2323740726 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f5350e00640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c067130 0x7f534c1591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3751008830 0 0) 0x7f534c0677a0 con 0x7f534c06c080 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5334003530 con 0x7f534c06c080 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1481665021 0 0) 0x7f5334003530 con 0x7f534c06c080 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c067130 msgr2=0x7f534c1591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c067130 0x7f534c1591d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 >> v1:172.21.15.204:6789/0 conn(0x7f534c152be0 legacy=0x7f534c0c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534c06b5a0 con 0x7f534c06c080 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f5344001a00 con 0x7f534c06c080 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 >> v1:172.21.15.98:6789/0 conn(0x7f534c06c080 legacy=0x7f534c06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:52.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c067130 0x7f53340039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5334003fc0 0x7f5334004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5334004d50 0x7f53340051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f534c159710 con 0x7f5334003fc0 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f533400bc60 con 0x7f5334004d50 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f533400be60 con 0x7f534c067130 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.413+0000 7f5350e00640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5334003fc0 0x7f5334004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5352200640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5334004d50 0x7f53340051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5351800640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c067130 0x7f53340039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5350e00640 1 -- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c067130 msgr2=0x7f53340039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5350e00640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c067130 0x7f53340039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5350e00640 1 -- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5334004d50 msgr2=0x7f53340051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5350e00640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5334004d50 0x7f53340051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:52.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5350e00640 1 -- 172.21.15.98:0/2323740726 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5334003530 con 0x7f5334003fc0 2024-05-27T00:23:52.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5350e00640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5334003fc0 0x7f5334004810 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5338009ae0 tx=0x7f533801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119ba41c3bf85974 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:52.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53380096b0 con 0x7f5334003fc0 2024-05-27T00:23:52.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53380a6450 con 0x7f5334003fc0 2024-05-27T00:23:52.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f534be00640 1 -- 172.21.15.98:0/2323740726 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53380a6840 con 0x7f5334003fc0 2024-05-27T00:23:52.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5354200640 1 -- 172.21.15.98:0/2323740726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5334003fc0 msgr2=0x7f5334004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:52.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5354200640 1 --2- 172.21.15.98:0/2323740726 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5334003fc0 0x7f5334004810 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5338009ae0 tx=0x7f533801ca70 comp rx=0 tx=0).stop 2024-05-27T00:23:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5352200640 1 -- 172.21.15.98:0/2323740726 reap_dead start 2024-05-27T00:23:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5354200640 1 -- 172.21.15.98:0/2323740726 shutdown_connections 2024-05-27T00:23:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.417+0000 7f5354200640 1 -- 172.21.15.98:0/2323740726 >> 172.21.15.98:0/2323740726 conn(0x7f534c0c9af0 msgr2=0x7f534c0c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 -- 172.21.15.98:0/2323740726 shutdown_connections 2024-05-27T00:23:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 -- 172.21.15.98:0/2323740726 wait complete. 2024-05-27T00:23:52.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 Processor -- start 2024-05-27T00:23:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 -- start start 2024-05-27T00:23:52.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5334003fc0 0x7f534c0c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:52.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f534c0c38e0 0x7f534c1f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:52.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c1f7600 0x7f534c14ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:52.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f534c0c8100 con 0x7f534c0c38e0 2024-05-27T00:23:52.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f534c067d60 con 0x7f5334003fc0 2024-05-27T00:23:52.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f534c0c8280 con 0x7f534c1f7600 2024-05-27T00:23:52.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5350e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f534c0c38e0 0x7f534c1f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:52.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5350e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f534c0c38e0 0x7f534c1f70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55862/0 (socket says 172.21.15.98:55862) 2024-05-27T00:23:52.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5350e00640 1 -- 172.21.15.98:0/3602861602 learned_addr learned my addr 172.21.15.98:0/3602861602 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:52.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5352200640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c1f7600 0x7f534c14ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:52.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5351800640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5334003fc0 0x7f534c0c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:52.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5350e00640 1 -- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c1f7600 msgr2=0x7f534c14ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:52.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5350e00640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c1f7600 0x7f534c14ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:52.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5350e00640 1 -- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5334003fc0 msgr2=0x7f534c0c33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:52.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5350e00640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5334003fc0 0x7f534c0c33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:52.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5350e00640 1 -- 172.21.15.98:0/3602861602 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534c0c5f90 con 0x7f534c0c38e0 2024-05-27T00:23:52.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5350e00640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f534c0c38e0 0x7f534c1f70c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f533801c8a0 tx=0x7f5334005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:52.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5349600640 1 -- 172.21.15.98:0/3602861602 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53380be070 con 0x7f534c0c38e0 2024-05-27T00:23:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5349600640 1 -- 172.21.15.98:0/3602861602 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53380a6600 con 0x7f534c0c38e0 2024-05-27T00:23:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5349600640 1 -- 172.21.15.98:0/3602861602 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53380096b0 con 0x7f534c0c38e0 2024-05-27T00:23:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 -- 172.21.15.98:0/3602861602 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534c0c6200 con 0x7f534c0c38e0 2024-05-27T00:23:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.421+0000 7f5354200640 1 -- 172.21.15.98:0/3602861602 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f534c154390 con 0x7f534c0c38e0 2024-05-27T00:23:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.425+0000 7f5354200640 1 -- 172.21.15.98:0/3602861602 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f534c075cb0 con 0x7f534c0c38e0 2024-05-27T00:23:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.425+0000 7f5349600640 1 -- 172.21.15.98:0/3602861602 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f53380a7430 con 0x7f534c0c38e0 2024-05-27T00:23:52.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.425+0000 7f5349600640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f532c0678c0 0x7f532c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:52.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.425+0000 7f5349600640 1 -- 172.21.15.98:0/3602861602 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f533811f950 con 0x7f534c0c38e0 2024-05-27T00:23:52.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.425+0000 7f5351800640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f532c0678c0 0x7f532c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.429+0000 7f5351800640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f532c0678c0 0x7f532c069d80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5340000c10 tx=0x7f5340009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.429+0000 7f5349600640 1 -- 172.21.15.98:0/3602861602 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f53380ac020 con 0x7f534c0c38e0 2024-05-27T00:23:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 -- 172.21.15.98:0/3602861602 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f534c0ca930 con 0x7f534c0c38e0 2024-05-27T00:23:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5349600640 1 -- 172.21.15.98:0/3602861602 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6007 (secure 0 0 0) 0x7f5338016780 con 0x7f534c0c38e0 2024-05-27T00:23:52.623 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:52.623 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:51.374512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:23:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-27T00:23:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 -- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f532c0678c0 msgr2=0x7f532c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f532c0678c0 0x7f532c069d80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5340000c10 tx=0x7f5340009040 comp rx=0 tx=0).stop 2024-05-27T00:23:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 -- 172.21.15.98:0/3602861602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f534c0c38e0 msgr2=0x7f534c1f70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f534c0c38e0 0x7f534c1f70c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f533801c8a0 tx=0x7f5334005840 comp rx=0 tx=0).stop 2024-05-27T00:23:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 -- 172.21.15.98:0/3602861602 shutdown_connections 2024-05-27T00:23:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f534c1f7600 0x7f534c14ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f534c0c38e0 0x7f534c1f70c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f532c0678c0 0x7f532c069d80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 --2- 172.21.15.98:0/3602861602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5334003fc0 0x7f534c0c33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 -- 172.21.15.98:0/3602861602 >> 172.21.15.98:0/3602861602 conn(0x7f534c0c9af0 msgr2=0x7f534c0cb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 -- 172.21.15.98:0/3602861602 shutdown_connections 2024-05-27T00:23:52.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:52.621+0000 7f5354200640 1 -- 172.21.15.98:0/3602861602 wait complete. 2024-05-27T00:23:52.644 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5502": { "addr": "172.21.15.98:6837/518550803", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 518550803, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 518550803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5502, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:23:51.374512+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5502 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:23:54.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1124000640 1 Processor -- start 2024-05-27T00:23:54.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1124000640 1 -- start start 2024-05-27T00:23:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1124000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f111c070fe0 0x7f111c154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1124000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111c15a950 con 0x7f111c070c00 2024-05-27T00:23:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1124000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111c15e060 con 0x7f111c0cb570 2024-05-27T00:23:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1124000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f111c15e340 con 0x7f111c070fe0 2024-05-27T00:23:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1121600640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f111c070c00 0x7f111c0c8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47096/0 (socket says 172.21.15.98:47096) 2024-05-27T00:23:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1121600640 1 -- 172.21.15.98:0/3360709894 learned_addr learned my addr 172.21.15.98:0/3360709894 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:23:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1120c00640 1 --2- 172.21.15.98:0/3360709894 >> v2:172.21.15.98:3300/0 conn(0x7f111c070fe0 0x7f111c154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f111be00640 1 -- 172.21.15.98:0/3360709894 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3990878832 0 0) 0x7f111c15a950 con 0x7f111c070c00 2024-05-27T00:23:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1120c00640 1 -- 172.21.15.98:0/3360709894 >> v1:172.21.15.204:6789/0 conn(0x7f111c0cb570 legacy=0x7f111c15ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1120c00640 1 -- 172.21.15.98:0/3360709894 >> v1:172.21.15.98:6789/0 conn(0x7f111c070c00 legacy=0x7f111c0c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.793+0000 7f1120c00640 1 -- 172.21.15.98:0/3360709894 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111c15e620 con 0x7f111c070fe0 2024-05-27T00:23:54.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f1120c00640 1 --2- 172.21.15.98:0/3360709894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070fe0 0x7f111c154780 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f11080097c0 tx=0x7f110801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7d053295cfba57 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f111be00640 1 -- 172.21.15.98:0/3360709894 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11080b5070 con 0x7f111c070fe0 2024-05-27T00:23:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f111be00640 1 -- 172.21.15.98:0/3360709894 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11080a8990 con 0x7f111c070fe0 2024-05-27T00:23:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f111be00640 1 -- 172.21.15.98:0/3360709894 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11080a8d60 con 0x7f111c070fe0 2024-05-27T00:23:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f1124000640 1 -- 172.21.15.98:0/3360709894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070fe0 msgr2=0x7f111c154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f1124000640 1 --2- 172.21.15.98:0/3360709894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070fe0 0x7f111c154780 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f11080097c0 tx=0x7f110801ec90 comp rx=0 tx=0).stop 2024-05-27T00:23:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f1124000640 1 -- 172.21.15.98:0/3360709894 shutdown_connections 2024-05-27T00:23:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f1124000640 1 --2- 172.21.15.98:0/3360709894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070fe0 0x7f111c154780 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f1124000640 1 -- 172.21.15.98:0/3360709894 >> 172.21.15.98:0/3360709894 conn(0x7f111c072f90 msgr2=0x7f111c0733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f1124000640 1 -- 172.21.15.98:0/3360709894 shutdown_connections 2024-05-27T00:23:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.797+0000 7f1124000640 1 -- 172.21.15.98:0/3360709894 wait complete. 2024-05-27T00:23:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 Processor -- start 2024-05-27T00:23:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 -- start start 2024-05-27T00:23:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070c00 0x7f111c0c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f111c070fe0 0x7f111c15bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f111c0cb570 0x7f111c14be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f111c0c9110 con 0x7f111c070c00 2024-05-27T00:23:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f111c15ebe0 con 0x7f111c070fe0 2024-05-27T00:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f111c15b560 con 0x7f111c0cb570 2024-05-27T00:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1121600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070c00 0x7f111c0c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1121600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070c00 0x7f111c0c71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39736/0 (socket says 172.21.15.98:39736) 2024-05-27T00:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1121600640 1 -- 172.21.15.98:0/2132644541 learned_addr learned my addr 172.21.15.98:0/2132644541 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1120c00640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f111c070fe0 0x7f111c15bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1122000640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f111c0cb570 0x7f111c14be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1121600640 1 -- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f111c0cb570 msgr2=0x7f111c14be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1121600640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f111c0cb570 0x7f111c14be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1121600640 1 -- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f111c070fe0 msgr2=0x7f111c15bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1121600640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f111c070fe0 0x7f111c15bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1121600640 1 -- 172.21.15.98:0/2132644541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111c15dc60 con 0x7f111c070c00 2024-05-27T00:23:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1121600640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070c00 0x7f111c0c71d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f111000c4d0 tx=0x7f111000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1119600640 1 -- 172.21.15.98:0/2132644541 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1110018070 con 0x7f111c070c00 2024-05-27T00:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1119600640 1 -- 172.21.15.98:0/2132644541 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11100074e0 con 0x7f111c070c00 2024-05-27T00:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1119600640 1 -- 172.21.15.98:0/2132644541 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1110004a30 con 0x7f111c070c00 2024-05-27T00:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 -- 172.21.15.98:0/2132644541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f111c0c54a0 con 0x7f111c070c00 2024-05-27T00:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 -- 172.21.15.98:0/2132644541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f111c0c69d0 con 0x7f111c070c00 2024-05-27T00:23:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.801+0000 7f1124000640 1 -- 172.21.15.98:0/2132644541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f111c075cb0 con 0x7f111c070c00 2024-05-27T00:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.805+0000 7f1119600640 1 -- 172.21.15.98:0/2132644541 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1110004bd0 con 0x7f111c070c00 2024-05-27T00:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.805+0000 7f1119600640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10fc0678c0 0x7f10fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.805+0000 7f1119600640 1 -- 172.21.15.98:0/2132644541 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f1110085210 con 0x7f111c070c00 2024-05-27T00:23:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.805+0000 7f1120c00640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10fc0678c0 0x7f10fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.805+0000 7f1120c00640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10fc0678c0 0x7f10fc069d80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f11080b2000 tx=0x7f1108008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:54.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.809+0000 7f1119600640 1 -- 172.21.15.98:0/2132644541 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1110055a40 con 0x7f111c070c00 2024-05-27T00:23:54.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.997+0000 7f1124000640 1 -- 172.21.15.98:0/2132644541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f111c14c6f0 con 0x7f111c070c00 2024-05-27T00:23:55.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:54.997+0000 7f1119600640 1 -- 172.21.15.98:0/2132644541 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6007 (secure 0 0 0) 0x7f1110055340 con 0x7f111c070c00 2024-05-27T00:23:55.000 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:55.000 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:51.374512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:23:55.000 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 37 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 -- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10fc0678c0 msgr2=0x7f10fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10fc0678c0 0x7f10fc069d80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f11080b2000 tx=0x7f1108008040 comp rx=0 tx=0).stop 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 -- 172.21.15.98:0/2132644541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070c00 msgr2=0x7f111c0c71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070c00 0x7f111c0c71d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f111000c4d0 tx=0x7f111000c990 comp rx=0 tx=0).stop 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 -- 172.21.15.98:0/2132644541 shutdown_connections 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f111c0cb570 0x7f111c14be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10fc0678c0 0x7f10fc069d80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f111c070fe0 0x7f111c15bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 --2- 172.21.15.98:0/2132644541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f111c070c00 0x7f111c0c71d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 -- 172.21.15.98:0/2132644541 >> 172.21.15.98:0/2132644541 conn(0x7f111c072f90 msgr2=0x7f111c073370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 -- 172.21.15.98:0/2132644541 shutdown_connections 2024-05-27T00:23:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:55.001+0000 7f1124000640 1 -- 172.21.15.98:0/2132644541 wait complete. 2024-05-27T00:23:55.021 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:23:57.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917e00640 1 Processor -- start 2024-05-27T00:23:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917e00640 1 -- start start 2024-05-27T00:23:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7ff918067390 0x7ff918072ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff91806edb0 0x7ff918152940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9181585c0 con 0x7ff918067770 2024-05-27T00:23:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff918072a40 con 0x7ff91806edb0 2024-05-27T00:23:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7ff9180c78a0 con 0x7ff918067390 2024-05-27T00:23:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff91806edb0 0x7ff918152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff91806edb0 0x7ff918152940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39738/0 (socket says 172.21.15.98:39738) 2024-05-27T00:23:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917400640 1 -- 172.21.15.98:0/22290689 learned_addr learned my addr 172.21.15.98:0/22290689 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff916a00640 1 --2- 172.21.15.98:0/22290689 >> v2:172.21.15.204:3300/0 conn(0x7ff918067390 0x7ff918072ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917400640 1 -- 172.21.15.98:0/22290689 >> v2:172.21.15.204:3300/0 conn(0x7ff918067390 msgr2=0x7ff918072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917400640 1 --2- 172.21.15.98:0/22290689 >> v2:172.21.15.204:3300/0 conn(0x7ff918067390 0x7ff918072ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917400640 1 -- 172.21.15.98:0/22290689 >> v1:172.21.15.204:6789/0 conn(0x7ff918067770 legacy=0x7ff918152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.169+0000 7ff917400640 1 -- 172.21.15.98:0/22290689 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff918158880 con 0x7ff91806edb0 2024-05-27T00:23:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff917400640 1 --2- 172.21.15.98:0/22290689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff91806edb0 0x7ff918152940 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff91806cee0 tx=0x7ff91001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f45284b4da5101 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff916000640 1 -- 172.21.15.98:0/22290689 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9100b5070 con 0x7ff91806edb0 2024-05-27T00:23:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff916000640 1 -- 172.21.15.98:0/22290689 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9100a8a10 con 0x7ff91806edb0 2024-05-27T00:23:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff916000640 1 -- 172.21.15.98:0/22290689 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9100a8de0 con 0x7ff91806edb0 2024-05-27T00:23:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff917e00640 1 -- 172.21.15.98:0/22290689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff91806edb0 msgr2=0x7ff918152940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff917e00640 1 --2- 172.21.15.98:0/22290689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff91806edb0 0x7ff918152940 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff91806cee0 tx=0x7ff91001ed10 comp rx=0 tx=0).stop 2024-05-27T00:23:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff917e00640 1 -- 172.21.15.98:0/22290689 shutdown_connections 2024-05-27T00:23:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff917e00640 1 --2- 172.21.15.98:0/22290689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff91806edb0 0x7ff918152940 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff917e00640 1 --2- 172.21.15.98:0/22290689 >> v2:172.21.15.204:3300/0 conn(0x7ff918067390 0x7ff918072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff917e00640 1 -- 172.21.15.98:0/22290689 >> 172.21.15.98:0/22290689 conn(0x7ff91806f780 msgr2=0x7ff91806fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff917e00640 1 -- 172.21.15.98:0/22290689 shutdown_connections 2024-05-27T00:23:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.173+0000 7ff917e00640 1 -- 172.21.15.98:0/22290689 wait complete. 2024-05-27T00:23:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 Processor -- start 2024-05-27T00:23:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 -- start start 2024-05-27T00:23:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff918067390 0x7ff918144270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918067770 0x7ff9181edf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff91806edb0 0x7ff9181ee4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff918159120 con 0x7ff918067390 2024-05-27T00:23:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff918158e40 con 0x7ff918067770 2024-05-27T00:23:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff918159400 con 0x7ff91806edb0 2024-05-27T00:23:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff916a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff918067390 0x7ff918144270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff916a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff918067390 0x7ff918144270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39744/0 (socket says 172.21.15.98:39744) 2024-05-27T00:23:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff916a00640 1 -- 172.21.15.98:0/3147712116 learned_addr learned my addr 172.21.15.98:0/3147712116 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff90e000640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918067770 0x7ff9181edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917400640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff91806edb0 0x7ff9181ee4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff916a00640 1 -- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff91806edb0 msgr2=0x7ff9181ee4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff916a00640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff91806edb0 0x7ff9181ee4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff916a00640 1 -- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918067770 msgr2=0x7ff9181edf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff916a00640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918067770 0x7ff9181edf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff916a00640 1 -- 172.21.15.98:0/3147712116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff918154d30 con 0x7ff918067390 2024-05-27T00:23:57.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff916a00640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff918067390 0x7ff918144270 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff908007f20 tx=0x7ff90800e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff90fe00640 1 -- 172.21.15.98:0/3147712116 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff908014620 con 0x7ff918067390 2024-05-27T00:23:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff90fe00640 1 -- 172.21.15.98:0/3147712116 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff908019070 con 0x7ff918067390 2024-05-27T00:23:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 -- 172.21.15.98:0/3147712116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff918155df0 con 0x7ff918067390 2024-05-27T00:23:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff90fe00640 1 -- 172.21.15.98:0/3147712116 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff908014a00 con 0x7ff918067390 2024-05-27T00:23:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 -- 172.21.15.98:0/3147712116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff91814c210 con 0x7ff918067390 2024-05-27T00:23:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.177+0000 7ff917e00640 1 -- 172.21.15.98:0/3147712116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff918155fc0 con 0x7ff918067390 2024-05-27T00:23:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.181+0000 7ff90fe00640 1 -- 172.21.15.98:0/3147712116 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff908014ba0 con 0x7ff918067390 2024-05-27T00:23:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.181+0000 7ff90fe00640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff8f40678f0 0x7ff8f4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:57.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.181+0000 7ff90fe00640 1 -- 172.21.15.98:0/3147712116 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7ff908086880 con 0x7ff918067390 2024-05-27T00:23:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.181+0000 7ff90e000640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff8f40678f0 0x7ff8f4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:57.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.181+0000 7ff90e000640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff8f40678f0 0x7ff8f4069db0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff9181ef0a0 tx=0x7ff900009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:57.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.185+0000 7ff90fe00640 1 -- 172.21.15.98:0/3147712116 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7ff908057130 con 0x7ff918067390 2024-05-27T00:23:57.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.373+0000 7ff917e00640 1 -- 172.21.15.98:0/3147712116 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff918155be0 con 0x7ff918067390 2024-05-27T00:23:57.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.373+0000 7ff90fe00640 1 -- 172.21.15.98:0/3147712116 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6004 (secure 0 0 0) 0x7ff908056a30 con 0x7ff918067390 2024-05-27T00:23:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:57.376 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:56.505970+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:23:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 38 2024-05-27T00:23:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 -- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff8f40678f0 msgr2=0x7ff8f4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff8f40678f0 0x7ff8f4069db0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff9181ef0a0 tx=0x7ff900009040 comp rx=0 tx=0).stop 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 -- 172.21.15.98:0/3147712116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff918067390 msgr2=0x7ff918144270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff918067390 0x7ff918144270 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff908007f20 tx=0x7ff90800e570 comp rx=0 tx=0).stop 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 -- 172.21.15.98:0/3147712116 shutdown_connections 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff91806edb0 0x7ff9181ee4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff8f40678f0 0x7ff8f4069db0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918067770 0x7ff9181edf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 --2- 172.21.15.98:0/3147712116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff918067390 0x7ff918144270 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 -- 172.21.15.98:0/3147712116 >> 172.21.15.98:0/3147712116 conn(0x7ff91806f780 msgr2=0x7ff91806bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 -- 172.21.15.98:0/3147712116 shutdown_connections 2024-05-27T00:23:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:57.377+0000 7ff917e00640 1 -- 172.21.15.98:0/3147712116 wait complete. 2024-05-27T00:23:57.397 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:23:59.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:23:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c2e00640 1 Processor -- start 2024-05-27T00:23:59.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c2e00640 1 -- start start 2024-05-27T00:23:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c2e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f60bc0b3260 0x7f60bc0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c2e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 0x7f60bc0c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c2e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f60bc0c36c0 0x7f60bc0c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c2e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f60bc071010 con 0x7f60bc0b3260 2024-05-27T00:23:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c2e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f60bc0c88c0 con 0x7f60bc0c36c0 2024-05-27T00:23:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c2e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f60bc0c8b80 con 0x7f60bc070660 2024-05-27T00:23:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c1a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f60bc0b3260 0x7f60bc0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c1a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f60bc0b3260 0x7f60bc0c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39750/0 (socket says 172.21.15.98:39750) 2024-05-27T00:23:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c1a00640 1 -- 172.21.15.98:0/2709631069 learned_addr learned my addr 172.21.15.98:0/2709631069 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:59.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c1000640 1 --2- 172.21.15.98:0/2709631069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 0x7f60bc0c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c2400640 1 --2- 172.21.15.98:0/2709631069 >> v2:172.21.15.204:3300/0 conn(0x7f60bc0c36c0 0x7f60bc0c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c1a00640 1 -- 172.21.15.98:0/2709631069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 msgr2=0x7f60bc0c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c1a00640 1 --2- 172.21.15.98:0/2709631069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 0x7f60bc0c3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c1a00640 1 -- 172.21.15.98:0/2709631069 >> v2:172.21.15.204:3300/0 conn(0x7f60bc0c36c0 msgr2=0x7f60bc0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:23:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c1a00640 1 --2- 172.21.15.98:0/2709631069 >> v2:172.21.15.204:3300/0 conn(0x7f60bc0c36c0 0x7f60bc0c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.545+0000 7f60c1a00640 1 -- 172.21.15.98:0/2709631069 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60bc0c8e60 con 0x7f60bc0b3260 2024-05-27T00:23:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c1a00640 1 --2- 172.21.15.98:0/2709631069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0b3260 0x7f60bc0c7cc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0cbd60 tx=0x7f60b001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c904201bb7a2b921 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60bbe00640 1 -- 172.21.15.98:0/2709631069 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00b5070 con 0x7f60bc0b3260 2024-05-27T00:23:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60bbe00640 1 -- 172.21.15.98:0/2709631069 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60b00a89d0 con 0x7f60bc0b3260 2024-05-27T00:23:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60bbe00640 1 -- 172.21.15.98:0/2709631069 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00a8da0 con 0x7f60bc0b3260 2024-05-27T00:23:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c2e00640 1 -- 172.21.15.98:0/2709631069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0b3260 msgr2=0x7f60bc0c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c2e00640 1 --2- 172.21.15.98:0/2709631069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0b3260 0x7f60bc0c7cc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0cbd60 tx=0x7f60b001ed10 comp rx=0 tx=0).stop 2024-05-27T00:23:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c2e00640 1 -- 172.21.15.98:0/2709631069 shutdown_connections 2024-05-27T00:23:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c2e00640 1 --2- 172.21.15.98:0/2709631069 >> v2:172.21.15.204:3300/0 conn(0x7f60bc0c36c0 0x7f60bc0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c2e00640 1 --2- 172.21.15.98:0/2709631069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 0x7f60bc0c3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c2e00640 1 --2- 172.21.15.98:0/2709631069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0b3260 0x7f60bc0c7cc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c2e00640 1 -- 172.21.15.98:0/2709631069 >> 172.21.15.98:0/2709631069 conn(0x7f60bc0cdeb0 msgr2=0x7f60bc0ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c2e00640 1 -- 172.21.15.98:0/2709631069 shutdown_connections 2024-05-27T00:23:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c2e00640 1 -- 172.21.15.98:0/2709631069 wait complete. 2024-05-27T00:23:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.549+0000 7f60c2e00640 1 Processor -- start 2024-05-27T00:23:59.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2e00640 1 -- start start 2024-05-27T00:23:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 0x7f60bc1f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0c36c0 0x7f60bc1f0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60bc06c970 0x7f60bc0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc0c8200 con 0x7f60bc0c36c0 2024-05-27T00:23:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc0c9420 con 0x7f60bc06c970 2024-05-27T00:23:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f60bc0c9940 con 0x7f60bc070660 2024-05-27T00:23:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0c36c0 0x7f60bc1f0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0c36c0 0x7f60bc1f0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39754/0 (socket says 172.21.15.98:39754) 2024-05-27T00:23:59.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1000640 1 -- 172.21.15.98:0/1309446987 learned_addr learned my addr 172.21.15.98:0/1309446987 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:23:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1a00640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 0x7f60bc1f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2400640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60bc06c970 0x7f60bc0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1000640 1 -- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 msgr2=0x7f60bc1f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1000640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 0x7f60bc1f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1000640 1 -- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60bc06c970 msgr2=0x7f60bc0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1000640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60bc06c970 0x7f60bc0cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1000640 1 -- 172.21.15.98:0/1309446987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60bc1f1170 con 0x7f60bc0c36c0 2024-05-27T00:23:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1a00640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 0x7f60bc1f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:23:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2400640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60bc06c970 0x7f60bc0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:23:59.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c1000640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0c36c0 0x7f60bc1f0c30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f60a800d800 tx=0x7f60a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60b9600640 1 -- 172.21.15.98:0/1309446987 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a8018030 con 0x7f60bc0c36c0 2024-05-27T00:23:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60b9600640 1 -- 172.21.15.98:0/1309446987 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a80096a0 con 0x7f60bc0c36c0 2024-05-27T00:23:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60b9600640 1 -- 172.21.15.98:0/1309446987 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a8009870 con 0x7f60bc0c36c0 2024-05-27T00:23:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2e00640 1 -- 172.21.15.98:0/1309446987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60bc0cb770 con 0x7f60bc0c36c0 2024-05-27T00:23:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2e00640 1 -- 172.21.15.98:0/1309446987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60bc15cfa0 con 0x7f60bc0c36c0 2024-05-27T00:23:59.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.553+0000 7f60c2e00640 1 -- 172.21.15.98:0/1309446987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60bc075c40 con 0x7f60bc0c36c0 2024-05-27T00:23:59.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.557+0000 7f60b9600640 1 -- 172.21.15.98:0/1309446987 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f60a801b070 con 0x7f60bc0c36c0 2024-05-27T00:23:59.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.557+0000 7f60b9600640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f609c0678c0 0x7f609c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:23:59.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.557+0000 7f60b9600640 1 -- 172.21.15.98:0/1309446987 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f60a8086300 con 0x7f60bc0c36c0 2024-05-27T00:23:59.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.557+0000 7f60c1a00640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f609c0678c0 0x7f609c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:23:59.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.557+0000 7f60c1a00640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f609c0678c0 0x7f609c069d80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f60b00b2000 tx=0x7f60b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:23:59.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.561+0000 7f60b9600640 1 -- 172.21.15.98:0/1309446987 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f60a8056a90 con 0x7f60bc0c36c0 2024-05-27T00:23:59.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.749+0000 7f60c2e00640 1 -- 172.21.15.98:0/1309446987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60bc0c67f0 con 0x7f60bc0c36c0 2024-05-27T00:23:59.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60b9600640 1 -- 172.21.15.98:0/1309446987 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6004 (secure 0 0 0) 0x7f60a80860c0 con 0x7f60bc0c36c0 2024-05-27T00:23:59.754 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:23:59.754 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:56.505970+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:23:59.755 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 38 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 -- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f609c0678c0 msgr2=0x7f609c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f609c0678c0 0x7f609c069d80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f60b00b2000 tx=0x7f60b0008040 comp rx=0 tx=0).stop 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 -- 172.21.15.98:0/1309446987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0c36c0 msgr2=0x7f60bc1f0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0c36c0 0x7f60bc1f0c30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f60a800d800 tx=0x7f60a800dcc0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 -- 172.21.15.98:0/1309446987 shutdown_connections 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f609c0678c0 0x7f609c069d80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60bc06c970 0x7f60bc0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60bc0c36c0 0x7f60bc1f0c30 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 --2- 172.21.15.98:0/1309446987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60bc070660 0x7f60bc1f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 -- 172.21.15.98:0/1309446987 >> 172.21.15.98:0/1309446987 conn(0x7f60bc0cdeb0 msgr2=0x7f60bc0cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:23:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 -- 172.21.15.98:0/1309446987 shutdown_connections 2024-05-27T00:23:59.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:23:59.753+0000 7f60c2e00640 1 -- 172.21.15.98:0/1309446987 wait complete. 2024-05-27T00:23:59.775 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:24:01.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:01.909+0000 7f278e000640 1 Processor -- start 2024-05-27T00:24:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:01.909+0000 7f278e000640 1 -- start start 2024-05-27T00:24:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:01.913+0000 7f278e000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27880b1e20 0x7f27880c2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:01.913+0000 7f278e000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f278816b5b0 con 0x7f27880b1a40 2024-05-27T00:24:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:01.913+0000 7f278e000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f278816b890 con 0x7f27880c0eb0 2024-05-27T00:24:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:01.913+0000 7f278e000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f27880c1900 con 0x7f27880b1e20 2024-05-27T00:24:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:01.913+0000 7f2787400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f27880b1a40 0x7f278816aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47100/0 (socket says 172.21.15.98:47100) 2024-05-27T00:24:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:01.913+0000 7f2787400640 1 -- 172.21.15.98:0/2529742936 learned_addr learned my addr 172.21.15.98:0/2529742936 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:24:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:01.913+0000 7f2786a00640 1 --2- 172.21.15.98:0/2529742936 >> v2:172.21.15.98:3300/0 conn(0x7f27880b1e20 0x7f27880c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:01.913+0000 7f2786000640 1 -- 172.21.15.98:0/2529742936 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286593705 0 0) 0x7f278816b5b0 con 0x7f27880b1a40 2024-05-27T00:24:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.013+0000 7f2786000640 1 -- 172.21.15.98:0/2529742936 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f276c003510 con 0x7f27880b1a40 2024-05-27T00:24:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.013+0000 7f2786000640 1 -- 172.21.15.98:0/2529742936 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 335211076 0 0) 0x7f278816b890 con 0x7f27880c0eb0 2024-05-27T00:24:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.013+0000 7f2786000640 1 -- 172.21.15.98:0/2529742936 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f278816b5b0 con 0x7f27880c0eb0 2024-05-27T00:24:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.017+0000 7f2786000640 1 -- 172.21.15.98:0/2529742936 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1034995573 0 0) 0x7f276c003510 con 0x7f27880b1a40 2024-05-27T00:24:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.017+0000 7f2786a00640 1 -- 172.21.15.98:0/2529742936 >> v1:172.21.15.204:6789/0 conn(0x7f27880c0eb0 legacy=0x7f27880c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.017+0000 7f2786a00640 1 -- 172.21.15.98:0/2529742936 >> v1:172.21.15.98:6789/0 conn(0x7f27880b1a40 legacy=0x7f278816aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.017+0000 7f2786a00640 1 -- 172.21.15.98:0/2529742936 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27880c1b50 con 0x7f27880b1e20 2024-05-27T00:24:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.017+0000 7f2786a00640 1 --2- 172.21.15.98:0/2529742936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880b1e20 0x7f27880c2630 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2774007160 tx=0x7f27740272d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d054e987ac31dc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.017+0000 7f2786000640 1 -- 172.21.15.98:0/2529742936 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27740bc070 con 0x7f27880b1e20 2024-05-27T00:24:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.017+0000 7f2786000640 1 -- 172.21.15.98:0/2529742936 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27740078d0 con 0x7f27880b1e20 2024-05-27T00:24:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.017+0000 7f2786000640 1 -- 172.21.15.98:0/2529742936 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778001680 con 0x7f27880b1e20 2024-05-27T00:24:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 -- 172.21.15.98:0/2529742936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880b1e20 msgr2=0x7f27880c2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 --2- 172.21.15.98:0/2529742936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880b1e20 0x7f27880c2630 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2774007160 tx=0x7f27740272d0 comp rx=0 tx=0).stop 2024-05-27T00:24:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 -- 172.21.15.98:0/2529742936 shutdown_connections 2024-05-27T00:24:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 --2- 172.21.15.98:0/2529742936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880b1e20 0x7f27880c2630 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 -- 172.21.15.98:0/2529742936 >> 172.21.15.98:0/2529742936 conn(0x7f2788067530 msgr2=0x7f278806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 -- 172.21.15.98:0/2529742936 shutdown_connections 2024-05-27T00:24:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 -- 172.21.15.98:0/2529742936 wait complete. 2024-05-27T00:24:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 Processor -- start 2024-05-27T00:24:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 -- start start 2024-05-27T00:24:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27880b1a40 0x7f278816a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27880b1e20 0x7f2788201170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880c0eb0 0x7f2788162860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f278816b1a0 con 0x7f27880c0eb0 2024-05-27T00:24:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f27880c2110 con 0x7f27880b1a40 2024-05-27T00:24:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f278e000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f278816b320 con 0x7f27880b1e20 2024-05-27T00:24:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f2787e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880c0eb0 0x7f2788162860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f2787e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880c0eb0 0x7f2788162860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39762/0 (socket says 172.21.15.98:39762) 2024-05-27T00:24:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f2787e00640 1 -- 172.21.15.98:0/2061741842 learned_addr learned my addr 172.21.15.98:0/2061741842 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f2787400640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27880b1a40 0x7f278816a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f2786a00640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27880b1e20 0x7f2788201170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f2787e00640 1 -- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27880b1e20 msgr2=0x7f2788201170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f2787e00640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27880b1e20 0x7f2788201170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.021+0000 7f2787e00640 1 -- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27880b1a40 msgr2=0x7f278816a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f2787e00640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27880b1a40 0x7f278816a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f2787e00640 1 -- 172.21.15.98:0/2061741842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27880cb0c0 con 0x7f27880c0eb0 2024-05-27T00:24:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f2787e00640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880c0eb0 0x7f2788162860 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f278000c730 tx=0x7f278000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f277f400640 1 -- 172.21.15.98:0/2061741842 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780003b80 con 0x7f27880c0eb0 2024-05-27T00:24:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f277f400640 1 -- 172.21.15.98:0/2061741842 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2780003d20 con 0x7f27880c0eb0 2024-05-27T00:24:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f277f400640 1 -- 172.21.15.98:0/2061741842 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278000aa90 con 0x7f27880c0eb0 2024-05-27T00:24:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f278e000640 1 -- 172.21.15.98:0/2061741842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27882016b0 con 0x7f27880c0eb0 2024-05-27T00:24:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f278e000640 1 -- 172.21.15.98:0/2061741842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2788160f90 con 0x7f27880c0eb0 2024-05-27T00:24:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f278e000640 1 -- 172.21.15.98:0/2061741842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2788075cb0 con 0x7f27880c0eb0 2024-05-27T00:24:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f277f400640 1 -- 172.21.15.98:0/2061741842 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f278000ac30 con 0x7f27880c0eb0 2024-05-27T00:24:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f277f400640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27680678c0 0x7f2768069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.025+0000 7f277f400640 1 -- 172.21.15.98:0/2061741842 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f2780085d70 con 0x7f27880c0eb0 2024-05-27T00:24:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.029+0000 7f2787400640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27680678c0 0x7f2768069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.029+0000 7f2787400640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27680678c0 0x7f2768069d80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f27780011f0 tx=0x7f2778001700 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.033+0000 7f277f400640 1 -- 172.21.15.98:0/2061741842 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f2780055f50 con 0x7f27880c0eb0 2024-05-27T00:24:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 -- 172.21.15.98:0/2061741842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2788164e80 con 0x7f27880c0eb0 2024-05-27T00:24:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f277f400640 1 -- 172.21.15.98:0/2061741842 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6004 (secure 0 0 0) 0x7f278005a250 con 0x7f27880c0eb0 2024-05-27T00:24:02.223 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:02.223 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:23:56.505970+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:24:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 38 2024-05-27T00:24:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 -- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27680678c0 msgr2=0x7f2768069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27680678c0 0x7f2768069d80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f27780011f0 tx=0x7f2778001700 comp rx=0 tx=0).stop 2024-05-27T00:24:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 -- 172.21.15.98:0/2061741842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880c0eb0 msgr2=0x7f2788162860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880c0eb0 0x7f2788162860 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f278000c730 tx=0x7f278000cbf0 comp rx=0 tx=0).stop 2024-05-27T00:24:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 -- 172.21.15.98:0/2061741842 shutdown_connections 2024-05-27T00:24:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27880c0eb0 0x7f2788162860 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:02.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27880b1e20 0x7f2788201170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27680678c0 0x7f2768069d80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 --2- 172.21.15.98:0/2061741842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27880b1a40 0x7f278816a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 -- 172.21.15.98:0/2061741842 >> 172.21.15.98:0/2061741842 conn(0x7f2788067530 msgr2=0x7f27880ca3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 -- 172.21.15.98:0/2061741842 shutdown_connections 2024-05-27T00:24:02.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:02.221+0000 7f278e000640 1 -- 172.21.15.98:0/2061741842 wait complete. 2024-05-27T00:24:02.245 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:24:04.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.389+0000 7f5674a00640 1 Processor -- start 2024-05-27T00:24:04.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.389+0000 7f5674a00640 1 -- start start 2024-05-27T00:24:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f5674a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f56700c0eb0 0x7f56700c26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f5674a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56700c7dc0 0x7f567015ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f5674a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5670159170 con 0x7f5670158d90 2024-05-27T00:24:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f5674a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f56700c8210 con 0x7f56700c0eb0 2024-05-27T00:24:04.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f5674a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f567015f840 con 0x7f56700c7dc0 2024-05-27T00:24:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f566de00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5670158d90 0x7f56700c68c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52080/0 (socket says 172.21.15.98:52080) 2024-05-27T00:24:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f566de00640 1 -- 172.21.15.98:0/3375443964 learned_addr learned my addr 172.21.15.98:0/3375443964 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:24:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f5667e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f56700c0eb0 0x7f56700c26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f566e800640 1 --2- 172.21.15.98:0/3375443964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56700c7dc0 0x7f567015ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f566d400640 1 -- 172.21.15.98:0/3375443964 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1155712032 0 0) 0x7f5670159170 con 0x7f5670158d90 2024-05-27T00:24:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f5667e00640 1 -- 172.21.15.98:0/3375443964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56700c7dc0 msgr2=0x7f567015ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f5667e00640 1 --2- 172.21.15.98:0/3375443964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56700c7dc0 0x7f567015ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f5667e00640 1 -- 172.21.15.98:0/3375443964 >> v1:172.21.15.98:6789/0 conn(0x7f5670158d90 legacy=0x7f56700c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:04.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.393+0000 7f5667e00640 1 -- 172.21.15.98:0/3375443964 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567015fb20 con 0x7f56700c0eb0 2024-05-27T00:24:04.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5667e00640 1 --2- 172.21.15.98:0/3375443964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c0eb0 0x7f56700c26a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f56580097c0 tx=0x7f565801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f999b5d24e7d1883 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f566d400640 1 -- 172.21.15.98:0/3375443964 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56580b5070 con 0x7f56700c0eb0 2024-05-27T00:24:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f566d400640 1 -- 172.21.15.98:0/3375443964 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56580a8990 con 0x7f56700c0eb0 2024-05-27T00:24:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f566d400640 1 -- 172.21.15.98:0/3375443964 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56580a8d60 con 0x7f56700c0eb0 2024-05-27T00:24:04.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 -- 172.21.15.98:0/3375443964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c0eb0 msgr2=0x7f56700c26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 --2- 172.21.15.98:0/3375443964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c0eb0 0x7f56700c26a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f56580097c0 tx=0x7f565801ec90 comp rx=0 tx=0).stop 2024-05-27T00:24:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 -- 172.21.15.98:0/3375443964 shutdown_connections 2024-05-27T00:24:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 --2- 172.21.15.98:0/3375443964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56700c7dc0 0x7f567015ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 --2- 172.21.15.98:0/3375443964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c0eb0 0x7f56700c26a0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 -- 172.21.15.98:0/3375443964 >> 172.21.15.98:0/3375443964 conn(0x7f5670067530 msgr2=0x7f567006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 -- 172.21.15.98:0/3375443964 shutdown_connections 2024-05-27T00:24:04.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 -- 172.21.15.98:0/3375443964 wait complete. 2024-05-27T00:24:04.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 Processor -- start 2024-05-27T00:24:04.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 -- start start 2024-05-27T00:24:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.397+0000 7f5674a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56700c0eb0 0x7f5670162540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5674a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c7dc0 0x7f567015d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5674a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5670158d90 0x7f567015df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5674a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f56700c1ea0 con 0x7f56700c7dc0 2024-05-27T00:24:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5674a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f56700c1bc0 con 0x7f5670158d90 2024-05-27T00:24:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5674a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f56700c2180 con 0x7f56700c0eb0 2024-05-27T00:24:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5667e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c7dc0 0x7f567015d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5667e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c7dc0 0x7f567015d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47560/0 (socket says 172.21.15.98:47560) 2024-05-27T00:24:04.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5667e00640 1 -- 172.21.15.98:0/1364554715 learned_addr learned my addr 172.21.15.98:0/1364554715 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:04.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f566de00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56700c0eb0 0x7f5670162540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:04.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f566e800640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5670158d90 0x7f567015df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:04.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5667e00640 1 -- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56700c0eb0 msgr2=0x7f5670162540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:04.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5667e00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56700c0eb0 0x7f5670162540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:04.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5667e00640 1 -- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5670158d90 msgr2=0x7f567015df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:04.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5667e00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5670158d90 0x7f567015df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:04.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5667e00640 1 -- 172.21.15.98:0/1364554715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56700c1710 con 0x7f56700c7dc0 2024-05-27T00:24:04.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5667e00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c7dc0 0x7f567015d9c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f56580b3040 tx=0x7f565801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:04.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5666000640 1 -- 172.21.15.98:0/1364554715 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56580bf070 con 0x7f56700c7dc0 2024-05-27T00:24:04.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5666000640 1 -- 172.21.15.98:0/1364554715 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56580b0820 con 0x7f56700c7dc0 2024-05-27T00:24:04.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5674a00640 1 -- 172.21.15.98:0/1364554715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5670152c70 con 0x7f56700c7dc0 2024-05-27T00:24:04.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5666000640 1 -- 172.21.15.98:0/1364554715 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56580b5040 con 0x7f56700c7dc0 2024-05-27T00:24:04.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5674a00640 1 -- 172.21.15.98:0/1364554715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56701f60d0 con 0x7f56700c7dc0 2024-05-27T00:24:04.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5674a00640 1 -- 172.21.15.98:0/1364554715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5670075cb0 con 0x7f56700c7dc0 2024-05-27T00:24:04.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.401+0000 7f5666000640 1 -- 172.21.15.98:0/1364554715 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f56580c2070 con 0x7f56700c7dc0 2024-05-27T00:24:04.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.405+0000 7f5666000640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f564c067910 0x7f564c069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:04.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.405+0000 7f5666000640 1 -- 172.21.15.98:0/1364554715 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f5658122890 con 0x7f56700c7dc0 2024-05-27T00:24:04.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.405+0000 7f566de00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f564c067910 0x7f564c069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:04.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.405+0000 7f566de00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f564c067910 0x7f564c069dd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5660000f70 tx=0x7f5660007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:04.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.409+0000 7f5666000640 1 -- 172.21.15.98:0/1364554715 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f56580187f0 con 0x7f56700c7dc0 2024-05-27T00:24:04.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 -- 172.21.15.98:0/1364554715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56700b1830 con 0x7f56700c7dc0 2024-05-27T00:24:04.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5666000640 1 -- 172.21.15.98:0/1364554715 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6010 (secure 0 0 0) 0x7f56580f2af0 con 0x7f56700c7dc0 2024-05-27T00:24:04.602 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:04.602 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:01.646108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:clientreplay","state_seq":20,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:24:04.602 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 39 2024-05-27T00:24:04.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 -- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f564c067910 msgr2=0x7f564c069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:04.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f564c067910 0x7f564c069dd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5660000f70 tx=0x7f5660007df0 comp rx=0 tx=0).stop 2024-05-27T00:24:04.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 -- 172.21.15.98:0/1364554715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c7dc0 msgr2=0x7f567015d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:04.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c7dc0 0x7f567015d9c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f56580b3040 tx=0x7f565801ba30 comp rx=0 tx=0).stop 2024-05-27T00:24:04.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 -- 172.21.15.98:0/1364554715 shutdown_connections 2024-05-27T00:24:04.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f564c067910 0x7f564c069dd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:04.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5670158d90 0x7f567015df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:04.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56700c7dc0 0x7f567015d9c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:04.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 --2- 172.21.15.98:0/1364554715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56700c0eb0 0x7f5670162540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:04.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 -- 172.21.15.98:0/1364554715 >> 172.21.15.98:0/1364554715 conn(0x7f5670067530 msgr2=0x7f5670150b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:04.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 -- 172.21.15.98:0/1364554715 shutdown_connections 2024-05-27T00:24:04.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:04.601+0000 7f5674a00640 1 -- 172.21.15.98:0/1364554715 wait complete. 2024-05-27T00:24:04.624 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 39, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 39, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5502": { "addr": "172.21.15.98:6837/518550803", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 518550803, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 518550803, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5502, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:24:01.646108+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5502 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:24:06.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:06.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.769+0000 7f973cc00640 1 Processor -- start 2024-05-27T00:24:06.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.769+0000 7f973cc00640 1 -- start start 2024-05-27T00:24:06.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f973cc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97380cbe10 0x7f97381592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:06.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f973cc00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9738073570 0x7f97381559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:06.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f973cc00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97380c1f60 con 0x7f97380cc1f0 2024-05-27T00:24:06.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f973cc00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9738159810 con 0x7f9738073570 2024-05-27T00:24:06.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f973cc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f97380c6dd0 con 0x7f97380cbe10 2024-05-27T00:24:06.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f9736800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9738073570 0x7f97381559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:06.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f9735400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f97380cc1f0 0x7f97380c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52086/0 (socket says 172.21.15.98:52086) 2024-05-27T00:24:06.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f9735400640 1 -- 172.21.15.98:0/1210888169 learned_addr learned my addr 172.21.15.98:0/1210888169 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:24:06.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f9735e00640 1 --2- 172.21.15.98:0/1210888169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97380cbe10 0x7f97381592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:06.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f9736800640 1 -- 172.21.15.98:0/1210888169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97380cbe10 msgr2=0x7f97381592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:06.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f9736800640 1 --2- 172.21.15.98:0/1210888169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97380cbe10 0x7f97381592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:06.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f9736800640 1 -- 172.21.15.98:0/1210888169 >> v1:172.21.15.98:6789/0 conn(0x7f97380cc1f0 legacy=0x7f97380c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:06.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f9736800640 1 -- 172.21.15.98:0/1210888169 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97380c70b0 con 0x7f9738073570 2024-05-27T00:24:06.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.773+0000 7f9734a00640 1 -- 172.21.15.98:0/1210888169 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3850859586 0 0) 0x7f97380c1f60 con 0x7f97380cc1f0 2024-05-27T00:24:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f9736800640 1 --2- 172.21.15.98:0/1210888169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9738073570 0x7f97381559e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9730009d90 tx=0x7f973001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2318551798944697 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f9734a00640 1 -- 172.21.15.98:0/1210888169 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97300b4070 con 0x7f9738073570 2024-05-27T00:24:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f9734a00640 1 -- 172.21.15.98:0/1210888169 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97300af600 con 0x7f9738073570 2024-05-27T00:24:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f9734a00640 1 -- 172.21.15.98:0/1210888169 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97300af9d0 con 0x7f9738073570 2024-05-27T00:24:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f973cc00640 1 -- 172.21.15.98:0/1210888169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9738073570 msgr2=0x7f97381559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f973cc00640 1 --2- 172.21.15.98:0/1210888169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9738073570 0x7f97381559e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9730009d90 tx=0x7f973001f150 comp rx=0 tx=0).stop 2024-05-27T00:24:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f973cc00640 1 -- 172.21.15.98:0/1210888169 shutdown_connections 2024-05-27T00:24:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f973cc00640 1 --2- 172.21.15.98:0/1210888169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9738073570 0x7f97381559e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f973cc00640 1 --2- 172.21.15.98:0/1210888169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97380cbe10 0x7f97381592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f973cc00640 1 -- 172.21.15.98:0/1210888169 >> 172.21.15.98:0/1210888169 conn(0x7f9738067530 msgr2=0x7f973806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f973cc00640 1 -- 172.21.15.98:0/1210888169 shutdown_connections 2024-05-27T00:24:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f973cc00640 1 -- 172.21.15.98:0/1210888169 wait complete. 2024-05-27T00:24:06.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f973cc00640 1 Processor -- start 2024-05-27T00:24:06.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.777+0000 7f973cc00640 1 -- start start 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f973cc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9738073570 0x7f9738159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f973cc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97380cbe10 0x7f97380c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f973cc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380cc1f0 0x7f97380720c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f973cc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97380c1330 con 0x7f97380cc1f0 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f973cc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f97380c1050 con 0x7f97380cbe10 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f973cc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f97380c79d0 con 0x7f9738073570 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9736800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380cc1f0 0x7f97380720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9736800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380cc1f0 0x7f97380720c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47568/0 (socket says 172.21.15.98:47568) 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9736800640 1 -- 172.21.15.98:0/1220174872 learned_addr learned my addr 172.21.15.98:0/1220174872 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9735400640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97380cbe10 0x7f97380c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:06.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9735e00640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9738073570 0x7f9738159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9736800640 1 -- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9738073570 msgr2=0x7f9738159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9736800640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9738073570 0x7f9738159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9736800640 1 -- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97380cbe10 msgr2=0x7f97380c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9736800640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97380cbe10 0x7f97380c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9736800640 1 -- 172.21.15.98:0/1220174872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973814f0c0 con 0x7f97380cc1f0 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9735400640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97380cbe10 0x7f97380c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9735e00640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9738073570 0x7f9738159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f9736800640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380cc1f0 0x7f97380720c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f97300b2040 tx=0x7f973001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f972e000640 1 -- 172.21.15.98:0/1220174872 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97300b4040 con 0x7f97380cc1f0 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f972e000640 1 -- 172.21.15.98:0/1220174872 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97300039a0 con 0x7f97380cc1f0 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f973cc00640 1 -- 172.21.15.98:0/1220174872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97381f0230 con 0x7f97380cc1f0 2024-05-27T00:24:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f972e000640 1 -- 172.21.15.98:0/1220174872 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9730009450 con 0x7f97380cc1f0 2024-05-27T00:24:06.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f973cc00640 1 -- 172.21.15.98:0/1220174872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97381f04d0 con 0x7f97380cc1f0 2024-05-27T00:24:06.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f973cc00640 1 -- 172.21.15.98:0/1220174872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9738075cb0 con 0x7f97380cc1f0 2024-05-27T00:24:06.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.781+0000 7f972e000640 1 -- 172.21.15.98:0/1220174872 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f97300c4060 con 0x7f97380cc1f0 2024-05-27T00:24:06.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.785+0000 7f972e000640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97180678c0 0x7f9718069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:06.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.785+0000 7f972e000640 1 -- 172.21.15.98:0/1220174872 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f9730123cb0 con 0x7f97380cc1f0 2024-05-27T00:24:06.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.785+0000 7f9735e00640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97180678c0 0x7f9718069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:06.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.785+0000 7f9735e00640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97180678c0 0x7f9718069d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9728007a70 tx=0x7f9728008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:06.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.785+0000 7f972e000640 1 -- 172.21.15.98:0/1220174872 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9730018cb0 con 0x7f97380cc1f0 2024-05-27T00:24:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.977+0000 7f973cc00640 1 -- 172.21.15.98:0/1220174872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97380c8840 con 0x7f97380cc1f0 2024-05-27T00:24:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.977+0000 7f972e000640 1 -- 172.21.15.98:0/1220174872 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6010 (secure 0 0 0) 0x7f97300b8370 con 0x7f97380cc1f0 2024-05-27T00:24:06.979 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:06.980 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:01.646108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:clientreplay","state_seq":20,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:24:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 39 2024-05-27T00:24:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 -- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97180678c0 msgr2=0x7f9718069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97180678c0 0x7f9718069d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9728007a70 tx=0x7f9728008040 comp rx=0 tx=0).stop 2024-05-27T00:24:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 -- 172.21.15.98:0/1220174872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380cc1f0 msgr2=0x7f97380720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380cc1f0 0x7f97380720c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f97300b2040 tx=0x7f973001bef0 comp rx=0 tx=0).stop 2024-05-27T00:24:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 -- 172.21.15.98:0/1220174872 shutdown_connections 2024-05-27T00:24:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97380cc1f0 0x7f97380720c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f97180678c0 0x7f9718069d80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97380cbe10 0x7f97380c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 --2- 172.21.15.98:0/1220174872 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9738073570 0x7f9738159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 -- 172.21.15.98:0/1220174872 >> 172.21.15.98:0/1220174872 conn(0x7f9738067530 msgr2=0x7f97380b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 -- 172.21.15.98:0/1220174872 shutdown_connections 2024-05-27T00:24:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:06.981+0000 7f973cc00640 1 -- 172.21.15.98:0/1220174872 wait complete. 2024-05-27T00:24:07.001 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:24:09.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e3400640 1 Processor -- start 2024-05-27T00:24:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e3400640 1 -- start start 2024-05-27T00:24:09.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e3400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f31dc0cbe10 0x7f31dc1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:09.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc0cc1f0 0x7f31dc0c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:09.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e3400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31dc073570 0x7f31dc1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:09.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e3400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f31dc159810 con 0x7f31dc073570 2024-05-27T00:24:09.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e3400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f31dc0c6dd0 con 0x7f31dc0cbe10 2024-05-27T00:24:09.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e3400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc0c70b0 con 0x7f31dc0cc1f0 2024-05-27T00:24:09.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e1400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31dc073570 0x7f31dc1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:09.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e1400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31dc073570 0x7f31dc1559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47580/0 (socket says 172.21.15.98:47580) 2024-05-27T00:24:09.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e1400640 1 -- 172.21.15.98:0/1885172928 learned_addr learned my addr 172.21.15.98:0/1885172928 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:09.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e0a00640 1 --2- 172.21.15.98:0/1885172928 >> v2:172.21.15.204:3300/0 conn(0x7f31dc0cbe10 0x7f31dc1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:09.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31d3e00640 1 --2- 172.21.15.98:0/1885172928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc0cc1f0 0x7f31dc0c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:09.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e1400640 1 -- 172.21.15.98:0/1885172928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc0cc1f0 msgr2=0x7f31dc0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:09.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e1400640 1 --2- 172.21.15.98:0/1885172928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc0cc1f0 0x7f31dc0c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e1400640 1 -- 172.21.15.98:0/1885172928 >> v2:172.21.15.204:3300/0 conn(0x7f31dc0cbe10 msgr2=0x7f31dc1592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:09.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e1400640 1 --2- 172.21.15.98:0/1885172928 >> v2:172.21.15.204:3300/0 conn(0x7f31dc0cbe10 0x7f31dc1592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.149+0000 7f31e1400640 1 -- 172.21.15.98:0/1885172928 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc0c7390 con 0x7f31dc073570 2024-05-27T00:24:09.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31e1400640 1 --2- 172.21.15.98:0/1885172928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc073570 0x7f31dc1559e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0cc760 tx=0x7f31d401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0653dd82bed20bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:09.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31dbe00640 1 -- 172.21.15.98:0/1885172928 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d40b4070 con 0x7f31dc073570 2024-05-27T00:24:09.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31dbe00640 1 -- 172.21.15.98:0/1885172928 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d40af400 con 0x7f31dc073570 2024-05-27T00:24:09.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31dbe00640 1 -- 172.21.15.98:0/1885172928 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d40af7d0 con 0x7f31dc073570 2024-05-27T00:24:09.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31e3400640 1 -- 172.21.15.98:0/1885172928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc073570 msgr2=0x7f31dc1559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:09.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31e3400640 1 --2- 172.21.15.98:0/1885172928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc073570 0x7f31dc1559e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0cc760 tx=0x7f31d401f1e0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31e3400640 1 -- 172.21.15.98:0/1885172928 shutdown_connections 2024-05-27T00:24:09.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31e3400640 1 --2- 172.21.15.98:0/1885172928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc073570 0x7f31dc1559e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31e3400640 1 --2- 172.21.15.98:0/1885172928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc0cc1f0 0x7f31dc0c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31e3400640 1 --2- 172.21.15.98:0/1885172928 >> v2:172.21.15.204:3300/0 conn(0x7f31dc0cbe10 0x7f31dc1592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31e3400640 1 -- 172.21.15.98:0/1885172928 >> 172.21.15.98:0/1885172928 conn(0x7f31dc067530 msgr2=0x7f31dc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:09.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31e3400640 1 -- 172.21.15.98:0/1885172928 shutdown_connections 2024-05-27T00:24:09.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.153+0000 7f31e3400640 1 -- 172.21.15.98:0/1885172928 wait complete. 2024-05-27T00:24:09.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 Processor -- start 2024-05-27T00:24:09.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 -- start start 2024-05-27T00:24:09.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc073570 0x7f31dc1569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:09.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc0cbe10 0x7f31dc072ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:09.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0cc1f0 0x7f31dc151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:09.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc0c79d0 con 0x7f31dc0cbe10 2024-05-27T00:24:09.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc0c1330 con 0x7f31dc0cc1f0 2024-05-27T00:24:09.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc0c6990 con 0x7f31dc073570 2024-05-27T00:24:09.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc0cbe10 0x7f31dc072ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:09.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc0cbe10 0x7f31dc072ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47590/0 (socket says 172.21.15.98:47590) 2024-05-27T00:24:09.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d3e00640 1 -- 172.21.15.98:0/2884361819 learned_addr learned my addr 172.21.15.98:0/2884361819 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:09.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e0a00640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc073570 0x7f31dc1569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e1400640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0cc1f0 0x7f31dc151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d3e00640 1 -- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc073570 msgr2=0x7f31dc1569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d3e00640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc073570 0x7f31dc1569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d3e00640 1 -- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0cc1f0 msgr2=0x7f31dc151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d3e00640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0cc1f0 0x7f31dc151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d3e00640 1 -- 172.21.15.98:0/2884361819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc156fd0 con 0x7f31dc0cbe10 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e1400640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0cc1f0 0x7f31dc151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e0a00640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc073570 0x7f31dc1569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d3e00640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc0cbe10 0x7f31dc072ac0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f31c400d620 tx=0x7f31c400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d9600640 1 -- 172.21.15.98:0/2884361819 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c400b620 con 0x7f31dc0cbe10 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d9600640 1 -- 172.21.15.98:0/2884361819 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c4019070 con 0x7f31dc0cbe10 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 -- 172.21.15.98:0/2884361819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31dc1f04a0 con 0x7f31dc0cbe10 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31d9600640 1 -- 172.21.15.98:0/2884361819 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c400b9e0 con 0x7f31dc0cbe10 2024-05-27T00:24:09.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 -- 172.21.15.98:0/2884361819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31dc0b0b20 con 0x7f31dc0cbe10 2024-05-27T00:24:09.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.157+0000 7f31e3400640 1 -- 172.21.15.98:0/2884361819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31dc075cb0 con 0x7f31dc0cbe10 2024-05-27T00:24:09.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.161+0000 7f31d9600640 1 -- 172.21.15.98:0/2884361819 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f31c400bb80 con 0x7f31dc0cbe10 2024-05-27T00:24:09.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.161+0000 7f31d9600640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31b80678c0 0x7f31b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:09.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.161+0000 7f31d9600640 1 -- 172.21.15.98:0/2884361819 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f31c40863d0 con 0x7f31dc0cbe10 2024-05-27T00:24:09.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.161+0000 7f31e0a00640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31b80678c0 0x7f31b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:09.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.161+0000 7f31e0a00640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31b80678c0 0x7f31b8069d80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f31cc004660 tx=0x7f31cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:09.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.165+0000 7f31d9600640 1 -- 172.21.15.98:0/2884361819 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f31c4056bf0 con 0x7f31dc0cbe10 2024-05-27T00:24:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.353+0000 7f31e3400640 1 -- 172.21.15.98:0/2884361819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31dc0c8c80 con 0x7f31dc0cbe10 2024-05-27T00:24:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.353+0000 7f31d9600640 1 -- 172.21.15.98:0/2884361819 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6011 (secure 0 0 0) 0x7f31c40564f0 con 0x7f31dc0cbe10 2024-05-27T00:24:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:09.357 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 40 2024-05-27T00:24:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 -- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31b80678c0 msgr2=0x7f31b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31b80678c0 0x7f31b8069d80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f31cc004660 tx=0x7f31cc008040 comp rx=0 tx=0).stop 2024-05-27T00:24:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 -- 172.21.15.98:0/2884361819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc0cbe10 msgr2=0x7f31dc072ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc0cbe10 0x7f31dc072ac0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f31c400d620 tx=0x7f31c400dae0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 -- 172.21.15.98:0/2884361819 shutdown_connections 2024-05-27T00:24:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31b80678c0 0x7f31b8069d80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31dc0cc1f0 0x7f31dc151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31dc0cbe10 0x7f31dc072ac0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 --2- 172.21.15.98:0/2884361819 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31dc073570 0x7f31dc1569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 -- 172.21.15.98:0/2884361819 >> 172.21.15.98:0/2884361819 conn(0x7f31dc067530 msgr2=0x7f31dc1533c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 -- 172.21.15.98:0/2884361819 shutdown_connections 2024-05-27T00:24:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:09.357+0000 7f31e3400640 1 -- 172.21.15.98:0/2884361819 wait complete. 2024-05-27T00:24:09.378 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=5343 2024-05-27T00:24:09.378 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.a 2024-05-27T00:24:13.958 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 18 -eq 48 ] 2024-05-27T00:24:13.958 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.18 2024-05-27T00:24:13.958 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.18 2024-05-27T00:24:13.958 INFO:tasks.workunit.client.0.smithi098.stdout:v0.18 2024-05-27T00:24:13.959 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.18 2024-05-27T00:24:15.379 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-05-27T00:24:15.379 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-27T00:24:15.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-27T00:24:15.383 INFO:tasks.ceph.mds.a:Started 2024-05-27T00:24:15.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:15.487 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2024-05-27T00:24:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.532+0000 7f7ed8400640 1 Processor -- start 2024-05-27T00:24:15.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.532+0000 7f7ed8400640 1 -- start start 2024-05-27T00:24:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.532+0000 7f7ed8400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ed0070c00 0x7f7ed00c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.532+0000 7f7ed8400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7ed0154e90 0x7f7ed015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed8400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ed015a950 con 0x7f7ed0070fe0 2024-05-27T00:24:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed8400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7ed015e060 con 0x7f7ed0154e90 2024-05-27T00:24:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed8400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed015e340 con 0x7f7ed0070c00 2024-05-27T00:24:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7ed0070fe0 0x7f7ed0154780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35988/0 (socket says 172.21.15.98:35988) 2024-05-27T00:24:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5000640 1 -- 172.21.15.98:0/4036902018 learned_addr learned my addr 172.21.15.98:0/4036902018 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:24:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5a00640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ed0070c00 0x7f7ed00c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed6400640 1 --2- 172.21.15.98:0/4036902018 >> v2:172.21.15.204:3300/0 conn(0x7f7ed0154e90 0x7f7ed015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 233717074 0 0) 0x7f7ed015a950 con 0x7f7ed0070fe0 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb8003530 con 0x7f7ed0070fe0 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2980476430 0 0) 0x7f7eb8003530 con 0x7f7ed0070fe0 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ed0070c00 msgr2=0x7f7ed00c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ed0070c00 0x7f7ed00c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 >> v2:172.21.15.204:3300/0 conn(0x7f7ed0154e90 msgr2=0x7f7ed015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 --2- 172.21.15.98:0/4036902018 >> v2:172.21.15.204:3300/0 conn(0x7f7ed0154e90 0x7f7ed015ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed015e620 con 0x7f7ed0070fe0 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f7ebc001680 con 0x7f7ed0070fe0 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 >> v1:172.21.15.98:6789/0 conn(0x7f7ed0070fe0 legacy=0x7f7ed0154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7eb80039c0 0x7f7eb8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 0x7f7eb8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7eb80051d0 0x7f7eb800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed015e060 con 0x7f7eb8004810 2024-05-27T00:24:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed015e340 con 0x7f7eb80051d0 2024-05-27T00:24:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7eb800bf10 con 0x7f7eb80039c0 2024-05-27T00:24:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5a00640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ed0070c00 0x7f7ed00c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:24:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5000640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 0x7f7eb8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed6400640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7eb80051d0 0x7f7eb800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5a00640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7eb80039c0 0x7f7eb8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5000640 1 -- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7eb80039c0 msgr2=0x7f7eb8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5000640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7eb80039c0 0x7f7eb8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5000640 1 -- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7eb80051d0 msgr2=0x7f7eb800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5000640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7eb80051d0 0x7f7eb800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.536+0000 7f7ed5000640 1 -- 172.21.15.98:0/4036902018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb8003530 con 0x7f7eb8004810 2024-05-27T00:24:15.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ed5000640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 0x7f7eb8004c90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0071400 tx=0x7f7ebc01cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab66d510aac3c546 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:15.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc007c90 con 0x7f7eb8004810 2024-05-27T00:24:15.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ebc0a9020 con 0x7f7eb8004810 2024-05-27T00:24:15.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ecfe00640 1 -- 172.21.15.98:0/4036902018 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc005ae0 con 0x7f7eb8004810 2024-05-27T00:24:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ed8400640 1 -- 172.21.15.98:0/4036902018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 msgr2=0x7f7eb8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ed8400640 1 --2- 172.21.15.98:0/4036902018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 0x7f7eb8004c90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0071400 tx=0x7f7ebc01cbc0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ed6400640 1 -- 172.21.15.98:0/4036902018 reap_dead start 2024-05-27T00:24:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ed8400640 1 -- 172.21.15.98:0/4036902018 shutdown_connections 2024-05-27T00:24:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ed8400640 1 -- 172.21.15.98:0/4036902018 >> 172.21.15.98:0/4036902018 conn(0x7f7ed0072f90 msgr2=0x7f7ed00733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:15.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ed8400640 1 -- 172.21.15.98:0/4036902018 shutdown_connections 2024-05-27T00:24:15.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ed8400640 1 -- 172.21.15.98:0/4036902018 wait complete. 2024-05-27T00:24:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.540+0000 7f7ed8400640 1 Processor -- start 2024-05-27T00:24:15.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed8400640 1 -- start start 2024-05-27T00:24:15.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed8400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 0x7f7ed01e7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:15.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed8400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ed01563b0 0x7f7ed01f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:15.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed8400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ed01552c0 0x7f7ed01f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:15.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed8400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed00cb570 con 0x7f7eb8004810 2024-05-27T00:24:15.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed8400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed015ebe0 con 0x7f7ed01563b0 2024-05-27T00:24:15.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed8400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed00cb6f0 con 0x7f7ed01552c0 2024-05-27T00:24:15.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 0x7f7ed01e7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:15.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 0x7f7ed01e7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48020/0 (socket says 172.21.15.98:48020) 2024-05-27T00:24:15.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5a00640 1 -- 172.21.15.98:0/4205306715 learned_addr learned my addr 172.21.15.98:0/4205306715 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:15.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed6400640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ed01552c0 0x7f7ed01f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:15.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5000640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ed01563b0 0x7f7ed01f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:15.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5a00640 1 -- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ed01552c0 msgr2=0x7f7ed01f0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:15.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5a00640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ed01552c0 0x7f7ed01f0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5a00640 1 -- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ed01563b0 msgr2=0x7f7ed01f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:15.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5a00640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ed01563b0 0x7f7ed01f06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5a00640 1 -- 172.21.15.98:0/4205306715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed01e7d30 con 0x7f7eb8004810 2024-05-27T00:24:15.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5000640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ed01563b0 0x7f7ed01f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed5a00640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 0x7f7ed01e7090 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4011830 tx=0x7f7ec4011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ecd600640 1 -- 172.21.15.98:0/4205306715 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec400f9c0 con 0x7f7eb8004810 2024-05-27T00:24:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ecd600640 1 -- 172.21.15.98:0/4205306715 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ec400fb60 con 0x7f7eb8004810 2024-05-27T00:24:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ecd600640 1 -- 172.21.15.98:0/4205306715 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4015460 con 0x7f7eb8004810 2024-05-27T00:24:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed8400640 1 -- 172.21.15.98:0/4205306715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed00b3bc0 con 0x7f7eb8004810 2024-05-27T00:24:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed8400640 1 -- 172.21.15.98:0/4205306715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed0150d80 con 0x7f7eb8004810 2024-05-27T00:24:15.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.544+0000 7f7ed8400640 1 -- 172.21.15.98:0/4205306715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed0075cb0 con 0x7f7eb8004810 2024-05-27T00:24:15.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.548+0000 7f7ecd600640 1 -- 172.21.15.98:0/4205306715 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7ec4015810 con 0x7f7eb8004810 2024-05-27T00:24:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.548+0000 7f7ecd600640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7eb00678f0 0x7f7eb0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.548+0000 7f7ecd600640 1 -- 172.21.15.98:0/4205306715 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f7ec4086660 con 0x7f7eb8004810 2024-05-27T00:24:15.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.548+0000 7f7ed5000640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7eb00678f0 0x7f7eb0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:15.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.552+0000 7f7ed5000640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7eb00678f0 0x7f7eb0069db0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0aa000 tx=0x7f7ebc001980 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:15.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.552+0000 7f7ecd600640 1 -- 172.21.15.98:0/4205306715 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7ec4019030 con 0x7f7eb8004810 2024-05-27T00:24:15.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.868+0000 7f7ed8400640 1 -- 172.21.15.98:0/4205306715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ed00c9720 con 0x7f7eb8004810 2024-05-27T00:24:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.868+0000 7f7ecd600640 1 -- 172.21.15.98:0/4205306715 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6011 (secure 0 0 0) 0x7f7ec4056f20 con 0x7f7eb8004810 2024-05-27T00:24:15.873 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:15.873 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 40 2024-05-27T00:24:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 -- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7eb00678f0 msgr2=0x7f7eb0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7eb00678f0 0x7f7eb0069db0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0aa000 tx=0x7f7ebc001980 comp rx=0 tx=0).stop 2024-05-27T00:24:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 -- 172.21.15.98:0/4205306715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 msgr2=0x7f7ed01e7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 0x7f7ed01e7090 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4011830 tx=0x7f7ec4011cf0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 -- 172.21.15.98:0/4205306715 shutdown_connections 2024-05-27T00:24:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7ed01552c0 0x7f7ed01f0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7eb00678f0 0x7f7eb0069db0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7ed01563b0 0x7f7ed01f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 --2- 172.21.15.98:0/4205306715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8004810 0x7f7ed01e7090 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 -- 172.21.15.98:0/4205306715 >> 172.21.15.98:0/4205306715 conn(0x7f7ed0072f90 msgr2=0x7f7ed015d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 -- 172.21.15.98:0/4205306715 shutdown_connections 2024-05-27T00:24:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:15.872+0000 7f7ed8400640 1 -- 172.21.15.98:0/4205306715 wait complete. 2024-05-27T00:24:15.896 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-05-27T00:24:17.459 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 47% (372/779) Updating files: 48% (374/779) Updating files: 49% (382/779) Updating files: 50% (390/779) Updating files: 51% (398/779) Updating files: 52% (406/779) Updating files: 53% (413/779) Updating files: 54% (421/779) Updating files: 55% (429/779) Updating files: 56% (437/779) Updating files: 57% (445/779) Updating files: 58% (452/779) Updating files: 59% (460/779) Updating files: 60% (468/779) Updating files: 61% (476/779) Updating files: 62% (483/779) Updating files: 63% (491/779) Updating files: 64% (499/779) Updating files: 65% (507/779) Updating files: 66% (515/779) Updating files: 67% (522/779) Updating files: 68% (530/779) Updating files: 69% (538/779) Updating files: 70% (546/779) Updating files: 71% (554/779) Updating files: 72% (561/779) Updating files: 73% (569/779) Updating files: 74% (577/779) Updating files: 75% (585/779) Updating files: 75% (592/779) Updating files: 76% (593/779) Updating files: 77% (600/779) Updating files: 78% (608/779) Updating files: 79% (616/779) Updating files: 80% (624/779) Updating files: 81% (631/779) Updating files: 82% (639/779) Updating files: 83% (647/779) Updating files: 84% (655/779) Updating files: 85% (663/779) Updating files: 86% (670/779) Updating files: 87% (678/779) Updating files: 88% (686/779) Updating files: 89% (694/779) Updating files: 90% (702/779) Updating files: 91% (709/779) Updating files: 92% (717/779) Updating files: 93% (725/779) Updating files: 94% (733/779) Updating files: 95% (741/779) Updating files: 96% (748/779) Updating files: 97% (756/779) Updating files: 98% (764/779) Updating files: 98% (765/779) Updating files: 99% (772/779) Updating files: 100% (779/779) Updating files: 100% (779/779), done. 2024-05-27T00:24:17.551 INFO:tasks.ceph.mds.c.smithi098.stderr:2024-05-27T00:24:17.548+0000 7f8a31600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:24:17.569 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 2f10137986a v0.18 2024-05-27T00:24:17.573 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.18 2024-05-27T00:24:17.588 INFO:tasks.workunit.client.0.smithi098.stdout:v0.19 2024-05-27T00:24:17.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 19 -eq 48 ] 2024-05-27T00:24:17.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.19 2024-05-27T00:24:17.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.19 2024-05-27T00:24:17.588 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.19 2024-05-27T00:24:17.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:18.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.036+0000 7fb055800640 1 Processor -- start 2024-05-27T00:24:18.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.036+0000 7fb055800640 1 -- start start 2024-05-27T00:24:18.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb055800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb050152880 0x7fb0500c6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:18.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb055800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152c60 0x7fb0500c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:18.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb055800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb050071fc0 0x7fb0500c8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:18.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb055800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb0500c7af0 con 0x7fb050152880 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb055800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb050158a80 con 0x7fb050071fc0 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb055800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb050158d60 con 0x7fb050152c60 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb04ea00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb050152880 0x7fb0500c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb04ea00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb050152880 0x7fb0500c6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48026/0 (socket says 172.21.15.98:48026) 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb04ea00640 1 -- 172.21.15.98:0/1195240488 learned_addr learned my addr 172.21.15.98:0/1195240488 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb04f400640 1 --2- 172.21.15.98:0/1195240488 >> v2:172.21.15.204:3300/0 conn(0x7fb050071fc0 0x7fb0500c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb04e000640 1 --2- 172.21.15.98:0/1195240488 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152c60 0x7fb0500c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb04ea00640 1 -- 172.21.15.98:0/1195240488 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152c60 msgr2=0x7fb0500c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb04ea00640 1 --2- 172.21.15.98:0/1195240488 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152c60 0x7fb0500c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb04ea00640 1 -- 172.21.15.98:0/1195240488 >> v2:172.21.15.204:3300/0 conn(0x7fb050071fc0 msgr2=0x7fb0500c8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb04ea00640 1 --2- 172.21.15.98:0/1195240488 >> v2:172.21.15.204:3300/0 conn(0x7fb050071fc0 0x7fb0500c8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.040+0000 7fb04ea00640 1 -- 172.21.15.98:0/1195240488 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb050159040 con 0x7fb050152880 2024-05-27T00:24:18.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb04ea00640 1 --2- 172.21.15.98:0/1195240488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050152880 0x7fb0500c6770 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb05006f6e0 tx=0x7fb04001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12f9213ef82e6a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:18.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb04d600640 1 -- 172.21.15.98:0/1195240488 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0400b1070 con 0x7fb050152880 2024-05-27T00:24:18.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb04d600640 1 -- 172.21.15.98:0/1195240488 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0400ac620 con 0x7fb050152880 2024-05-27T00:24:18.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb04d600640 1 -- 172.21.15.98:0/1195240488 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0400ac9f0 con 0x7fb050152880 2024-05-27T00:24:18.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 -- 172.21.15.98:0/1195240488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050152880 msgr2=0x7fb0500c6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:18.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 --2- 172.21.15.98:0/1195240488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050152880 0x7fb0500c6770 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb05006f6e0 tx=0x7fb04001c4f0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 -- 172.21.15.98:0/1195240488 shutdown_connections 2024-05-27T00:24:18.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 --2- 172.21.15.98:0/1195240488 >> v2:172.21.15.204:3300/0 conn(0x7fb050071fc0 0x7fb0500c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 --2- 172.21.15.98:0/1195240488 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152c60 0x7fb0500c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 --2- 172.21.15.98:0/1195240488 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050152880 0x7fb0500c6770 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 -- 172.21.15.98:0/1195240488 >> 172.21.15.98:0/1195240488 conn(0x7fb0500cd060 msgr2=0x7fb0500cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:18.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 -- 172.21.15.98:0/1195240488 shutdown_connections 2024-05-27T00:24:18.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 -- 172.21.15.98:0/1195240488 wait complete. 2024-05-27T00:24:18.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 Processor -- start 2024-05-27T00:24:18.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.044+0000 7fb055800640 1 -- start start 2024-05-27T00:24:18.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb055800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050071fc0 0x7fb0501f7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:18.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb055800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152880 0x7fb0500cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:18.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb055800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb050152c60 0x7fb050162130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:18.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb055800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0500672b0 con 0x7fb050071fc0 2024-05-27T00:24:18.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb055800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb050067130 con 0x7fb050152c60 2024-05-27T00:24:18.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb055800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb050067430 con 0x7fb050152880 2024-05-27T00:24:18.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050071fc0 0x7fb0501f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:18.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050071fc0 0x7fb0501f7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48042/0 (socket says 172.21.15.98:48042) 2024-05-27T00:24:18.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04ea00640 1 -- 172.21.15.98:0/2749172971 learned_addr learned my addr 172.21.15.98:0/2749172971 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:18.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04e000640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152880 0x7fb0500cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:18.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04f400640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb050152c60 0x7fb050162130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:18.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04ea00640 1 -- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152880 msgr2=0x7fb0500cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:18.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04ea00640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152880 0x7fb0500cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04ea00640 1 -- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb050152c60 msgr2=0x7fb050162130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:18.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04ea00640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb050152c60 0x7fb050162130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04ea00640 1 -- 172.21.15.98:0/2749172971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0500cb150 con 0x7fb050071fc0 2024-05-27T00:24:18.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04e000640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152880 0x7fb0500cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:18.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04f400640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb050152c60 0x7fb050162130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:18.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb04ea00640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050071fc0 0x7fb0501f7170 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb0400af040 tx=0x7fb040019290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:18.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb046a00640 1 -- 172.21.15.98:0/2749172971 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0400b1040 con 0x7fb050071fc0 2024-05-27T00:24:18.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb046a00640 1 -- 172.21.15.98:0/2749172971 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0400ba070 con 0x7fb050071fc0 2024-05-27T00:24:18.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb046a00640 1 -- 172.21.15.98:0/2749172971 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0400b5680 con 0x7fb050071fc0 2024-05-27T00:24:18.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb055800640 1 -- 172.21.15.98:0/2749172971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0500cb430 con 0x7fb050071fc0 2024-05-27T00:24:18.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb055800640 1 -- 172.21.15.98:0/2749172971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0500cbd20 con 0x7fb050071fc0 2024-05-27T00:24:18.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.048+0000 7fb055800640 1 -- 172.21.15.98:0/2749172971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb050075cb0 con 0x7fb050071fc0 2024-05-27T00:24:18.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.052+0000 7fb046a00640 1 -- 172.21.15.98:0/2749172971 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb0400b5820 con 0x7fb050071fc0 2024-05-27T00:24:18.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.052+0000 7fb046a00640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0300678c0 0x7fb030069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:18.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.052+0000 7fb046a00640 1 -- 172.21.15.98:0/2749172971 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7fb04011fe70 con 0x7fb050071fc0 2024-05-27T00:24:18.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.052+0000 7fb04e000640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0300678c0 0x7fb030069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:18.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.052+0000 7fb04e000640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0300678c0 0x7fb030069d80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb0501f1810 tx=0x7fb03c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:18.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.056+0000 7fb046a00640 1 -- 172.21.15.98:0/2749172971 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb040016110 con 0x7fb050071fc0 2024-05-27T00:24:18.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 -- 172.21.15.98:0/2749172971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0500c6bd0 con 0x7fb050071fc0 2024-05-27T00:24:18.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb046a00640 1 -- 172.21.15.98:0/2749172971 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6836 (secure 0 0 0) 0x7fb0400f0690 con 0x7fb050071fc0 2024-05-27T00:24:18.258 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:18.258 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:18.259 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2024-05-27T00:24:18.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 -- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0300678c0 msgr2=0x7fb030069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:18.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0300678c0 0x7fb030069d80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb0501f1810 tx=0x7fb03c008040 comp rx=0 tx=0).stop 2024-05-27T00:24:18.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 -- 172.21.15.98:0/2749172971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050071fc0 msgr2=0x7fb0501f7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:18.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050071fc0 0x7fb0501f7170 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb0400af040 tx=0x7fb040019290 comp rx=0 tx=0).stop 2024-05-27T00:24:18.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 -- 172.21.15.98:0/2749172971 shutdown_connections 2024-05-27T00:24:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb0300678c0 0x7fb030069d80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb050152c60 0x7fb050162130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb050152880 0x7fb0500cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 --2- 172.21.15.98:0/2749172971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb050071fc0 0x7fb0501f7170 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 -- 172.21.15.98:0/2749172971 >> 172.21.15.98:0/2749172971 conn(0x7fb0500cd060 msgr2=0x7fb05006c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 -- 172.21.15.98:0/2749172971 shutdown_connections 2024-05-27T00:24:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:18.256+0000 7fb055800640 1 -- 172.21.15.98:0/2749172971 wait complete. 2024-05-27T00:24:18.282 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-05-27T00:24:18.282 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 16 secs before thrashing 2024-05-27T00:24:18.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:24:18.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:24:18.588 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 98f5be53f85 v0.19 2024-05-27T00:24:18.593 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.19 2024-05-27T00:24:18.721 INFO:tasks.workunit.client.0.smithi098.stdout:v0.20 2024-05-27T00:24:18.721 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 20 -eq 48 ] 2024-05-27T00:24:18.721 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.20 2024-05-27T00:24:18.721 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.20 2024-05-27T00:24:18.721 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.20 2024-05-27T00:24:27.309 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 1% (3/245) Updating files: 2% (5/245) Updating files: 3% (8/245) Updating files: 4% (10/245) Updating files: 5% (13/245) Updating files: 6% (15/245) Updating files: 7% (18/245) Updating files: 8% (20/245) Updating files: 9% (23/245) Updating files: 10% (25/245) Updating files: 11% (27/245) Updating files: 12% (30/245) Updating files: 13% (32/245) Updating files: 14% (35/245) Updating files: 15% (37/245) Updating files: 16% (40/245) Updating files: 17% (42/245) Updating files: 18% (45/245) Updating files: 19% (47/245) Updating files: 20% (49/245) Updating files: 21% (52/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2024-05-27T00:24:27.486 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 266d2076b6c debian: fix up debian scripts 2024-05-27T00:24:27.489 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.20 2024-05-27T00:24:27.499 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 21 -eq 48 ] 2024-05-27T00:24:27.499 INFO:tasks.workunit.client.0.smithi098.stdout:v0.21 2024-05-27T00:24:27.499 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.21 2024-05-27T00:24:27.500 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.21 2024-05-27T00:24:27.500 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.21 2024-05-27T00:24:29.591 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 56% (303/537) Updating files: 57% (307/537) Updating files: 58% (312/537) Updating files: 59% (317/537) Updating files: 60% (323/537) Updating files: 61% (328/537) Updating files: 62% (333/537) Updating files: 63% (339/537) Updating files: 64% (344/537) Updating files: 65% (350/537) Updating files: 66% (355/537) Updating files: 67% (360/537) Updating files: 68% (366/537) Updating files: 69% (371/537) Updating files: 70% (376/537) Updating files: 71% (382/537) Updating files: 72% (387/537) Updating files: 73% (393/537) Updating files: 74% (398/537) Updating files: 75% (403/537) Updating files: 76% (409/537) Updating files: 77% (414/537) Updating files: 78% (419/537) Updating files: 79% (425/537) Updating files: 80% (430/537) Updating files: 81% (435/537) Updating files: 82% (441/537) Updating files: 83% (446/537) Updating files: 84% (452/537) Updating files: 85% (457/537) Updating files: 86% (462/537) Updating files: 87% (468/537) Updating files: 88% (473/537) Updating files: 89% (478/537) Updating files: 90% (484/537) Updating files: 91% (489/537) Updating files: 92% (495/537) Updating files: 93% (500/537) Updating files: 94% (505/537) Updating files: 95% (511/537) Updating files: 96% (516/537) Updating files: 97% (521/537) Updating files: 98% (527/537) Updating files: 99% (532/537) Updating files: 100% (537/537) Updating files: 100% (537/537), done. 2024-05-27T00:24:29.625 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 090436f56ac v0.21 2024-05-27T00:24:29.631 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.21 2024-05-27T00:24:29.726 INFO:tasks.workunit.client.0.smithi098.stdout:v0.22 2024-05-27T00:24:29.726 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 22 -eq 48 ] 2024-05-27T00:24:29.727 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.22 2024-05-27T00:24:29.727 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.22 2024-05-27T00:24:29.727 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.22 2024-05-27T00:24:30.504 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 8a7c95f60ad v0.22 2024-05-27T00:24:30.509 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.22 2024-05-27T00:24:30.703 INFO:tasks.workunit.client.0.smithi098.stdout:v0.23 2024-05-27T00:24:30.703 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 23 -eq 48 ] 2024-05-27T00:24:30.704 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.23 2024-05-27T00:24:30.704 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.23 2024-05-27T00:24:30.704 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.23 2024-05-27T00:24:31.572 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 5d1d8d0c460 v0.23 2024-05-27T00:24:31.576 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.23 2024-05-27T00:24:31.796 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 24 -eq 48 ] 2024-05-27T00:24:31.796 INFO:tasks.workunit.client.0.smithi098.stdout:v0.24 2024-05-27T00:24:31.797 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.24 2024-05-27T00:24:31.797 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.24 2024-05-27T00:24:31.797 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.24 2024-05-27T00:24:32.767 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 180a4176035 v0.24 2024-05-27T00:24:32.772 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.24 2024-05-27T00:24:32.950 INFO:tasks.workunit.client.0.smithi098.stdout:v0.25 2024-05-27T00:24:32.950 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 25 -eq 48 ] 2024-05-27T00:24:32.950 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.25 2024-05-27T00:24:32.950 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.25 2024-05-27T00:24:32.951 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.25 2024-05-27T00:24:34.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:34.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff670e00640 1 Processor -- start 2024-05-27T00:24:34.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff670e00640 1 -- start start 2024-05-27T00:24:34.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff670e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c070660 0x7ff66c0c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff670e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff66c071010 con 0x7ff66c0c36c0 2024-05-27T00:24:34.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff670e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff66c0c88c0 con 0x7ff66c0b3260 2024-05-27T00:24:34.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff670e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff66c0c8b80 con 0x7ff66c070660 2024-05-27T00:24:34.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66be00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff66c0c36c0 0x7ff66c0c9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54440/0 (socket says 172.21.15.98:54440) 2024-05-27T00:24:34.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66be00640 1 -- 172.21.15.98:0/3924621923 learned_addr learned my addr 172.21.15.98:0/3924621923 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:24:34.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66aa00640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c070660 0x7ff66c0c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2226153756 0 0) 0x7ff6640014a0 con 0x7ff66c0c36c0 2024-05-27T00:24:34.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff650003530 con 0x7ff66c0c36c0 2024-05-27T00:24:34.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3431996360 0 0) 0x7ff66c0c88c0 con 0x7ff66c0b3260 2024-05-27T00:24:34.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6640014a0 con 0x7ff66c0b3260 2024-05-27T00:24:34.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 117636465 0 0) 0x7ff650003530 con 0x7ff66c0c36c0 2024-05-27T00:24:34.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c070660 msgr2=0x7ff66c0c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c070660 0x7ff66c0c3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 >> v1:172.21.15.204:6789/0 conn(0x7ff66c0b3260 legacy=0x7ff66c0c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66c0c8e60 con 0x7ff66c0c36c0 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7ff664001bd0 con 0x7ff66c0c36c0 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 >> v1:172.21.15.98:6789/0 conn(0x7ff66c0c36c0 legacy=0x7ff66c0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 0x7ff650003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c070660 0x7ff650004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff650004d50 0x7ff6500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c0c8b80 con 0x7ff650003b80 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff65000ba80 con 0x7ff650004d50 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff65000bc80 con 0x7ff66c070660 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66b400640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 0x7ff650003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66be00640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff650004d50 0x7ff6500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.428+0000 7ff66aa00640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c070660 0x7ff650004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff66b400640 1 -- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c070660 msgr2=0x7ff650004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:34.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff66b400640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c070660 0x7ff650004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff66b400640 1 -- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff650004d50 msgr2=0x7ff6500051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:34.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff66b400640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff650004d50 0x7ff6500051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff66b400640 1 -- 172.21.15.98:0/3924621923 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff650003530 con 0x7ff650003b80 2024-05-27T00:24:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff66b400640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 0x7ff650003f80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff65c0018d0 tx=0x7ff65c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205199a3193783d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c001520 con 0x7ff650003b80 2024-05-27T00:24:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff65c0052f0 con 0x7ff650003b80 2024-05-27T00:24:34.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff66a000640 1 -- 172.21.15.98:0/3924621923 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c0a8950 con 0x7ff650003b80 2024-05-27T00:24:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff670e00640 1 -- 172.21.15.98:0/3924621923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 msgr2=0x7ff650003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff670e00640 1 --2- 172.21.15.98:0/3924621923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 0x7ff650003f80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff65c0018d0 tx=0x7ff65c01ecf0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.432+0000 7ff66be00640 1 -- 172.21.15.98:0/3924621923 reap_dead start 2024-05-27T00:24:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 -- 172.21.15.98:0/3924621923 shutdown_connections 2024-05-27T00:24:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 -- 172.21.15.98:0/3924621923 >> 172.21.15.98:0/3924621923 conn(0x7ff66c0cdeb0 msgr2=0x7ff66c0ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 -- 172.21.15.98:0/3924621923 shutdown_connections 2024-05-27T00:24:34.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 -- 172.21.15.98:0/3924621923 wait complete. 2024-05-27T00:24:34.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 Processor -- start 2024-05-27T00:24:34.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 -- start start 2024-05-27T00:24:34.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 0x7ff66c155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c156fa0 0x7ff66c15b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff66c15d1f0 0x7ff66c1f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c0c9940 con 0x7ff650003b80 2024-05-27T00:24:34.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c0c9420 con 0x7ff66c15d1f0 2024-05-27T00:24:34.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff670e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff66c0c9ac0 con 0x7ff66c156fa0 2024-05-27T00:24:34.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff66b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 0x7ff66c155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff66b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 0x7ff66c155370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46508/0 (socket says 172.21.15.98:46508) 2024-05-27T00:24:34.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff66b400640 1 -- 172.21.15.98:0/1143364817 learned_addr learned my addr 172.21.15.98:0/1143364817 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:34.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff66aa00640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c156fa0 0x7ff66c15b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff66be00640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff66c15d1f0 0x7ff66c1f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff66b400640 1 -- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c156fa0 msgr2=0x7ff66c15b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff66b400640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c156fa0 0x7ff66c15b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff66b400640 1 -- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff66c15d1f0 msgr2=0x7ff66c1f0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff66b400640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff66c15d1f0 0x7ff66c1f0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.436+0000 7ff66b400640 1 -- 172.21.15.98:0/1143364817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66c15c180 con 0x7ff650003b80 2024-05-27T00:24:34.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.440+0000 7ff66b400640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 0x7ff66c155370 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff65c01eb20 tx=0x7ff650005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:34.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.440+0000 7ff663400640 1 -- 172.21.15.98:0/1143364817 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c0054a0 con 0x7ff650003b80 2024-05-27T00:24:34.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.440+0000 7ff663400640 1 -- 172.21.15.98:0/1143364817 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff65c001500 con 0x7ff650003b80 2024-05-27T00:24:34.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.440+0000 7ff670e00640 1 -- 172.21.15.98:0/1143364817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff66c1f0420 con 0x7ff650003b80 2024-05-27T00:24:34.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.440+0000 7ff663400640 1 -- 172.21.15.98:0/1143364817 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c0a8e30 con 0x7ff650003b80 2024-05-27T00:24:34.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.440+0000 7ff670e00640 1 -- 172.21.15.98:0/1143364817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff66c15c720 con 0x7ff650003b80 2024-05-27T00:24:34.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.440+0000 7ff670e00640 1 -- 172.21.15.98:0/1143364817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff66c075c40 con 0x7ff650003b80 2024-05-27T00:24:34.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.444+0000 7ff663400640 1 -- 172.21.15.98:0/1143364817 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff65c003600 con 0x7ff650003b80 2024-05-27T00:24:34.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.444+0000 7ff663400640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff6480678c0 0x7ff648069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.444+0000 7ff663400640 1 -- 172.21.15.98:0/1143364817 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7ff65c123480 con 0x7ff650003b80 2024-05-27T00:24:34.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.444+0000 7ff66aa00640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff6480678c0 0x7ff648069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.444+0000 7ff66aa00640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff6480678c0 0x7ff648069d80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff66c158510 tx=0x7ff654009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:34.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.448+0000 7ff663400640 1 -- 172.21.15.98:0/1143364817 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7ff65c018a00 con 0x7ff650003b80 2024-05-27T00:24:34.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 -- 172.21.15.98:0/1143364817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff66c0cd270 con 0x7ff650003b80 2024-05-27T00:24:34.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff663400640 1 -- 172.21.15.98:0/1143364817 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6836 (secure 0 0 0) 0x7ff65c0f3d30 con 0x7ff650003b80 2024-05-27T00:24:34.642 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:34.642 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:34.642 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2024-05-27T00:24:34.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 -- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff6480678c0 msgr2=0x7ff648069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff6480678c0 0x7ff648069d80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff66c158510 tx=0x7ff654009190 comp rx=0 tx=0).stop 2024-05-27T00:24:34.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 -- 172.21.15.98:0/1143364817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 msgr2=0x7ff66c155370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 0x7ff66c155370 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff65c01eb20 tx=0x7ff650005840 comp rx=0 tx=0).stop 2024-05-27T00:24:34.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 -- 172.21.15.98:0/1143364817 shutdown_connections 2024-05-27T00:24:34.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff6480678c0 0x7ff648069d80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff66c15d1f0 0x7ff66c1f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff66c156fa0 0x7ff66c15b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 --2- 172.21.15.98:0/1143364817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff650003b80 0x7ff66c155370 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 -- 172.21.15.98:0/1143364817 >> 172.21.15.98:0/1143364817 conn(0x7ff66c0cdeb0 msgr2=0x7ff66c0c4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:34.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 -- 172.21.15.98:0/1143364817 shutdown_connections 2024-05-27T00:24:34.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.640+0000 7ff670e00640 1 -- 172.21.15.98:0/1143364817 wait complete. 2024-05-27T00:24:34.662 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-05-27T00:24:34.662 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-27T00:24:34.663 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:24:34.728 INFO:tasks.ceph.mds.c:Stopped 2024-05-27T00:24:34.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.904+0000 7f966d000640 1 Processor -- start 2024-05-27T00:24:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.904+0000 7f966d000640 1 -- start start 2024-05-27T00:24:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f966d000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f96680731f0 0x7f96680c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f966d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96680735d0 0x7f9668151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f966d000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96680c17d0 con 0x7f96680c79f0 2024-05-27T00:24:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f966d000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9668071990 con 0x7f96680731f0 2024-05-27T00:24:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f966d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9668070e10 con 0x7f96680735d0 2024-05-27T00:24:34.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9666800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f96680c79f0 0x7f9668157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54442/0 (socket says 172.21.15.98:54442) 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9666800640 1 -- 172.21.15.98:0/1257356966 learned_addr learned my addr 172.21.15.98:0/1257356966 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f965d400640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96680735d0 0x7f9668151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665e00640 1 --2- 172.21.15.98:0/1257356966 >> v2:172.21.15.204:3300/0 conn(0x7f96680731f0 0x7f96680c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876131563 0 0) 0x7f96680c17d0 con 0x7f96680c79f0 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f964c005740 con 0x7f96680c79f0 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 404604535 0 0) 0x7f964c005740 con 0x7f96680c79f0 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96680735d0 msgr2=0x7f9668151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96680735d0 0x7f9668151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 >> v2:172.21.15.204:3300/0 conn(0x7f96680731f0 msgr2=0x7f96680c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 --2- 172.21.15.98:0/1257356966 >> v2:172.21.15.204:3300/0 conn(0x7f96680731f0 0x7f96680c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96680710f0 con 0x7f96680c79f0 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f9660001680 con 0x7f96680c79f0 2024-05-27T00:24:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 >> v1:172.21.15.98:6789/0 conn(0x7f96680c79f0 legacy=0x7f9668157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f964c005bd0 0x7f964c005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96680735d0 0x7f964c006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f964c006f60 0x7f964c0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9668071990 con 0x7f964c005bd0 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9668070e10 con 0x7f964c006f60 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f964c00dbe0 con 0x7f96680735d0 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665e00640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f964c005bd0 0x7f964c005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f965d400640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96680735d0 0x7f964c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9666800640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f964c006f60 0x7f964c0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665e00640 1 -- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96680735d0 msgr2=0x7f964c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665e00640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96680735d0 0x7f964c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665e00640 1 -- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f964c006f60 msgr2=0x7f964c0073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665e00640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f964c006f60 0x7f964c0073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9665e00640 1 -- 172.21.15.98:0/1257356966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964c005740 con 0x7f964c005bd0 2024-05-27T00:24:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.908+0000 7f9666800640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f964c006f60 0x7f964c0073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:24:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f9665e00640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f964c005bd0 0x7f964c005ff0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9658009fc0 tx=0x7f965801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d621b567a595d28d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96580b2070 con 0x7f964c005bd0 2024-05-27T00:24:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96580a5cd0 con 0x7f964c005bd0 2024-05-27T00:24:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f9665400640 1 -- 172.21.15.98:0/1257356966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658009c10 con 0x7f964c005bd0 2024-05-27T00:24:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f966d000640 1 -- 172.21.15.98:0/1257356966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f964c005bd0 msgr2=0x7f964c005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f966d000640 1 --2- 172.21.15.98:0/1257356966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f964c005bd0 0x7f964c005ff0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9658009fc0 tx=0x7f965801ccc0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f9666800640 1 -- 172.21.15.98:0/1257356966 reap_dead start 2024-05-27T00:24:34.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f966d000640 1 -- 172.21.15.98:0/1257356966 shutdown_connections 2024-05-27T00:24:34.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f966d000640 1 -- 172.21.15.98:0/1257356966 >> 172.21.15.98:0/1257356966 conn(0x7f9668067530 msgr2=0x7f966806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:34.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f966d000640 1 -- 172.21.15.98:0/1257356966 shutdown_connections 2024-05-27T00:24:34.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f966d000640 1 -- 172.21.15.98:0/1257356966 wait complete. 2024-05-27T00:24:34.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f966d000640 1 Processor -- start 2024-05-27T00:24:34.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.912+0000 7f966d000640 1 -- start start 2024-05-27T00:24:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f966d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f964c005bd0 0x7f9668155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f966d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966815b590 0x7f9668156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f966d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9668144620 0x7f9668144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f966d000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96680c8400 con 0x7f9668144620 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f966d000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f96680716b0 con 0x7f966815b590 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f966d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f96680c13f0 con 0x7f964c005bd0 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f9666800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9668144620 0x7f9668144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f9666800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9668144620 0x7f9668144aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46528/0 (socket says 172.21.15.98:46528) 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f9666800640 1 -- 172.21.15.98:0/304477693 learned_addr learned my addr 172.21.15.98:0/304477693 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f9665e00640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f964c005bd0 0x7f9668155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f965d400640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966815b590 0x7f9668156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f9666800640 1 -- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f964c005bd0 msgr2=0x7f9668155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f9666800640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f964c005bd0 0x7f9668155ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f9666800640 1 -- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966815b590 msgr2=0x7f9668156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f9666800640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966815b590 0x7f9668156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f9666800640 1 -- 172.21.15.98:0/304477693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9668155470 con 0x7f9668144620 2024-05-27T00:24:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f965d400640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966815b590 0x7f9668156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f9666800640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9668144620 0x7f9668144aa0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f96680c81f0 tx=0x7f9660010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f965ea00640 1 -- 172.21.15.98:0/304477693 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966001a070 con 0x7f9668144620 2024-05-27T00:24:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f965ea00640 1 -- 172.21.15.98:0/304477693 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9660011040 con 0x7f9668144620 2024-05-27T00:24:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f965ea00640 1 -- 172.21.15.98:0/304477693 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966000ec80 con 0x7f9668144620 2024-05-27T00:24:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f966d000640 1 -- 172.21.15.98:0/304477693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9668159940 con 0x7f9668144620 2024-05-27T00:24:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f966d000640 1 -- 172.21.15.98:0/304477693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96681556e0 con 0x7f9668144620 2024-05-27T00:24:34.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f966d000640 1 -- 172.21.15.98:0/304477693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9668075cb0 con 0x7f9668144620 2024-05-27T00:24:34.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.916+0000 7f965ea00640 1 -- 172.21.15.98:0/304477693 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f966001d070 con 0x7f9668144620 2024-05-27T00:24:34.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.920+0000 7f965ea00640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96440678c0 0x7f9644069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:34.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.920+0000 7f965ea00640 1 -- 172.21.15.98:0/304477693 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f9660088630 con 0x7f9668144620 2024-05-27T00:24:34.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.920+0000 7f9665e00640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96440678c0 0x7f9644069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:34.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.920+0000 7f9665e00640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96440678c0 0x7f9644069d80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f96580af000 tx=0x7f9658004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:34.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:34.924+0000 7f965ea00640 1 -- 172.21.15.98:0/304477693 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9660058800 con 0x7f9668144620 2024-05-27T00:24:35.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.112+0000 7f966d000640 1 -- 172.21.15.98:0/304477693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9668158510 con 0x7f9668144620 2024-05-27T00:24:35.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.112+0000 7f965ea00640 1 -- 172.21.15.98:0/304477693 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6836 (secure 0 0 0) 0x7f966005cb00 con 0x7f9668144620 2024-05-27T00:24:35.117 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:35.117 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:35.117 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2024-05-27T00:24:35.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 -- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96440678c0 msgr2=0x7f9644069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96440678c0 0x7f9644069d80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f96580af000 tx=0x7f9658004ac0 comp rx=0 tx=0).stop 2024-05-27T00:24:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 -- 172.21.15.98:0/304477693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9668144620 msgr2=0x7f9668144aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9668144620 0x7f9668144aa0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f96680c81f0 tx=0x7f9660010d70 comp rx=0 tx=0).stop 2024-05-27T00:24:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 -- 172.21.15.98:0/304477693 shutdown_connections 2024-05-27T00:24:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9668144620 0x7f9668144aa0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96440678c0 0x7f9644069d80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f966815b590 0x7f9668156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 --2- 172.21.15.98:0/304477693 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f964c005bd0 0x7f9668155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 -- 172.21.15.98:0/304477693 >> 172.21.15.98:0/304477693 conn(0x7f9668067530 msgr2=0x7f966814f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:35.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 -- 172.21.15.98:0/304477693 shutdown_connections 2024-05-27T00:24:35.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:35.116+0000 7f966d000640 1 -- 172.21.15.98:0/304477693 wait complete. 2024-05-27T00:24:35.138 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:24:37.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:37.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2fe00640 1 Processor -- start 2024-05-27T00:24:37.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2fe00640 1 -- start start 2024-05-27T00:24:37.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2fe00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4b28070c00 0x7f4b280c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:37.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b28070fe0 0x7f4b28154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:37.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2fe00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b2815a950 con 0x7f4b280c8f40 2024-05-27T00:24:37.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2fe00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f4b2815e060 con 0x7f4b28070c00 2024-05-27T00:24:37.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4b2815e340 con 0x7f4b28070fe0 2024-05-27T00:24:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2d400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4b28070c00 0x7f4b280c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2d400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4b28070c00 0x7f4b280c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:43112/0 (socket says 172.21.15.98:43112) 2024-05-27T00:24:37.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2d400640 1 -- 172.21.15.98:0/1810570988 learned_addr learned my addr 172.21.15.98:0/1810570988 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2de00640 1 --1- 172.21.15.98:0/1810570988 >> v1:172.21.15.204:6789/0 conn(0x7f4b280c8f40 0x7f4b2815ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:58152/0 (socket says 172.21.15.98:58152) 2024-05-27T00:24:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2ca00640 1 --2- 172.21.15.98:0/1810570988 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b28070fe0 0x7f4b28154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2d400640 1 -- 172.21.15.98:0/1810570988 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b28070fe0 msgr2=0x7f4b28154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2d400640 1 --2- 172.21.15.98:0/1810570988 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b28070fe0 0x7f4b28154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2d400640 1 -- 172.21.15.98:0/1810570988 >> v1:172.21.15.204:6789/0 conn(0x7f4b280c8f40 legacy=0x7f4b2815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:37.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.284+0000 7f4b2d400640 1 -- 172.21.15.98:0/1810570988 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2815e620 con 0x7f4b28070c00 2024-05-27T00:24:37.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b2d400640 1 --2- 172.21.15.98:0/1810570988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b28070c00 0x7f4b280c8a00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009e50 tx=0x7f4b1c01f210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfad772e6c2703c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:37.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b27e00640 1 -- 172.21.15.98:0/1810570988 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c0b4070 con 0x7f4b28070c00 2024-05-27T00:24:37.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b27e00640 1 -- 172.21.15.98:0/1810570988 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b1c0af400 con 0x7f4b28070c00 2024-05-27T00:24:37.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b27e00640 1 -- 172.21.15.98:0/1810570988 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c0af7d0 con 0x7f4b28070c00 2024-05-27T00:24:37.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b2fe00640 1 -- 172.21.15.98:0/1810570988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b28070c00 msgr2=0x7f4b280c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:37.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b2fe00640 1 --2- 172.21.15.98:0/1810570988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b28070c00 0x7f4b280c8a00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009e50 tx=0x7f4b1c01f210 comp rx=0 tx=0).stop 2024-05-27T00:24:37.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b2fe00640 1 -- 172.21.15.98:0/1810570988 shutdown_connections 2024-05-27T00:24:37.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b2fe00640 1 --2- 172.21.15.98:0/1810570988 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b28070fe0 0x7f4b28154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:37.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b2fe00640 1 --2- 172.21.15.98:0/1810570988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b28070c00 0x7f4b280c8a00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:37.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b2fe00640 1 -- 172.21.15.98:0/1810570988 >> 172.21.15.98:0/1810570988 conn(0x7f4b28072f90 msgr2=0x7f4b280733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:37.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b2fe00640 1 -- 172.21.15.98:0/1810570988 shutdown_connections 2024-05-27T00:24:37.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.288+0000 7f4b2fe00640 1 -- 172.21.15.98:0/1810570988 wait complete. 2024-05-27T00:24:37.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2fe00640 1 Processor -- start 2024-05-27T00:24:37.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2fe00640 1 -- start start 2024-05-27T00:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b28070c00 0x7f4b281f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b28070fe0 0x7f4b280c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b280c8f40 0x7f4b281f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2fe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2815b560 con 0x7f4b28070fe0 2024-05-27T00:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2815ebe0 con 0x7f4b28070c00 2024-05-27T00:24:37.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4b2815b6e0 con 0x7f4b280c8f40 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2ca00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b28070fe0 0x7f4b280c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2ca00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b28070fe0 0x7f4b280c7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46532/0 (socket says 172.21.15.98:46532) 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2ca00640 1 -- 172.21.15.98:0/978140981 learned_addr learned my addr 172.21.15.98:0/978140981 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2d400640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b28070c00 0x7f4b281f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2de00640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b280c8f40 0x7f4b281f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2ca00640 1 -- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b280c8f40 msgr2=0x7f4b281f1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2ca00640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b280c8f40 0x7f4b281f1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2ca00640 1 -- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b28070c00 msgr2=0x7f4b281f3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2ca00640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b28070c00 0x7f4b281f3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2ca00640 1 -- 172.21.15.98:0/978140981 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b281f42c0 con 0x7f4b28070fe0 2024-05-27T00:24:37.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2ca00640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b28070fe0 0x7f4b280c7da0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400e930 tx=0x7f4b1400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b25600640 1 -- 172.21.15.98:0/978140981 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b14017070 con 0x7f4b28070fe0 2024-05-27T00:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b25600640 1 -- 172.21.15.98:0/978140981 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b1400ca00 con 0x7f4b28070fe0 2024-05-27T00:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2fe00640 1 -- 172.21.15.98:0/978140981 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b281f45a0 con 0x7f4b28070fe0 2024-05-27T00:24:37.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b25600640 1 -- 172.21.15.98:0/978140981 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1400cdf0 con 0x7f4b28070fe0 2024-05-27T00:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.292+0000 7f4b2fe00640 1 -- 172.21.15.98:0/978140981 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b281f0860 con 0x7f4b28070fe0 2024-05-27T00:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.296+0000 7f4b2fe00640 1 -- 172.21.15.98:0/978140981 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b28075cb0 con 0x7f4b28070fe0 2024-05-27T00:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.296+0000 7f4b25600640 1 -- 172.21.15.98:0/978140981 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4b1401a070 con 0x7f4b28070fe0 2024-05-27T00:24:37.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.296+0000 7f4b25600640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4b080678c0 0x7f4b08069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.296+0000 7f4b25600640 1 -- 172.21.15.98:0/978140981 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f4b14086300 con 0x7f4b28070fe0 2024-05-27T00:24:37.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.296+0000 7f4b2d400640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4b080678c0 0x7f4b08069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:37.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.300+0000 7f4b2d400640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4b080678c0 0x7f4b08069d80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c0b1000 tx=0x7f4b1c0aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:37.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.300+0000 7f4b25600640 1 -- 172.21.15.98:0/978140981 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4b14056590 con 0x7f4b28070fe0 2024-05-27T00:24:37.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 -- 172.21.15.98:0/978140981 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b280c9720 con 0x7f4b28070fe0 2024-05-27T00:24:37.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b25600640 1 -- 172.21.15.98:0/978140981 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6836 (secure 0 0 0) 0x7f4b1405a890 con 0x7f4b28070fe0 2024-05-27T00:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:37.499 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:37.499 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2024-05-27T00:24:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 -- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4b080678c0 msgr2=0x7f4b08069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4b080678c0 0x7f4b08069d80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c0b1000 tx=0x7f4b1c0aa000 comp rx=0 tx=0).stop 2024-05-27T00:24:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 -- 172.21.15.98:0/978140981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b28070fe0 msgr2=0x7f4b280c7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b28070fe0 0x7f4b280c7da0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400e930 tx=0x7f4b1400edf0 comp rx=0 tx=0).stop 2024-05-27T00:24:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 -- 172.21.15.98:0/978140981 shutdown_connections 2024-05-27T00:24:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b280c8f40 0x7f4b281f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:37.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b28070fe0 0x7f4b280c7da0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4b080678c0 0x7f4b08069d80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 --2- 172.21.15.98:0/978140981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b28070c00 0x7f4b281f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 -- 172.21.15.98:0/978140981 >> 172.21.15.98:0/978140981 conn(0x7f4b28072f90 msgr2=0x7f4b2815ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 -- 172.21.15.98:0/978140981 shutdown_connections 2024-05-27T00:24:37.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:37.496+0000 7f4b2fe00640 1 -- 172.21.15.98:0/978140981 wait complete. 2024-05-27T00:24:37.520 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:24:39.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:39.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.664+0000 7f076e200640 1 Processor -- start 2024-05-27T00:24:39.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.664+0000 7f076e200640 1 -- start start 2024-05-27T00:24:39.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f076e200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f07680735d0 0x7f0768151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:39.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f076e200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07680c17d0 con 0x7f07680c79f0 2024-05-27T00:24:39.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f076e200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0768070e10 con 0x7f07680731f0 2024-05-27T00:24:39.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f076e200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f0768071990 con 0x7f07680735d0 2024-05-27T00:24:39.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f0767e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f07680c79f0 0x7f0768157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54456/0 (socket says 172.21.15.98:54456) 2024-05-27T00:24:39.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f0767e00640 1 -- 172.21.15.98:0/2847117550 learned_addr learned my addr 172.21.15.98:0/2847117550 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:24:39.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f075ea00640 1 --2- 172.21.15.98:0/2847117550 >> v2:172.21.15.204:3300/0 conn(0x7f07680735d0 0x7f0768151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:39.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f0766a00640 1 -- 172.21.15.98:0/2847117550 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3512166122 0 0) 0x7f07680c17d0 con 0x7f07680c79f0 2024-05-27T00:24:39.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f0766a00640 1 -- 172.21.15.98:0/2847117550 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f074c003530 con 0x7f07680c79f0 2024-05-27T00:24:39.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f0766a00640 1 -- 172.21.15.98:0/2847117550 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2842233633 0 0) 0x7f074c003530 con 0x7f07680c79f0 2024-05-27T00:24:39.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f075ea00640 1 -- 172.21.15.98:0/2847117550 >> v1:172.21.15.204:6789/0 conn(0x7f07680731f0 legacy=0x7f07680c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:39.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f075ea00640 1 -- 172.21.15.98:0/2847117550 >> v1:172.21.15.98:6789/0 conn(0x7f07680c79f0 legacy=0x7f0768157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:39.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.668+0000 7f075ea00640 1 -- 172.21.15.98:0/2847117550 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07680710f0 con 0x7f07680735d0 2024-05-27T00:24:39.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f075ea00640 1 --2- 172.21.15.98:0/2847117550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07680735d0 0x7f0768151430 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0768071b10 tx=0x7f075001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d758a0f099fd0466 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f0766a00640 1 -- 172.21.15.98:0/2847117550 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07500b0070 con 0x7f07680735d0 2024-05-27T00:24:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f0766a00640 1 -- 172.21.15.98:0/2847117550 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07500ab400 con 0x7f07680735d0 2024-05-27T00:24:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f0766a00640 1 -- 172.21.15.98:0/2847117550 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0760001a00 con 0x7f07680735d0 2024-05-27T00:24:39.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f076e200640 1 -- 172.21.15.98:0/2847117550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07680735d0 msgr2=0x7f0768151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f076e200640 1 --2- 172.21.15.98:0/2847117550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07680735d0 0x7f0768151430 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0768071b10 tx=0x7f075001c3c0 comp rx=0 tx=0).stop 2024-05-27T00:24:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f076e200640 1 -- 172.21.15.98:0/2847117550 shutdown_connections 2024-05-27T00:24:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f076e200640 1 --2- 172.21.15.98:0/2847117550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07680735d0 0x7f0768151430 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f076e200640 1 -- 172.21.15.98:0/2847117550 >> 172.21.15.98:0/2847117550 conn(0x7f0768067530 msgr2=0x7f076806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f076e200640 1 -- 172.21.15.98:0/2847117550 shutdown_connections 2024-05-27T00:24:39.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f076e200640 1 -- 172.21.15.98:0/2847117550 wait complete. 2024-05-27T00:24:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f076e200640 1 Processor -- start 2024-05-27T00:24:39.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.672+0000 7f076e200640 1 -- start start 2024-05-27T00:24:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f076e200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07680731f0 0x7f07681563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f076e200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07680735d0 0x7f07681418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:39.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f076e200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07680c79f0 0x7f076814dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f076e200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0768165740 con 0x7f07680731f0 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f076e200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0768151d80 con 0x7f07680c79f0 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f076e200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f07681658c0 con 0x7f07680735d0 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07680731f0 0x7f07681563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07680731f0 0x7f07681563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46542/0 (socket says 172.21.15.98:46542) 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767400640 1 -- 172.21.15.98:0/774820247 learned_addr learned my addr 172.21.15.98:0/774820247 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f075ea00640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07680735d0 0x7f07681418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767e00640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07680c79f0 0x7f076814dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767400640 1 -- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07680735d0 msgr2=0x7f07681418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767400640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07680735d0 0x7f07681418e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767400640 1 -- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07680c79f0 msgr2=0x7f076814dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:39.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767400640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07680c79f0 0x7f076814dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767400640 1 -- 172.21.15.98:0/774820247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076815b170 con 0x7f07680731f0 2024-05-27T00:24:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767e00640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07680c79f0 0x7f076814dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f0767400640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07680731f0 0x7f07681563e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f075800c560 tx=0x7f075800ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f075fe00640 1 -- 172.21.15.98:0/774820247 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758016020 con 0x7f07680731f0 2024-05-27T00:24:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f075fe00640 1 -- 172.21.15.98:0/774820247 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0758005220 con 0x7f07680731f0 2024-05-27T00:24:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f075fe00640 1 -- 172.21.15.98:0/774820247 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758007030 con 0x7f07680731f0 2024-05-27T00:24:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f076e200640 1 -- 172.21.15.98:0/774820247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0768158b40 con 0x7f07680731f0 2024-05-27T00:24:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f076e200640 1 -- 172.21.15.98:0/774820247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07681457d0 con 0x7f07680731f0 2024-05-27T00:24:39.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f076e200640 1 -- 172.21.15.98:0/774820247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07681459a0 con 0x7f07680731f0 2024-05-27T00:24:39.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.676+0000 7f075fe00640 1 -- 172.21.15.98:0/774820247 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f07580071d0 con 0x7f07680731f0 2024-05-27T00:24:39.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.680+0000 7f075fe00640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07440678f0 0x7f0744069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:39.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.680+0000 7f075fe00640 1 -- 172.21.15.98:0/774820247 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f07580857b0 con 0x7f07680731f0 2024-05-27T00:24:39.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.680+0000 7f075ea00640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07440678f0 0x7f0744069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:39.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.680+0000 7f075ea00640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07440678f0 0x7f0744069db0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f07500ad000 tx=0x7f07500a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:39.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.684+0000 7f075fe00640 1 -- 172.21.15.98:0/774820247 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f0758020020 con 0x7f07680731f0 2024-05-27T00:24:39.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.872+0000 7f076e200640 1 -- 172.21.15.98:0/774820247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0768075cb0 con 0x7f07680731f0 2024-05-27T00:24:39.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.872+0000 7f075fe00640 1 -- 172.21.15.98:0/774820247 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6836 (secure 0 0 0) 0x7f0768158b40 con 0x7f07680731f0 2024-05-27T00:24:39.877 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:39.877 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:39.878 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2024-05-27T00:24:39.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 -- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07440678f0 msgr2=0x7f0744069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:39.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07440678f0 0x7f0744069db0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f07500ad000 tx=0x7f07500a6000 comp rx=0 tx=0).stop 2024-05-27T00:24:39.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 -- 172.21.15.98:0/774820247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07680731f0 msgr2=0x7f07681563e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:39.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07680731f0 0x7f07681563e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f075800c560 tx=0x7f075800ca20 comp rx=0 tx=0).stop 2024-05-27T00:24:39.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 -- 172.21.15.98:0/774820247 shutdown_connections 2024-05-27T00:24:39.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f07440678f0 0x7f0744069db0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07680c79f0 0x7f076814dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07680735d0 0x7f07681418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 --2- 172.21.15.98:0/774820247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07680731f0 0x7f07681563e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 -- 172.21.15.98:0/774820247 >> 172.21.15.98:0/774820247 conn(0x7f0768067530 msgr2=0x7f07680c6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 -- 172.21.15.98:0/774820247 shutdown_connections 2024-05-27T00:24:39.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:39.876+0000 7f076e200640 1 -- 172.21.15.98:0/774820247 wait complete. 2024-05-27T00:24:39.898 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:24:41.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:42.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb3800640 1 Processor -- start 2024-05-27T00:24:42.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb3800640 1 -- start start 2024-05-27T00:24:42.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb3800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbcac0cbe10 0x7fbcac1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:42.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb3800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fbcac073570 0x7fbcac1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:42.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb3800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcac0c1f60 con 0x7fbcac0cc1f0 2024-05-27T00:24:42.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb3800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbcac159810 con 0x7fbcac0cbe10 2024-05-27T00:24:42.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb3800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fbcac0c6dd0 con 0x7fbcac073570 2024-05-27T00:24:42.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb0e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbcac0cbe10 0x7fbcac1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:42.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb0e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbcac0cbe10 0x7fbcac1592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46556/0 (socket says 172.21.15.98:46556) 2024-05-27T00:24:42.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb0e00640 1 -- 172.21.15.98:0/4153383822 learned_addr learned my addr 172.21.15.98:0/4153383822 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:42.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbca3e00640 1 --1- 172.21.15.98:0/4153383822 >> v1:172.21.15.98:6789/0 conn(0x7fbcac0cc1f0 0x7fbcac0c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54468/0 (socket says 172.21.15.98:54468) 2024-05-27T00:24:42.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb1800640 1 --2- 172.21.15.98:0/4153383822 >> v2:172.21.15.204:3300/0 conn(0x7fbcac073570 0x7fbcac1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:42.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb0e00640 1 -- 172.21.15.98:0/4153383822 >> v2:172.21.15.204:3300/0 conn(0x7fbcac073570 msgr2=0x7fbcac1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:42.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb0e00640 1 --2- 172.21.15.98:0/4153383822 >> v2:172.21.15.204:3300/0 conn(0x7fbcac073570 0x7fbcac1559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:42.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb0e00640 1 -- 172.21.15.98:0/4153383822 >> v1:172.21.15.98:6789/0 conn(0x7fbcac0cc1f0 legacy=0x7fbcac0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:42.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb0e00640 1 -- 172.21.15.98:0/4153383822 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac0c70b0 con 0x7fbcac0cbe10 2024-05-27T00:24:42.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcabe00640 1 -- 172.21.15.98:0/4153383822 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1974082857 0 0) 0x7fbcac0c1f60 con 0x7fbcac0cc1f0 2024-05-27T00:24:42.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.044+0000 7fbcb1800640 1 --2- 172.21.15.98:0/4153383822 >> v2:172.21.15.204:3300/0 conn(0x7fbcac073570 0x7fbcac1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:24:42.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcb0e00640 1 --2- 172.21.15.98:0/4153383822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 0x7fbcac1592d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbcac06e230 tx=0x7fbc9c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74e4a05ec85308a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:42.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcabe00640 1 -- 172.21.15.98:0/4153383822 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c0b5070 con 0x7fbcac0cbe10 2024-05-27T00:24:42.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcabe00640 1 -- 172.21.15.98:0/4153383822 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc9c0a8a10 con 0x7fbcac0cbe10 2024-05-27T00:24:42.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcabe00640 1 -- 172.21.15.98:0/4153383822 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c0a8de0 con 0x7fbcac0cbe10 2024-05-27T00:24:42.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcb3800640 1 -- 172.21.15.98:0/4153383822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 msgr2=0x7fbcac1592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:42.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcb3800640 1 --2- 172.21.15.98:0/4153383822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 0x7fbcac1592d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbcac06e230 tx=0x7fbc9c01ed10 comp rx=0 tx=0).stop 2024-05-27T00:24:42.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcb3800640 1 -- 172.21.15.98:0/4153383822 shutdown_connections 2024-05-27T00:24:42.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcb3800640 1 --2- 172.21.15.98:0/4153383822 >> v2:172.21.15.204:3300/0 conn(0x7fbcac073570 0x7fbcac1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:42.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcb3800640 1 --2- 172.21.15.98:0/4153383822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 0x7fbcac1592d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:42.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcb3800640 1 -- 172.21.15.98:0/4153383822 >> 172.21.15.98:0/4153383822 conn(0x7fbcac067530 msgr2=0x7fbcac06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:42.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcb3800640 1 -- 172.21.15.98:0/4153383822 shutdown_connections 2024-05-27T00:24:42.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.048+0000 7fbcb3800640 1 -- 172.21.15.98:0/4153383822 wait complete. 2024-05-27T00:24:42.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 Processor -- start 2024-05-27T00:24:42.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 -- start start 2024-05-27T00:24:42.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbcac073570 0x7fbcac157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:42.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 0x7fbcac1534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:42.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcac0cc1f0 0x7fbcac1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:42.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac0c1330 con 0x7fbcac0cbe10 2024-05-27T00:24:42.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac0c1050 con 0x7fbcac0cc1f0 2024-05-27T00:24:42.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbcac0c79d0 con 0x7fbcac073570 2024-05-27T00:24:42.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 0x7fbcac1534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:42.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 0x7fbcac1534a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46566/0 (socket says 172.21.15.98:46566) 2024-05-27T00:24:42.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca3e00640 1 -- 172.21.15.98:0/3672819800 learned_addr learned my addr 172.21.15.98:0/3672819800 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:42.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb1800640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcac0cc1f0 0x7fbcac1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:42.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb0e00640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbcac073570 0x7fbcac157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:42.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca3e00640 1 -- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbcac073570 msgr2=0x7fbcac157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:42.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca3e00640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbcac073570 0x7fbcac157a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca3e00640 1 -- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcac0cc1f0 msgr2=0x7fbcac1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca3e00640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcac0cc1f0 0x7fbcac1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca3e00640 1 -- 172.21.15.98:0/3672819800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac1f3f30 con 0x7fbcac0cbe10 2024-05-27T00:24:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca3e00640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 0x7fbcac1534a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400c480 tx=0x7fbc9400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca9600640 1 -- 172.21.15.98:0/3672819800 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94018070 con 0x7fbcac0cbe10 2024-05-27T00:24:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca9600640 1 -- 172.21.15.98:0/3672819800 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc94007500 con 0x7fbcac0cbe10 2024-05-27T00:24:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbca9600640 1 -- 172.21.15.98:0/3672819800 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94004a30 con 0x7fbcac0cbe10 2024-05-27T00:24:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 -- 172.21.15.98:0/3672819800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcac0c38d0 con 0x7fbcac0cbe10 2024-05-27T00:24:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 -- 172.21.15.98:0/3672819800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcac152f80 con 0x7fbcac0cbe10 2024-05-27T00:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.052+0000 7fbcb3800640 1 -- 172.21.15.98:0/3672819800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcac075cb0 con 0x7fbcac0cbe10 2024-05-27T00:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.056+0000 7fbca9600640 1 -- 172.21.15.98:0/3672819800 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbc9401b070 con 0x7fbcac0cbe10 2024-05-27T00:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.056+0000 7fbca9600640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc880678f0 0x7fbc88069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.056+0000 7fbca9600640 1 -- 172.21.15.98:0/3672819800 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7fbc94085200 con 0x7fbcac0cbe10 2024-05-27T00:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.056+0000 7fbcb0e00640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc880678f0 0x7fbc88069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:42.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.056+0000 7fbcb0e00640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc880678f0 0x7fbc88069db0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0b2000 tx=0x7fbc9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:42.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.060+0000 7fbca9600640 1 -- 172.21.15.98:0/3672819800 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fbc94055ab0 con 0x7fbcac0cbe10 2024-05-27T00:24:42.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.428+0000 7fbcb3800640 1 -- 172.21.15.98:0/3672819800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcac0c8920 con 0x7fbcac0cbe10 2024-05-27T00:24:42.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.428+0000 7fbca9600640 1 -- 172.21.15.98:0/3672819800 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6836 (secure 0 0 0) 0x7fbc940553b0 con 0x7fbcac0cbe10 2024-05-27T00:24:42.432 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:42.432 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:42.432 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2024-05-27T00:24:42.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 -- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc880678f0 msgr2=0x7fbc88069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:42.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc880678f0 0x7fbc88069db0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0b2000 tx=0x7fbc9c008040 comp rx=0 tx=0).stop 2024-05-27T00:24:42.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 -- 172.21.15.98:0/3672819800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 msgr2=0x7fbcac1534a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:42.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 0x7fbcac1534a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400c480 tx=0x7fbc9400c940 comp rx=0 tx=0).stop 2024-05-27T00:24:42.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 -- 172.21.15.98:0/3672819800 shutdown_connections 2024-05-27T00:24:42.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc880678f0 0x7fbc88069db0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:42.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbcac0cc1f0 0x7fbcac1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:42.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbcac0cbe10 0x7fbcac1534a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:42.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 --2- 172.21.15.98:0/3672819800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbcac073570 0x7fbcac157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:42.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 -- 172.21.15.98:0/3672819800 >> 172.21.15.98:0/3672819800 conn(0x7fbcac067530 msgr2=0x7fbcac06bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:42.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 -- 172.21.15.98:0/3672819800 shutdown_connections 2024-05-27T00:24:42.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:42.432+0000 7fbcb3800640 1 -- 172.21.15.98:0/3672819800 wait complete. 2024-05-27T00:24:42.455 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:24:44.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.600+0000 7f94b8000640 1 Processor -- start 2024-05-27T00:24:44.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.600+0000 7f94b8000640 1 -- start start 2024-05-27T00:24:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.600+0000 7f94b8000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00731f0 0x7f94b00c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.600+0000 7f94b8000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f94b00735d0 0x7f94b0151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.600+0000 7f94b8000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94b00c17d0 con 0x7f94b00c79f0 2024-05-27T00:24:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.600+0000 7f94b8000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f94b0071990 con 0x7f94b00735d0 2024-05-27T00:24:44.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.600+0000 7f94b8000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f94b0070e10 con 0x7f94b00731f0 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b4c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f94b00735d0 0x7f94b0151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b5600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00731f0 0x7f94b00c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b4c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f94b00735d0 0x7f94b0151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:60800/0 (socket says 172.21.15.98:60800) 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b5600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00731f0 0x7f94b00c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:42614/0 (socket says 172.21.15.98:42614) 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b4c00640 1 -- 172.21.15.98:0/1202706586 learned_addr learned my addr 172.21.15.98:0/1202706586 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b6000640 1 --1- 172.21.15.98:0/1202706586 >> v1:172.21.15.204:6789/0 conn(0x7f94b00c79f0 0x7f94b0157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:59456/0 (socket says 172.21.15.98:59456) 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b5600640 1 -- 172.21.15.98:0/1202706586 >> v2:172.21.15.204:3300/0 conn(0x7f94b00735d0 msgr2=0x7f94b0151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b5600640 1 --2- 172.21.15.98:0/1202706586 >> v2:172.21.15.204:3300/0 conn(0x7f94b00735d0 0x7f94b0151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b5600640 1 -- 172.21.15.98:0/1202706586 >> v1:172.21.15.204:6789/0 conn(0x7f94b00c79f0 legacy=0x7f94b0157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b5600640 1 -- 172.21.15.98:0/1202706586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b00710f0 con 0x7f94b00731f0 2024-05-27T00:24:44.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b4c00640 1 --2- 172.21.15.98:0/1202706586 >> v2:172.21.15.204:3300/0 conn(0x7f94b00735d0 0x7f94b0151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:24:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.604+0000 7f94b5600640 1 --2- 172.21.15.98:0/1202706586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00731f0 0x7f94b00c0eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f94a40097c0 tx=0x7f94a401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87ca71ff58dcc78a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94afe00640 1 -- 172.21.15.98:0/1202706586 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a40b5070 con 0x7f94b00731f0 2024-05-27T00:24:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94afe00640 1 -- 172.21.15.98:0/1202706586 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a40a8990 con 0x7f94b00731f0 2024-05-27T00:24:44.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94afe00640 1 -- 172.21.15.98:0/1202706586 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a40a8d60 con 0x7f94b00731f0 2024-05-27T00:24:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 -- 172.21.15.98:0/1202706586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00731f0 msgr2=0x7f94b00c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 --2- 172.21.15.98:0/1202706586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00731f0 0x7f94b00c0eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f94a40097c0 tx=0x7f94a401ec90 comp rx=0 tx=0).stop 2024-05-27T00:24:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 -- 172.21.15.98:0/1202706586 shutdown_connections 2024-05-27T00:24:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 --2- 172.21.15.98:0/1202706586 >> v2:172.21.15.204:3300/0 conn(0x7f94b00735d0 0x7f94b0151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 --2- 172.21.15.98:0/1202706586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00731f0 0x7f94b00c0eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 -- 172.21.15.98:0/1202706586 >> 172.21.15.98:0/1202706586 conn(0x7f94b0067530 msgr2=0x7f94b006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 -- 172.21.15.98:0/1202706586 shutdown_connections 2024-05-27T00:24:44.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 -- 172.21.15.98:0/1202706586 wait complete. 2024-05-27T00:24:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 Processor -- start 2024-05-27T00:24:44.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 -- start start 2024-05-27T00:24:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b00731f0 0x7f94b00c2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b00735d0 0x7f94b01f0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00c79f0 0x7f94b01f1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94b00c8400 con 0x7f94b00735d0 2024-05-27T00:24:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f94b00716b0 con 0x7f94b00731f0 2024-05-27T00:24:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b8000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f94b00c13f0 con 0x7f94b00c79f0 2024-05-27T00:24:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b4c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b00735d0 0x7f94b01f0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b4c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b00735d0 0x7f94b01f0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56532/0 (socket says 172.21.15.98:56532) 2024-05-27T00:24:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.608+0000 7f94b4c00640 1 -- 172.21.15.98:0/2429622605 learned_addr learned my addr 172.21.15.98:0/2429622605 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:44.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b5600640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b00731f0 0x7f94b00c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:44.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b6000640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00c79f0 0x7f94b01f1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b4c00640 1 -- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00c79f0 msgr2=0x7f94b01f1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b4c00640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00c79f0 0x7f94b01f1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b4c00640 1 -- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b00731f0 msgr2=0x7f94b00c2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b4c00640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b00731f0 0x7f94b00c2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b4c00640 1 -- 172.21.15.98:0/2429622605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b01f1f20 con 0x7f94b00735d0 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b4c00640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b00735d0 0x7f94b01f0950 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f949c00e9a0 tx=0x7f949c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94ad600640 1 -- 172.21.15.98:0/2429622605 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c017070 con 0x7f94b00735d0 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94ad600640 1 -- 172.21.15.98:0/2429622605 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f949c00ca00 con 0x7f94b00735d0 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94ad600640 1 -- 172.21.15.98:0/2429622605 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c00cdb0 con 0x7f94b00735d0 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b8000640 1 -- 172.21.15.98:0/2429622605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b01f21e0 con 0x7f94b00735d0 2024-05-27T00:24:44.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b8000640 1 -- 172.21.15.98:0/2429622605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b0148740 con 0x7f94b00735d0 2024-05-27T00:24:44.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94b8000640 1 -- 172.21.15.98:0/2429622605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b0075cb0 con 0x7f94b00735d0 2024-05-27T00:24:44.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.612+0000 7f94ad600640 1 -- 172.21.15.98:0/2429622605 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f949c01a070 con 0x7f94b00735d0 2024-05-27T00:24:44.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.616+0000 7f94ad600640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f94900678c0 0x7f9490069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:44.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.616+0000 7f94ad600640 1 -- 172.21.15.98:0/2429622605 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f949c085d70 con 0x7f94b00735d0 2024-05-27T00:24:44.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.616+0000 7f94b5600640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f94900678c0 0x7f9490069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:44.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.616+0000 7f94b5600640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f94900678c0 0x7f9490069d80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f94a40b2000 tx=0x7f94a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:44.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.620+0000 7f94ad600640 1 -- 172.21.15.98:0/2429622605 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f949c055fd0 con 0x7f94b00735d0 2024-05-27T00:24:44.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.808+0000 7f94b8000640 1 -- 172.21.15.98:0/2429622605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94b0158510 con 0x7f94b00735d0 2024-05-27T00:24:44.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.808+0000 7f94ad600640 1 -- 172.21.15.98:0/2429622605 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6836 (secure 0 0 0) 0x7f949c05a2d0 con 0x7f94b00735d0 2024-05-27T00:24:44.812 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:44.812 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:44.812 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2024-05-27T00:24:44.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 -- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f94900678c0 msgr2=0x7f9490069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:44.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f94900678c0 0x7f9490069d80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f94a40b2000 tx=0x7f94a4008040 comp rx=0 tx=0).stop 2024-05-27T00:24:44.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 -- 172.21.15.98:0/2429622605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b00735d0 msgr2=0x7f94b01f0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:44.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b00735d0 0x7f94b01f0950 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f949c00e9a0 tx=0x7f949c00ee60 comp rx=0 tx=0).stop 2024-05-27T00:24:44.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 -- 172.21.15.98:0/2429622605 shutdown_connections 2024-05-27T00:24:44.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b00c79f0 0x7f94b01f1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:44.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94b00735d0 0x7f94b01f0950 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:44.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f94900678c0 0x7f9490069d80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:44.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 --2- 172.21.15.98:0/2429622605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b00731f0 0x7f94b00c2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:44.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 -- 172.21.15.98:0/2429622605 >> 172.21.15.98:0/2429622605 conn(0x7f94b0067530 msgr2=0x7f94b0150700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:44.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 -- 172.21.15.98:0/2429622605 shutdown_connections 2024-05-27T00:24:44.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:44.812+0000 7f94b8000640 1 -- 172.21.15.98:0/2429622605 wait complete. 2024-05-27T00:24:44.834 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:24:46.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:46.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.976+0000 7f9968800640 1 Processor -- start 2024-05-27T00:24:46.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.976+0000 7f9968800640 1 -- start start 2024-05-27T00:24:46.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9968800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99600731f0 0x7f99600c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:46.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9968800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f99600c79f0 0x7f9960157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:46.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9968800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99600c17d0 con 0x7f99600735d0 2024-05-27T00:24:46.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9968800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9960071990 con 0x7f99600c79f0 2024-05-27T00:24:46.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9968800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9960070e10 con 0x7f99600731f0 2024-05-27T00:24:46.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9966800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f99600c79f0 0x7f9960157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:46.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9965e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99600731f0 0x7f99600c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:46.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9966800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f99600c79f0 0x7f9960157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:60818/0 (socket says 172.21.15.98:60818) 2024-05-27T00:24:46.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9966800640 1 -- 172.21.15.98:0/4239280065 learned_addr learned my addr 172.21.15.98:0/4239280065 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:46.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9966800640 1 -- 172.21.15.98:0/4239280065 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99600731f0 msgr2=0x7f99600c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:46.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9966800640 1 --2- 172.21.15.98:0/4239280065 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99600731f0 0x7f99600c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:46.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9966800640 1 -- 172.21.15.98:0/4239280065 >> v1:172.21.15.204:6789/0 conn(0x7f99600735d0 legacy=0x7f9960151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:46.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.980+0000 7f9966800640 1 -- 172.21.15.98:0/4239280065 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99600710f0 con 0x7f99600c79f0 2024-05-27T00:24:46.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9966800640 1 --2- 172.21.15.98:0/4239280065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99600c79f0 0x7f9960157970 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f995c009e50 tx=0x7f995c01f210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42f00d08b0106937 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:46.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9964a00640 1 -- 172.21.15.98:0/4239280065 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c0b4070 con 0x7f99600c79f0 2024-05-27T00:24:46.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9964a00640 1 -- 172.21.15.98:0/4239280065 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f995c0af400 con 0x7f99600c79f0 2024-05-27T00:24:46.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9964a00640 1 -- 172.21.15.98:0/4239280065 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c0af7d0 con 0x7f99600c79f0 2024-05-27T00:24:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9968800640 1 -- 172.21.15.98:0/4239280065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99600c79f0 msgr2=0x7f9960157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9968800640 1 --2- 172.21.15.98:0/4239280065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99600c79f0 0x7f9960157970 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f995c009e50 tx=0x7f995c01f210 comp rx=0 tx=0).stop 2024-05-27T00:24:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9968800640 1 -- 172.21.15.98:0/4239280065 shutdown_connections 2024-05-27T00:24:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9968800640 1 --2- 172.21.15.98:0/4239280065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99600c79f0 0x7f9960157970 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9968800640 1 --2- 172.21.15.98:0/4239280065 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99600731f0 0x7f99600c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9968800640 1 -- 172.21.15.98:0/4239280065 >> 172.21.15.98:0/4239280065 conn(0x7f9960067530 msgr2=0x7f996006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9968800640 1 -- 172.21.15.98:0/4239280065 shutdown_connections 2024-05-27T00:24:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9968800640 1 -- 172.21.15.98:0/4239280065 wait complete. 2024-05-27T00:24:46.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9968800640 1 Processor -- start 2024-05-27T00:24:46.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.984+0000 7f9968800640 1 -- start start 2024-05-27T00:24:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9968800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99600731f0 0x7f996015a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9968800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99600735d0 0x7f99601f0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9968800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99600c79f0 0x7f99601f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9968800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f99600c8400 con 0x7f99600731f0 2024-05-27T00:24:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9968800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f99600716b0 con 0x7f99600c79f0 2024-05-27T00:24:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9968800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f99600c13f0 con 0x7f99600735d0 2024-05-27T00:24:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9965e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99600731f0 0x7f996015a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9965e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99600731f0 0x7f996015a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56536/0 (socket says 172.21.15.98:56536) 2024-05-27T00:24:46.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9965e00640 1 -- 172.21.15.98:0/2839064737 learned_addr learned my addr 172.21.15.98:0/2839064737 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:46.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9965400640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99600735d0 0x7f99601f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:46.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9966800640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99600c79f0 0x7f99601f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:46.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9965e00640 1 -- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99600735d0 msgr2=0x7f99601f0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:46.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9965e00640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99600735d0 0x7f99601f0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:46.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9965e00640 1 -- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99600c79f0 msgr2=0x7f99601f0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:46.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9965e00640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99600c79f0 0x7f99601f0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:46.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9965e00640 1 -- 172.21.15.98:0/2839064737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9960158bb0 con 0x7f99600731f0 2024-05-27T00:24:46.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9965e00640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99600731f0 0x7f996015a270 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f995400e960 tx=0x7f995400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:46.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f995a000640 1 -- 172.21.15.98:0/2839064737 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954017070 con 0x7f99600731f0 2024-05-27T00:24:46.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f995a000640 1 -- 172.21.15.98:0/2839064737 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f995400ca00 con 0x7f99600731f0 2024-05-27T00:24:46.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f995a000640 1 -- 172.21.15.98:0/2839064737 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995400cdb0 con 0x7f99600731f0 2024-05-27T00:24:46.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9968800640 1 -- 172.21.15.98:0/2839064737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9960158e90 con 0x7f99600731f0 2024-05-27T00:24:46.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9968800640 1 -- 172.21.15.98:0/2839064737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9960159450 con 0x7f99600731f0 2024-05-27T00:24:46.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f9968800640 1 -- 172.21.15.98:0/2839064737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9960075cb0 con 0x7f99600731f0 2024-05-27T00:24:46.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.988+0000 7f995a000640 1 -- 172.21.15.98:0/2839064737 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f995401a070 con 0x7f99600731f0 2024-05-27T00:24:46.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.992+0000 7f995a000640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99440678f0 0x7f9944069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:46.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.992+0000 7f995a000640 1 -- 172.21.15.98:0/2839064737 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7f99540862c0 con 0x7f99600731f0 2024-05-27T00:24:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.992+0000 7f9965400640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99440678f0 0x7f9944069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:46.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.992+0000 7f9965400640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99440678f0 0x7f9944069db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f996014cc70 tx=0x7f9950009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:46.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:46.996+0000 7f995a000640 1 -- 172.21.15.98:0/2839064737 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9954056490 con 0x7f99600731f0 2024-05-27T00:24:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.184+0000 7f9968800640 1 -- 172.21.15.98:0/2839064737 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99601f0330 con 0x7f99600731f0 2024-05-27T00:24:47.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.184+0000 7f995a000640 1 -- 172.21.15.98:0/2839064737 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6836 (secure 0 0 0) 0x7f995405a790 con 0x7f99600731f0 2024-05-27T00:24:47.188 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:47.189 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:47.189 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2024-05-27T00:24:47.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 -- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99440678f0 msgr2=0x7f9944069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99440678f0 0x7f9944069db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f996014cc70 tx=0x7f9950009040 comp rx=0 tx=0).stop 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 -- 172.21.15.98:0/2839064737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99600731f0 msgr2=0x7f996015a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99600731f0 0x7f996015a270 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f995400e960 tx=0x7f995400ee20 comp rx=0 tx=0).stop 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 -- 172.21.15.98:0/2839064737 shutdown_connections 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f99440678f0 0x7f9944069db0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99600c79f0 0x7f99601f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99600735d0 0x7f99601f0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 --2- 172.21.15.98:0/2839064737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99600731f0 0x7f996015a270 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 -- 172.21.15.98:0/2839064737 >> 172.21.15.98:0/2839064737 conn(0x7f9960067530 msgr2=0x7f996014e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 -- 172.21.15.98:0/2839064737 shutdown_connections 2024-05-27T00:24:47.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:47.188+0000 7f9968800640 1 -- 172.21.15.98:0/2839064737 wait complete. 2024-05-27T00:24:47.210 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:24:48.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:24:48.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:24:49.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:49.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b3e00640 1 Processor -- start 2024-05-27T00:24:49.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b3e00640 1 -- start start 2024-05-27T00:24:49.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b3e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd0b4070110 0x7fd0b40c3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:49.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0b40c5490 0x7fd0b40c3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:49.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b3e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0b406e7e0 con 0x7fd0b4157d80 2024-05-27T00:24:49.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b3e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd0b415b3e0 con 0x7fd0b4070110 2024-05-27T00:24:49.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b406ea80 con 0x7fd0b40c5490 2024-05-27T00:24:49.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b2000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd0b4070110 0x7fd0b40c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:49.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0b40c5490 0x7fd0b40c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:49.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b2000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd0b4070110 0x7fd0b40c3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:60832/0 (socket says 172.21.15.98:60832) 2024-05-27T00:24:49.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b2000640 1 -- 172.21.15.98:0/3619326598 learned_addr learned my addr 172.21.15.98:0/3619326598 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:49.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b2a00640 1 --1- 172.21.15.98:0/3619326598 >> v1:172.21.15.204:6789/0 conn(0x7fd0b4157d80 0x7fd0b40c4850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:59478/0 (socket says 172.21.15.98:59478) 2024-05-27T00:24:49.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b2000640 1 -- 172.21.15.98:0/3619326598 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0b40c5490 msgr2=0x7fd0b40c3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:49.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b2000640 1 --2- 172.21.15.98:0/3619326598 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0b40c5490 0x7fd0b40c3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:49.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b2000640 1 -- 172.21.15.98:0/3619326598 >> v1:172.21.15.204:6789/0 conn(0x7fd0b4157d80 legacy=0x7fd0b40c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:49.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.356+0000 7fd0b2000640 1 -- 172.21.15.98:0/3619326598 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b40c67f0 con 0x7fd0b4070110 2024-05-27T00:24:49.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b2000640 1 --2- 172.21.15.98:0/3619326598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0b4070110 0x7fd0b40c3350 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0009870 tx=0x7fd0a001c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e677e6a1f9b1911 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:49.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b1600640 1 -- 172.21.15.98:0/3619326598 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a00aa650 con 0x7fd0b4070110 2024-05-27T00:24:49.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b1600640 1 -- 172.21.15.98:0/3619326598 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a00af070 con 0x7fd0b4070110 2024-05-27T00:24:49.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b1600640 1 -- 172.21.15.98:0/3619326598 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a00aaa10 con 0x7fd0b4070110 2024-05-27T00:24:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b3e00640 1 -- 172.21.15.98:0/3619326598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0b4070110 msgr2=0x7fd0b40c3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b3e00640 1 --2- 172.21.15.98:0/3619326598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0b4070110 0x7fd0b40c3350 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0009870 tx=0x7fd0a001c4d0 comp rx=0 tx=0).stop 2024-05-27T00:24:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b3e00640 1 -- 172.21.15.98:0/3619326598 shutdown_connections 2024-05-27T00:24:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b3e00640 1 --2- 172.21.15.98:0/3619326598 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0b40c5490 0x7fd0b40c3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:49.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b3e00640 1 --2- 172.21.15.98:0/3619326598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0b4070110 0x7fd0b40c3350 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:49.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b3e00640 1 -- 172.21.15.98:0/3619326598 >> 172.21.15.98:0/3619326598 conn(0x7fd0b4070a10 msgr2=0x7fd0b4070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:49.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b3e00640 1 -- 172.21.15.98:0/3619326598 shutdown_connections 2024-05-27T00:24:49.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.360+0000 7fd0b3e00640 1 -- 172.21.15.98:0/3619326598 wait complete. 2024-05-27T00:24:49.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3e00640 1 Processor -- start 2024-05-27T00:24:49.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3e00640 1 -- start start 2024-05-27T00:24:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0b4070110 0x7fd0b414d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b40c5490 0x7fd0b4151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:49.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0b4157d80 0x7fd0b41f6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b4167a00 con 0x7fd0b40c5490 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b4167880 con 0x7fd0b4070110 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b4167b80 con 0x7fd0b4157d80 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b2000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b40c5490 0x7fd0b4151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b2000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b40c5490 0x7fd0b4151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56542/0 (socket says 172.21.15.98:56542) 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b2000640 1 -- 172.21.15.98:0/2455818574 learned_addr learned my addr 172.21.15.98:0/2455818574 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b2a00640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0b4070110 0x7fd0b414d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3400640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0b4157d80 0x7fd0b41f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b2000640 1 -- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0b4157d80 msgr2=0x7fd0b41f6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b2000640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0b4157d80 0x7fd0b41f6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b2000640 1 -- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0b4070110 msgr2=0x7fd0b414d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b2000640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0b4070110 0x7fd0b414d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:49.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b2000640 1 -- 172.21.15.98:0/2455818574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b4150a00 con 0x7fd0b40c5490 2024-05-27T00:24:49.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b2000640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b40c5490 0x7fd0b4151860 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd0a001cd70 tx=0x7fd0a001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:49.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0ab400640 1 -- 172.21.15.98:0/2455818574 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a00bd5f0 con 0x7fd0b40c5490 2024-05-27T00:24:49.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0ab400640 1 -- 172.21.15.98:0/2455818574 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a00a6040 con 0x7fd0b40c5490 2024-05-27T00:24:49.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0ab400640 1 -- 172.21.15.98:0/2455818574 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a00bd9f0 con 0x7fd0b40c5490 2024-05-27T00:24:49.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3e00640 1 -- 172.21.15.98:0/2455818574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b41586f0 con 0x7fd0b40c5490 2024-05-27T00:24:49.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.364+0000 7fd0b3e00640 1 -- 172.21.15.98:0/2455818574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b4150ca0 con 0x7fd0b40c5490 2024-05-27T00:24:49.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.368+0000 7fd0b3e00640 1 -- 172.21.15.98:0/2455818574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b4075c40 con 0x7fd0b40c5490 2024-05-27T00:24:49.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.368+0000 7fd0ab400640 1 -- 172.21.15.98:0/2455818574 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd0a00b33f0 con 0x7fd0b40c5490 2024-05-27T00:24:49.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.368+0000 7fd0ab400640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0940678c0 0x7fd094069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:49.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.368+0000 7fd0ab400640 1 -- 172.21.15.98:0/2455818574 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6341+0+0 (secure 0 0 0) 0x7fd0a011fc10 con 0x7fd0b40c5490 2024-05-27T00:24:49.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.368+0000 7fd0b2a00640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0940678c0 0x7fd094069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:49.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.368+0000 7fd0b2a00640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0940678c0 0x7fd094069d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4000f70 tx=0x7fd0a4007e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:49.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.372+0000 7fd0ab400640 1 -- 172.21.15.98:0/2455818574 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd0a0016150 con 0x7fd0b40c5490 2024-05-27T00:24:49.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 -- 172.21.15.98:0/2455818574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0b40c4630 con 0x7fd0b40c5490 2024-05-27T00:24:49.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0ab400640 1 -- 172.21.15.98:0/2455818574 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6836 (secure 0 0 0) 0x7fd0a00effa0 con 0x7fd0b40c5490 2024-05-27T00:24:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:49.567 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:08.589586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":22,"addr":"172.21.15.98:6837/518550803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":518550803},{"type":"v1","addr":"172.21.15.98:6837","nonce":518550803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":1}]} 2024-05-27T00:24:49.567 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 41 2024-05-27T00:24:49.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 -- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0940678c0 msgr2=0x7fd094069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:49.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0940678c0 0x7fd094069d80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4000f70 tx=0x7fd0a4007e50 comp rx=0 tx=0).stop 2024-05-27T00:24:49.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 -- 172.21.15.98:0/2455818574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b40c5490 msgr2=0x7fd0b4151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:49.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b40c5490 0x7fd0b4151860 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd0a001cd70 tx=0x7fd0a001ad10 comp rx=0 tx=0).stop 2024-05-27T00:24:49.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 -- 172.21.15.98:0/2455818574 shutdown_connections 2024-05-27T00:24:49.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0b4157d80 0x7fd0b41f6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:49.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0b40c5490 0x7fd0b4151860 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:49.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd0940678c0 0x7fd094069d80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:49.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 --2- 172.21.15.98:0/2455818574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0b4070110 0x7fd0b414d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:49.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 -- 172.21.15.98:0/2455818574 >> 172.21.15.98:0/2455818574 conn(0x7fd0b4070a10 msgr2=0x7fd0b4155f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:49.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 -- 172.21.15.98:0/2455818574 shutdown_connections 2024-05-27T00:24:49.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:49.564+0000 7fd0b3e00640 1 -- 172.21.15.98:0/2455818574 wait complete. 2024-05-27T00:24:49.588 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:24:51.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.732+0000 7f0080c00640 1 Processor -- start 2024-05-27T00:24:51.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.732+0000 7f0080c00640 1 -- start start 2024-05-27T00:24:51.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f0080c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f007c0731f0 0x7f007c0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:51.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f0080c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0735d0 0x7f007c151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:51.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f0080c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f007c0c79f0 0x7f007c151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:51.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f0080c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f007c071990 con 0x7f007c0c79f0 2024-05-27T00:24:51.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f0080c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f007c0c17d0 con 0x7f007c0731f0 2024-05-27T00:24:51.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f0080c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f007c070e10 con 0x7f007c0735d0 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f007a800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f007c0c79f0 0x7f007c151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f007a800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f007c0c79f0 0x7f007c151970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56548/0 (socket says 172.21.15.98:56548) 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f007a800640 1 -- 172.21.15.98:0/3988244144 learned_addr learned my addr 172.21.15.98:0/3988244144 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f0079e00640 1 --2- 172.21.15.98:0/3988244144 >> v2:172.21.15.204:3300/0 conn(0x7f007c0731f0 0x7f007c0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f0071400640 1 --2- 172.21.15.98:0/3988244144 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0735d0 0x7f007c151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f007a800640 1 -- 172.21.15.98:0/3988244144 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0735d0 msgr2=0x7f007c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f007a800640 1 --2- 172.21.15.98:0/3988244144 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0735d0 0x7f007c151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f007a800640 1 -- 172.21.15.98:0/3988244144 >> v2:172.21.15.204:3300/0 conn(0x7f007c0731f0 msgr2=0x7f007c0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f007a800640 1 --2- 172.21.15.98:0/3988244144 >> v2:172.21.15.204:3300/0 conn(0x7f007c0731f0 0x7f007c0c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f007a800640 1 -- 172.21.15.98:0/3988244144 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007c0710f0 con 0x7f007c0c79f0 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f0071400640 1 --2- 172.21.15.98:0/3988244144 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0735d0 0x7f007c151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:51.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f0079e00640 1 --2- 172.21.15.98:0/3988244144 >> v2:172.21.15.204:3300/0 conn(0x7f007c0731f0 0x7f007c0c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:51.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.736+0000 7f007a800640 1 --2- 172.21.15.98:0/3988244144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 0x7f007c151970 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f007c0cca90 tx=0x7f007401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a21944c3c9966aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:51.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0079400640 1 -- 172.21.15.98:0/3988244144 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00740b4070 con 0x7f007c0c79f0 2024-05-27T00:24:51.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0079400640 1 -- 172.21.15.98:0/3988244144 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00740af400 con 0x7f007c0c79f0 2024-05-27T00:24:51.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0079400640 1 -- 172.21.15.98:0/3988244144 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00740af7d0 con 0x7f007c0c79f0 2024-05-27T00:24:51.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 -- 172.21.15.98:0/3988244144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 msgr2=0x7f007c151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:51.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 --2- 172.21.15.98:0/3988244144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 0x7f007c151970 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f007c0cca90 tx=0x7f007401f1e0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 -- 172.21.15.98:0/3988244144 shutdown_connections 2024-05-27T00:24:51.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 --2- 172.21.15.98:0/3988244144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 0x7f007c151970 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 --2- 172.21.15.98:0/3988244144 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0735d0 0x7f007c151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 --2- 172.21.15.98:0/3988244144 >> v2:172.21.15.204:3300/0 conn(0x7f007c0731f0 0x7f007c0c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 -- 172.21.15.98:0/3988244144 >> 172.21.15.98:0/3988244144 conn(0x7f007c067530 msgr2=0x7f007c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:51.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 -- 172.21.15.98:0/3988244144 shutdown_connections 2024-05-27T00:24:51.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 -- 172.21.15.98:0/3988244144 wait complete. 2024-05-27T00:24:51.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 Processor -- start 2024-05-27T00:24:51.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 -- start start 2024-05-27T00:24:51.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0731f0 0x7f007c159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:51.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f007c0735d0 0x7f007c15b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:51.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 0x7f007c1f0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:51.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f007c0c8400 con 0x7f007c0c79f0 2024-05-27T00:24:51.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f007c0716b0 con 0x7f007c0735d0 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f0080c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f007c0c13f0 con 0x7f007c0731f0 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f007a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 0x7f007c1f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f007a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 0x7f007c1f0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56556/0 (socket says 172.21.15.98:56556) 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.740+0000 7f007a800640 1 -- 172.21.15.98:0/2427380746 learned_addr learned my addr 172.21.15.98:0/2427380746 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0079e00640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0731f0 0x7f007c159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0071400640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f007c0735d0 0x7f007c15b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f007a800640 1 -- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0731f0 msgr2=0x7f007c159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f007a800640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0731f0 0x7f007c159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f007a800640 1 -- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f007c0735d0 msgr2=0x7f007c15b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f007a800640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f007c0735d0 0x7f007c15b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f007a800640 1 -- 172.21.15.98:0/2427380746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007c148720 con 0x7f007c0c79f0 2024-05-27T00:24:51.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f007a800640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 0x7f007c1f0890 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f00740b2040 tx=0x7f007401bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0072a00640 1 -- 172.21.15.98:0/2427380746 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00740b4040 con 0x7f007c0c79f0 2024-05-27T00:24:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0072a00640 1 -- 172.21.15.98:0/2427380746 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00740afdb0 con 0x7f007c0c79f0 2024-05-27T00:24:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0080c00640 1 -- 172.21.15.98:0/2427380746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f007c15a3d0 con 0x7f007c0c79f0 2024-05-27T00:24:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0072a00640 1 -- 172.21.15.98:0/2427380746 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0074009510 con 0x7f007c0c79f0 2024-05-27T00:24:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0080c00640 1 -- 172.21.15.98:0/2427380746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f007c14d750 con 0x7f007c0c79f0 2024-05-27T00:24:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0080c00640 1 -- 172.21.15.98:0/2427380746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f007c075cb0 con 0x7f007c0c79f0 2024-05-27T00:24:51.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0072a00640 1 -- 172.21.15.98:0/2427380746 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f00740c3020 con 0x7f007c0c79f0 2024-05-27T00:24:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0072a00640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00580678c0 0x7f0058069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.744+0000 7f0072a00640 1 -- 172.21.15.98:0/2427380746 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7f0074123a60 con 0x7f007c0c79f0 2024-05-27T00:24:51.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.748+0000 7f0079e00640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00580678c0 0x7f0058069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:51.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.748+0000 7f0079e00640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00580678c0 0x7f0058069d80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f006c00c880 tx=0x7f006c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:51.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.748+0000 7f0072a00640 1 -- 172.21.15.98:0/2427380746 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f00740ca020 con 0x7f007c0c79f0 2024-05-27T00:24:51.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.936+0000 7f0080c00640 1 -- 172.21.15.98:0/2427380746 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f007c06f610 con 0x7f007c0c79f0 2024-05-27T00:24:51.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.936+0000 7f0072a00640 1 -- 172.21.15.98:0/2427380746 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6006 (secure 0 0 0) 0x7f0074018d40 con 0x7f007c0c79f0 2024-05-27T00:24:51.941 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:51.942 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:50.240256+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:24:51.942 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 42 2024-05-27T00:24:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 -- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00580678c0 msgr2=0x7f0058069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00580678c0 0x7f0058069d80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f006c00c880 tx=0x7f006c008040 comp rx=0 tx=0).stop 2024-05-27T00:24:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 -- 172.21.15.98:0/2427380746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 msgr2=0x7f007c1f0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 0x7f007c1f0890 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f00740b2040 tx=0x7f007401bf80 comp rx=0 tx=0).stop 2024-05-27T00:24:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 -- 172.21.15.98:0/2427380746 shutdown_connections 2024-05-27T00:24:51.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f007c0c79f0 0x7f007c1f0890 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f00580678c0 0x7f0058069d80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f007c0735d0 0x7f007c15b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 --2- 172.21.15.98:0/2427380746 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f007c0731f0 0x7f007c159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 -- 172.21.15.98:0/2427380746 >> 172.21.15.98:0/2427380746 conn(0x7f007c067530 msgr2=0x7f007c0c6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 -- 172.21.15.98:0/2427380746 shutdown_connections 2024-05-27T00:24:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:51.940+0000 7f0080c00640 1 -- 172.21.15.98:0/2427380746 wait complete. 2024-05-27T00:24:51.963 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-05-27T00:24:51.963 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:24:51.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f8085400640 1 Processor -- start 2024-05-27T00:24:52.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f8085400640 1 -- start start 2024-05-27T00:24:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f8085400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80800c79f0 0x7f808015bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f8085400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f808015c430 0x7f8080152380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f8085400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f808015c880 con 0x7f8080072f30 2024-05-27T00:24:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f8085400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f808006e4e0 con 0x7f808015c430 2024-05-27T00:24:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f8085400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8080073a20 con 0x7f80800c79f0 2024-05-27T00:24:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f807f200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f808015c430 0x7f8080152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f807f200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f808015c430 0x7f8080152380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56558/0 (socket says 172.21.15.98:56558) 2024-05-27T00:24:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f807f200640 1 -- 172.21.15.98:0/1338634386 learned_addr learned my addr 172.21.15.98:0/1338634386 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:52.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f807de00640 1 --2- 172.21.15.98:0/1338634386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80800c79f0 0x7f808015bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f807f200640 1 -- 172.21.15.98:0/1338634386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80800c79f0 msgr2=0x7f808015bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f807f200640 1 --2- 172.21.15.98:0/1338634386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80800c79f0 0x7f808015bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f807f200640 1 -- 172.21.15.98:0/1338634386 >> v1:172.21.15.204:6789/0 conn(0x7f8080072f30 legacy=0x7f8080073310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:52.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.096+0000 7f807f200640 1 -- 172.21.15.98:0/1338634386 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8080159270 con 0x7f808015c430 2024-05-27T00:24:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f807f200640 1 --2- 172.21.15.98:0/1338634386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f808015c430 0x7f8080152380 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8078009e50 tx=0x7f807801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e16538bb734dd063 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f807d400640 1 -- 172.21.15.98:0/1338634386 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80780b4070 con 0x7f808015c430 2024-05-27T00:24:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f807d400640 1 -- 172.21.15.98:0/1338634386 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80780af400 con 0x7f808015c430 2024-05-27T00:24:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f807d400640 1 -- 172.21.15.98:0/1338634386 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80780af7d0 con 0x7f808015c430 2024-05-27T00:24:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f8085400640 1 -- 172.21.15.98:0/1338634386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f808015c430 msgr2=0x7f8080152380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f8085400640 1 --2- 172.21.15.98:0/1338634386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f808015c430 0x7f8080152380 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8078009e50 tx=0x7f807801f210 comp rx=0 tx=0).stop 2024-05-27T00:24:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f8085400640 1 -- 172.21.15.98:0/1338634386 shutdown_connections 2024-05-27T00:24:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f8085400640 1 --2- 172.21.15.98:0/1338634386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f808015c430 0x7f8080152380 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f8085400640 1 --2- 172.21.15.98:0/1338634386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80800c79f0 0x7f808015bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f8085400640 1 -- 172.21.15.98:0/1338634386 >> 172.21.15.98:0/1338634386 conn(0x7f80800c9af0 msgr2=0x7f80800c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:52.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f8085400640 1 -- 172.21.15.98:0/1338634386 shutdown_connections 2024-05-27T00:24:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.100+0000 7f8085400640 1 -- 172.21.15.98:0/1338634386 wait complete. 2024-05-27T00:24:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 Processor -- start 2024-05-27T00:24:52.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 -- start start 2024-05-27T00:24:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8080072f30 0x7f808006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80800c79f0 0x7f80801f0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f808015c430 0x7f80801e4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80800c9250 con 0x7f8080072f30 2024-05-27T00:24:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f808006e690 con 0x7f80800c79f0 2024-05-27T00:24:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f80800c94a0 con 0x7f808015c430 2024-05-27T00:24:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807e800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8080072f30 0x7f808006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807e800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8080072f30 0x7f808006d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56560/0 (socket says 172.21.15.98:56560) 2024-05-27T00:24:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807e800640 1 -- 172.21.15.98:0/431216112 learned_addr learned my addr 172.21.15.98:0/431216112 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807de00640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80800c79f0 0x7f80801f0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807f200640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f808015c430 0x7f80801e4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807e800640 1 -- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f808015c430 msgr2=0x7f80801e4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807e800640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f808015c430 0x7f80801e4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807e800640 1 -- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80800c79f0 msgr2=0x7f80801f0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807e800640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80800c79f0 0x7f80801f0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807e800640 1 -- 172.21.15.98:0/431216112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80800cb0c0 con 0x7f8080072f30 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807f200640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f808015c430 0x7f80801e4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807de00640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80800c79f0 0x7f80801f0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f807e800640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8080072f30 0x7f808006d570 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f807000c450 tx=0x7f807000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8076a00640 1 -- 172.21.15.98:0/431216112 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80700052d0 con 0x7f8080072f30 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8076a00640 1 -- 172.21.15.98:0/431216112 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8070005470 con 0x7f8080072f30 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 -- 172.21.15.98:0/431216112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80800cb3a0 con 0x7f8080072f30 2024-05-27T00:24:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8076a00640 1 -- 172.21.15.98:0/431216112 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070007260 con 0x7f8080072f30 2024-05-27T00:24:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 -- 172.21.15.98:0/431216112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f808006dab0 con 0x7f8080072f30 2024-05-27T00:24:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.104+0000 7f8085400640 1 -- 172.21.15.98:0/431216112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8080075cb0 con 0x7f8080072f30 2024-05-27T00:24:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.108+0000 7f8076a00640 1 -- 172.21.15.98:0/431216112 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8070007400 con 0x7f8080072f30 2024-05-27T00:24:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.108+0000 7f8076a00640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f805c0678c0 0x7f805c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.108+0000 7f8076a00640 1 -- 172.21.15.98:0/431216112 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7f8070085520 con 0x7f8080072f30 2024-05-27T00:24:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.108+0000 7f807de00640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f805c0678c0 0x7f805c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:52.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.108+0000 7f807de00640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f805c0678c0 0x7f805c069d80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f808013fc50 tx=0x7f8068008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:52.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.112+0000 7f8076a00640 1 -- 172.21.15.98:0/431216112 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8070055cf0 con 0x7f8080072f30 2024-05-27T00:24:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 -- 172.21.15.98:0/431216112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8080140a40 con 0x7f8080072f30 2024-05-27T00:24:52.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8076a00640 1 -- 172.21.15.98:0/431216112 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6006 (secure 0 0 0) 0x7f80700555f0 con 0x7f8080072f30 2024-05-27T00:24:52.303 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:52.303 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:50.240256+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:24:52.303 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 42 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 -- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f805c0678c0 msgr2=0x7f805c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f805c0678c0 0x7f805c069d80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f808013fc50 tx=0x7f8068008040 comp rx=0 tx=0).stop 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 -- 172.21.15.98:0/431216112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8080072f30 msgr2=0x7f808006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8080072f30 0x7f808006d570 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f807000c450 tx=0x7f807000c910 comp rx=0 tx=0).stop 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 -- 172.21.15.98:0/431216112 shutdown_connections 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f808015c430 0x7f80801e4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f805c0678c0 0x7f805c069d80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80800c79f0 0x7f80801f0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 --2- 172.21.15.98:0/431216112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8080072f30 0x7f808006d570 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 -- 172.21.15.98:0/431216112 >> 172.21.15.98:0/431216112 conn(0x7f80800c9af0 msgr2=0x7f8080073310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 -- 172.21.15.98:0/431216112 shutdown_connections 2024-05-27T00:24:52.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:52.300+0000 7f8085400640 1 -- 172.21.15.98:0/431216112 wait complete. 2024-05-27T00:24:52.325 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 42, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 42, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5667": { "addr": "172.21.15.98:6834/1223646158", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 1223646158, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 1223646158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5667, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:24:50.240256+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5667 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:24:54.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:54.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.468+0000 7fceef200640 1 Processor -- start 2024-05-27T00:24:54.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.468+0000 7fceef200640 1 -- start start 2024-05-27T00:24:54.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceef200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fcef0157cd0 0x7fcef00c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:54.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceef200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcef01580b0 0x7fcef015e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:54.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceef200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef00c59d0 con 0x7fcef00b3cb0 2024-05-27T00:24:54.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceef200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fcef00c51e0 con 0x7fcef0157cd0 2024-05-27T00:24:54.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceef200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcef00cabb0 con 0x7fcef01580b0 2024-05-27T00:24:54.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceee800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcef00b3cb0 0x7fcef00c9a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51882/0 (socket says 172.21.15.98:51882) 2024-05-27T00:24:54.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceee800640 1 -- 172.21.15.98:0/1887589428 learned_addr learned my addr 172.21.15.98:0/1887589428 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:24:54.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fcee5400640 1 --2- 172.21.15.98:0/1887589428 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcef01580b0 0x7fcef015e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:54.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceede00640 1 --2- 172.21.15.98:0/1887589428 >> v2:172.21.15.204:3300/0 conn(0x7fcef0157cd0 0x7fcef00c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:54.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceed400640 1 -- 172.21.15.98:0/1887589428 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806269277 0 0) 0x7fcef00c59d0 con 0x7fcef00b3cb0 2024-05-27T00:24:54.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceed400640 1 -- 172.21.15.98:0/1887589428 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fced4003530 con 0x7fcef00b3cb0 2024-05-27T00:24:54.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceed400640 1 -- 172.21.15.98:0/1887589428 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3073972163 0 0) 0x7fced4003530 con 0x7fcef00b3cb0 2024-05-27T00:24:54.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceede00640 1 -- 172.21.15.98:0/1887589428 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcef01580b0 msgr2=0x7fcef015e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:54.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceede00640 1 --2- 172.21.15.98:0/1887589428 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcef01580b0 0x7fcef015e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:54.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceede00640 1 -- 172.21.15.98:0/1887589428 >> v1:172.21.15.98:6789/0 conn(0x7fcef00b3cb0 legacy=0x7fcef00c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:54.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.472+0000 7fceede00640 1 -- 172.21.15.98:0/1887589428 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef00cae50 con 0x7fcef0157cd0 2024-05-27T00:24:54.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceede00640 1 --2- 172.21.15.98:0/1887589428 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcef0157cd0 0x7fcef00c5490 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcee00097c0 tx=0x7fcee001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b502882b1e6c244a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:54.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceed400640 1 -- 172.21.15.98:0/1887589428 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee00b0070 con 0x7fcef0157cd0 2024-05-27T00:24:54.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceed400640 1 -- 172.21.15.98:0/1887589428 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee00ab400 con 0x7fcef0157cd0 2024-05-27T00:24:54.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceed400640 1 -- 172.21.15.98:0/1887589428 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee8001680 con 0x7fcef0157cd0 2024-05-27T00:24:54.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceef200640 1 -- 172.21.15.98:0/1887589428 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcef0157cd0 msgr2=0x7fcef00c5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:54.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceef200640 1 --2- 172.21.15.98:0/1887589428 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcef0157cd0 0x7fcef00c5490 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcee00097c0 tx=0x7fcee001c3c0 comp rx=0 tx=0).stop 2024-05-27T00:24:54.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceef200640 1 -- 172.21.15.98:0/1887589428 shutdown_connections 2024-05-27T00:24:54.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceef200640 1 --2- 172.21.15.98:0/1887589428 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcef01580b0 0x7fcef015e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:54.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceef200640 1 --2- 172.21.15.98:0/1887589428 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcef0157cd0 0x7fcef00c5490 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:54.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceef200640 1 -- 172.21.15.98:0/1887589428 >> 172.21.15.98:0/1887589428 conn(0x7fcef0070b30 msgr2=0x7fcef0070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:54.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceef200640 1 -- 172.21.15.98:0/1887589428 shutdown_connections 2024-05-27T00:24:54.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceef200640 1 -- 172.21.15.98:0/1887589428 wait complete. 2024-05-27T00:24:54.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceef200640 1 Processor -- start 2024-05-27T00:24:54.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.476+0000 7fceef200640 1 -- start start 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceef200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcef00b3cb0 0x7fcef01f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceef200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef0157cd0 0x7fcef0144b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceef200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcef01580b0 0x7fcef0147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceef200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcef00cb410 con 0x7fcef0157cd0 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceef200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcef015eb80 con 0x7fcef00b3cb0 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceef200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcef015a830 con 0x7fcef01580b0 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee5400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef0157cd0 0x7fcef0144b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee5400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef0157cd0 0x7fcef0144b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39028/0 (socket says 172.21.15.98:39028) 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee5400640 1 -- 172.21.15.98:0/3709505876 learned_addr learned my addr 172.21.15.98:0/3709505876 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceede00640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcef00b3cb0 0x7fcef01f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceee800640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcef01580b0 0x7fcef0147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee5400640 1 -- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcef01580b0 msgr2=0x7fcef0147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee5400640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcef01580b0 0x7fcef0147b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee5400640 1 -- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcef00b3cb0 msgr2=0x7fcef01f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee5400640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcef00b3cb0 0x7fcef01f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee5400640 1 -- 172.21.15.98:0/3709505876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef01528b0 con 0x7fcef0157cd0 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee5400640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef0157cd0 0x7fcef0144b50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fced8009d10 tx=0x7fced800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee6a00640 1 -- 172.21.15.98:0/3709505876 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8018020 con 0x7fcef0157cd0 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee6a00640 1 -- 172.21.15.98:0/3709505876 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fced8014480 con 0x7fcef0157cd0 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee6a00640 1 -- 172.21.15.98:0/3709505876 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8014830 con 0x7fcef0157cd0 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceef200640 1 -- 172.21.15.98:0/3709505876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef01522f0 con 0x7fcef0157cd0 2024-05-27T00:24:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceef200640 1 -- 172.21.15.98:0/3709505876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef01f2f20 con 0x7fcef0157cd0 2024-05-27T00:24:54.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fceef200640 1 -- 172.21.15.98:0/3709505876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef0075c40 con 0x7fcef0157cd0 2024-05-27T00:24:54.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.480+0000 7fcee6a00640 1 -- 172.21.15.98:0/3709505876 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fced801b020 con 0x7fcef0157cd0 2024-05-27T00:24:54.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.484+0000 7fcee6a00640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcecc0678c0 0x7fcecc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:54.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.484+0000 7fcee6a00640 1 -- 172.21.15.98:0/3709505876 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fced8086da0 con 0x7fcef0157cd0 2024-05-27T00:24:54.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.484+0000 7fceede00640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcecc0678c0 0x7fcecc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:54.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.484+0000 7fceede00640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcecc0678c0 0x7fcecc069d80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcee00ad000 tx=0x7fcee00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:54.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.488+0000 7fcee6a00640 1 -- 172.21.15.98:0/3709505876 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fced8022030 con 0x7fcef0157cd0 2024-05-27T00:24:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 -- 172.21.15.98:0/3709505876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcef00ca570 con 0x7fcef0157cd0 2024-05-27T00:24:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fcee6a00640 1 -- 172.21.15.98:0/3709505876 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6006 (secure 0 0 0) 0x7fced8010040 con 0x7fcef0157cd0 2024-05-27T00:24:54.682 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:54.683 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:50.240256+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:24:54.683 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 42 2024-05-27T00:24:54.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 -- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcecc0678c0 msgr2=0x7fcecc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:54.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcecc0678c0 0x7fcecc069d80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcee00ad000 tx=0x7fcee00a6000 comp rx=0 tx=0).stop 2024-05-27T00:24:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 -- 172.21.15.98:0/3709505876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef0157cd0 msgr2=0x7fcef0144b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef0157cd0 0x7fcef0144b50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fced8009d10 tx=0x7fced800c490 comp rx=0 tx=0).stop 2024-05-27T00:24:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 -- 172.21.15.98:0/3709505876 shutdown_connections 2024-05-27T00:24:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcef01580b0 0x7fcef0147b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcef0157cd0 0x7fcef0144b50 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fcecc0678c0 0x7fcecc069d80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 --2- 172.21.15.98:0/3709505876 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcef00b3cb0 0x7fcef01f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 -- 172.21.15.98:0/3709505876 >> 172.21.15.98:0/3709505876 conn(0x7fcef0070b30 msgr2=0x7fcef0158f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 -- 172.21.15.98:0/3709505876 shutdown_connections 2024-05-27T00:24:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:54.680+0000 7fceef200640 1 -- 172.21.15.98:0/3709505876 wait complete. 2024-05-27T00:24:54.704 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:24:56.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:56.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.844+0000 7f8737c00640 1 Processor -- start 2024-05-27T00:24:56.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.844+0000 7f8737c00640 1 -- start start 2024-05-27T00:24:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f8737c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f873015b470 0x7f87300c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f8737c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87300c1a90 con 0x7f87300c6e00 2024-05-27T00:24:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f8737c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87300c1d70 con 0x7f8730154830 2024-05-27T00:24:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f8737c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f873015b8d0 con 0x7f873015b470 2024-05-27T00:24:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f8735c00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f87300c6e00 0x7f8730071e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51890/0 (socket says 172.21.15.98:51890) 2024-05-27T00:24:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f8735c00640 1 -- 172.21.15.98:0/4104834821 learned_addr learned my addr 172.21.15.98:0/4104834821 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:24:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f8727e00640 1 --2- 172.21.15.98:0/4104834821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f873015b470 0x7f87300c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f872fe00640 1 -- 172.21.15.98:0/4104834821 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2395671753 0 0) 0x7f87300c1a90 con 0x7f87300c6e00 2024-05-27T00:24:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f872fe00640 1 -- 172.21.15.98:0/4104834821 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8714003530 con 0x7f87300c6e00 2024-05-27T00:24:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f872fe00640 1 -- 172.21.15.98:0/4104834821 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513065374 0 0) 0x7f87300c1d70 con 0x7f8730154830 2024-05-27T00:24:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f872fe00640 1 -- 172.21.15.98:0/4104834821 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87300c1a90 con 0x7f8730154830 2024-05-27T00:24:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f872fe00640 1 -- 172.21.15.98:0/4104834821 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 657922786 0 0) 0x7f8714003530 con 0x7f87300c6e00 2024-05-27T00:24:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f8727e00640 1 -- 172.21.15.98:0/4104834821 >> v1:172.21.15.204:6789/0 conn(0x7f8730154830 legacy=0x7f87300cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f8727e00640 1 -- 172.21.15.98:0/4104834821 >> v1:172.21.15.98:6789/0 conn(0x7f87300c6e00 legacy=0x7f8730071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:56.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.848+0000 7f8727e00640 1 -- 172.21.15.98:0/4104834821 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87300c2050 con 0x7f873015b470 2024-05-27T00:24:56.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f8727e00640 1 --2- 172.21.15.98:0/4104834821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f873015b470 0x7f87300c68c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f87180097f0 tx=0x7f871801c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75549aeeb5bed6af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f872fe00640 1 -- 172.21.15.98:0/4104834821 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87180b0070 con 0x7f873015b470 2024-05-27T00:24:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f872fe00640 1 -- 172.21.15.98:0/4104834821 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87180ab400 con 0x7f873015b470 2024-05-27T00:24:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f872fe00640 1 -- 172.21.15.98:0/4104834821 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728001a00 con 0x7f873015b470 2024-05-27T00:24:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f8737c00640 1 -- 172.21.15.98:0/4104834821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f873015b470 msgr2=0x7f87300c68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f8737c00640 1 --2- 172.21.15.98:0/4104834821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f873015b470 0x7f87300c68c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f87180097f0 tx=0x7f871801c400 comp rx=0 tx=0).stop 2024-05-27T00:24:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f8737c00640 1 -- 172.21.15.98:0/4104834821 shutdown_connections 2024-05-27T00:24:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f8737c00640 1 --2- 172.21.15.98:0/4104834821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f873015b470 0x7f87300c68c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f8737c00640 1 -- 172.21.15.98:0/4104834821 >> 172.21.15.98:0/4104834821 conn(0x7f8730067530 msgr2=0x7f873006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f8737c00640 1 -- 172.21.15.98:0/4104834821 shutdown_connections 2024-05-27T00:24:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.852+0000 7f8737c00640 1 -- 172.21.15.98:0/4104834821 wait complete. 2024-05-27T00:24:56.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8737c00640 1 Processor -- start 2024-05-27T00:24:56.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8737c00640 1 -- start start 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8737c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87300c6e00 0x7f873014fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8737c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8730154830 0x7f8730157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8737c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873015b470 0x7f87301f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8737c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87300725a0 con 0x7f873015b470 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8737c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f87300c2b10 con 0x7f87300c6e00 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8737c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8730072720 con 0x7f8730154830 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8735c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873015b470 0x7f87301f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8735c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873015b470 0x7f87301f57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39034/0 (socket says 172.21.15.98:39034) 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8735c00640 1 -- 172.21.15.98:0/3031887599 learned_addr learned my addr 172.21.15.98:0/3031887599 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8727e00640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8730154830 0x7f8730157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:56.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8735200640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87300c6e00 0x7f873014fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8735c00640 1 -- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8730154830 msgr2=0x7f8730157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:56.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8735c00640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8730154830 0x7f8730157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8735c00640 1 -- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87300c6e00 msgr2=0x7f873014fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8735c00640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87300c6e00 0x7f873014fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8735c00640 1 -- 172.21.15.98:0/3031887599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8730156280 con 0x7f873015b470 2024-05-27T00:24:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8735c00640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873015b470 0x7f87301f57d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f872800cc50 tx=0x7f872800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f872d600640 1 -- 172.21.15.98:0/3031887599 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872800ce60 con 0x7f873015b470 2024-05-27T00:24:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f872d600640 1 -- 172.21.15.98:0/3031887599 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8728012400 con 0x7f873015b470 2024-05-27T00:24:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f8737c00640 1 -- 172.21.15.98:0/3031887599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8730156560 con 0x7f873015b470 2024-05-27T00:24:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.856+0000 7f872d600640 1 -- 172.21.15.98:0/3031887599 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87280127f0 con 0x7f873015b470 2024-05-27T00:24:56.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.860+0000 7f8737c00640 1 -- 172.21.15.98:0/3031887599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87301f4fd0 con 0x7f873015b470 2024-05-27T00:24:56.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.860+0000 7f8737c00640 1 -- 172.21.15.98:0/3031887599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8730075cb0 con 0x7f873015b470 2024-05-27T00:24:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.860+0000 7f872d600640 1 -- 172.21.15.98:0/3031887599 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8728012990 con 0x7f873015b470 2024-05-27T00:24:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.864+0000 7f872d600640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f870c0678c0 0x7f870c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.864+0000 7f872d600640 1 -- 172.21.15.98:0/3031887599 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7f87280865c0 con 0x7f873015b470 2024-05-27T00:24:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.864+0000 7f8735200640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f870c0678c0 0x7f870c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.864+0000 7f8735200640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f870c0678c0 0x7f870c069d80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f87200011f0 tx=0x7f8720009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:56.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:56.868+0000 7f872d600640 1 -- 172.21.15.98:0/3031887599 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8728056870 con 0x7f873015b470 2024-05-27T00:24:57.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.072+0000 7f8737c00640 1 -- 172.21.15.98:0/3031887599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87300c1300 con 0x7f873015b470 2024-05-27T00:24:57.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.072+0000 7f872d600640 1 -- 172.21.15.98:0/3031887599 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6006 (secure 0 0 0) 0x7f872805ab70 con 0x7f873015b470 2024-05-27T00:24:57.077 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:57.077 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:50.240256+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:24:57.077 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 42 2024-05-27T00:24:57.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 -- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f870c0678c0 msgr2=0x7f870c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:57.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f870c0678c0 0x7f870c069d80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f87200011f0 tx=0x7f8720009040 comp rx=0 tx=0).stop 2024-05-27T00:24:57.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 -- 172.21.15.98:0/3031887599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873015b470 msgr2=0x7f87301f57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:57.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873015b470 0x7f87301f57d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f872800cc50 tx=0x7f872800a430 comp rx=0 tx=0).stop 2024-05-27T00:24:57.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 -- 172.21.15.98:0/3031887599 shutdown_connections 2024-05-27T00:24:57.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f873015b470 0x7f87301f57d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:57.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8730154830 0x7f8730157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:57.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f870c0678c0 0x7f870c069d80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:57.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 --2- 172.21.15.98:0/3031887599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87300c6e00 0x7f873014fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:57.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 -- 172.21.15.98:0/3031887599 >> 172.21.15.98:0/3031887599 conn(0x7f8730067530 msgr2=0x7f873015a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:57.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 -- 172.21.15.98:0/3031887599 shutdown_connections 2024-05-27T00:24:57.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:57.076+0000 7f8737c00640 1 -- 172.21.15.98:0/3031887599 wait complete. 2024-05-27T00:24:57.098 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:24:59.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:24:59.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5822000640 1 Processor -- start 2024-05-27T00:24:59.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5822000640 1 -- start start 2024-05-27T00:24:59.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5822000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0cdf90 0x7f581c06f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:59.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5822000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f581c0c4fd0 0x7f581c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:59.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5822000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f581c06cfb0 con 0x7f581c0c8fd0 2024-05-27T00:24:59.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5822000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f581c06d500 con 0x7f581c0c4fd0 2024-05-27T00:24:59.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5822000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f581c06ef50 con 0x7f581c0cdf90 2024-05-27T00:24:59.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5813e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f581c0c4fd0 0x7f581c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:59.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f581b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0cdf90 0x7f581c06f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:59.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f581be00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f581c0c8fd0 0x7f581c0caa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:46426/0 (socket says 172.21.15.98:46426) 2024-05-27T00:24:59.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5813e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f581c0c4fd0 0x7f581c0ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:37378/0 (socket says 172.21.15.98:37378) 2024-05-27T00:24:59.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f581be00640 1 -- 172.21.15.98:0/207586865 learned_addr learned my addr 172.21.15.98:0/207586865 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:24:59.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5813e00640 1 -- 172.21.15.98:0/207586865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0cdf90 msgr2=0x7f581c06f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:24:59.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5813e00640 1 --2- 172.21.15.98:0/207586865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0cdf90 0x7f581c06f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:59.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5813e00640 1 -- 172.21.15.98:0/207586865 >> v1:172.21.15.204:6789/0 conn(0x7f581c0c8fd0 legacy=0x7f581c0caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:59.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.240+0000 7f5813e00640 1 -- 172.21.15.98:0/207586865 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581c0ca840 con 0x7f581c0c4fd0 2024-05-27T00:24:59.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.244+0000 7f5813e00640 1 --2- 172.21.15.98:0/207586865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c0c4fd0 0x7f581c0ce370 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f58040097c0 tx=0x7f580401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b57db5214a09478a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:59.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.244+0000 7f581aa00640 1 -- 172.21.15.98:0/207586865 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58040b5070 con 0x7f581c0c4fd0 2024-05-27T00:24:59.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.244+0000 7f581aa00640 1 -- 172.21.15.98:0/207586865 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58040a8990 con 0x7f581c0c4fd0 2024-05-27T00:24:59.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.244+0000 7f581aa00640 1 -- 172.21.15.98:0/207586865 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58040a8d60 con 0x7f581c0c4fd0 2024-05-27T00:24:59.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.248+0000 7f5822000640 1 -- 172.21.15.98:0/207586865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c0c4fd0 msgr2=0x7f581c0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:59.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.248+0000 7f5822000640 1 --2- 172.21.15.98:0/207586865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c0c4fd0 0x7f581c0ce370 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f58040097c0 tx=0x7f580401ec90 comp rx=0 tx=0).stop 2024-05-27T00:24:59.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.248+0000 7f5822000640 1 -- 172.21.15.98:0/207586865 shutdown_connections 2024-05-27T00:24:59.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.248+0000 7f5822000640 1 --2- 172.21.15.98:0/207586865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c0c4fd0 0x7f581c0ce370 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:59.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.248+0000 7f5822000640 1 --2- 172.21.15.98:0/207586865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0cdf90 0x7f581c06f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:59.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.248+0000 7f5822000640 1 -- 172.21.15.98:0/207586865 >> 172.21.15.98:0/207586865 conn(0x7f581c06d680 msgr2=0x7f581c06da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:59.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.248+0000 7f5822000640 1 -- 172.21.15.98:0/207586865 shutdown_connections 2024-05-27T00:24:59.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.248+0000 7f5822000640 1 -- 172.21.15.98:0/207586865 wait complete. 2024-05-27T00:24:59.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.248+0000 7f5822000640 1 Processor -- start 2024-05-27T00:24:59.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.248+0000 7f5822000640 1 -- start start 2024-05-27T00:24:59.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5822000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c0c4fd0 0x7f581c1f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:59.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5822000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0c8fd0 0x7f581c1f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:59.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5822000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c0cdf90 0x7f581c0cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:59.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5822000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f581c06d220 con 0x7f581c0c4fd0 2024-05-27T00:24:59.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5822000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f581c06b610 con 0x7f581c0cdf90 2024-05-27T00:24:59.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5822000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f581c06b8b0 con 0x7f581c0c8fd0 2024-05-27T00:24:59.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f581b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c0c4fd0 0x7f581c1f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:59.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f581b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c0c4fd0 0x7f581c1f43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39042/0 (socket says 172.21.15.98:39042) 2024-05-27T00:24:59.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f581b400640 1 -- 172.21.15.98:0/2389774877 learned_addr learned my addr 172.21.15.98:0/2389774877 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:24:59.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f581be00640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c0cdf90 0x7f581c0cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:59.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5813e00640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0c8fd0 0x7f581c1f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:59.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f581b400640 1 -- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0c8fd0 msgr2=0x7f581c1f0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:59.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f581b400640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0c8fd0 0x7f581c1f0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f581b400640 1 -- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c0cdf90 msgr2=0x7f581c0cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f581b400640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c0cdf90 0x7f581c0cd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f581b400640 1 -- 172.21.15.98:0/2389774877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581c1f1390 con 0x7f581c0c4fd0 2024-05-27T00:24:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f581b400640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c0c4fd0 0x7f581c1f43c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f580c00e870 tx=0x7f580c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5813400640 1 -- 172.21.15.98:0/2389774877 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c017070 con 0x7f581c0c4fd0 2024-05-27T00:24:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5813400640 1 -- 172.21.15.98:0/2389774877 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580c00c940 con 0x7f581c0c4fd0 2024-05-27T00:24:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5822000640 1 -- 172.21.15.98:0/2389774877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f581c14d190 con 0x7f581c0c4fd0 2024-05-27T00:24:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5813400640 1 -- 172.21.15.98:0/2389774877 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c00ccf0 con 0x7f581c0c4fd0 2024-05-27T00:24:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5822000640 1 -- 172.21.15.98:0/2389774877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f581c14d750 con 0x7f581c0c4fd0 2024-05-27T00:24:59.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.252+0000 7f5822000640 1 -- 172.21.15.98:0/2389774877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f581c14d990 con 0x7f581c0c4fd0 2024-05-27T00:24:59.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.256+0000 7f5813400640 1 -- 172.21.15.98:0/2389774877 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f580c01a070 con 0x7f581c0c4fd0 2024-05-27T00:24:59.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.256+0000 7f5813400640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f57f80678c0 0x7f57f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:24:59.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.256+0000 7f5813400640 1 -- 172.21.15.98:0/2389774877 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7f580c085e90 con 0x7f581c0c4fd0 2024-05-27T00:24:59.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.256+0000 7f5813e00640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f57f80678c0 0x7f57f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:24:59.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.256+0000 7f5813e00640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f57f80678c0 0x7f57f8069d80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f58040b2000 tx=0x7f5804008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:24:59.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.260+0000 7f5813400640 1 -- 172.21.15.98:0/2389774877 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f581c14d990 con 0x7f581c0c4fd0 2024-05-27T00:24:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.460+0000 7f5822000640 1 -- 172.21.15.98:0/2389774877 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f581c075cb0 con 0x7f581c0c4fd0 2024-05-27T00:24:59.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.460+0000 7f5813400640 1 -- 172.21.15.98:0/2389774877 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6006 (secure 0 0 0) 0x7f580c0566f0 con 0x7f581c0c4fd0 2024-05-27T00:24:59.465 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:24:59.465 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:24:50.240256+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:24:59.465 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 42 2024-05-27T00:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 -- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f57f80678c0 msgr2=0x7f57f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f57f80678c0 0x7f57f8069d80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f58040b2000 tx=0x7f5804008040 comp rx=0 tx=0).stop 2024-05-27T00:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 -- 172.21.15.98:0/2389774877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c0c4fd0 msgr2=0x7f581c1f43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c0c4fd0 0x7f581c1f43c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f580c00e870 tx=0x7f580c00ec70 comp rx=0 tx=0).stop 2024-05-27T00:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 -- 172.21.15.98:0/2389774877 shutdown_connections 2024-05-27T00:24:59.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f57f80678c0 0x7f57f8069d80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:59.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c0cdf90 0x7f581c0cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:59.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0c8fd0 0x7f581c1f0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:59.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 --2- 172.21.15.98:0/2389774877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f581c0c4fd0 0x7f581c1f43c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:24:59.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 -- 172.21.15.98:0/2389774877 >> 172.21.15.98:0/2389774877 conn(0x7f581c06d680 msgr2=0x7f581c0ce370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:24:59.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 -- 172.21.15.98:0/2389774877 shutdown_connections 2024-05-27T00:24:59.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:24:59.464+0000 7f5822000640 1 -- 172.21.15.98:0/2389774877 wait complete. 2024-05-27T00:24:59.487 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:25:01.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.624+0000 7fc312400640 1 Processor -- start 2024-05-27T00:25:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc312400640 1 -- start start 2024-05-27T00:25:01.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc312400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc30c1514d0 0x7fc30c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:01.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc312400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0c7a60 0x7fc30c06e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:01.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc312400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc30c06c940 con 0x7fc30c0728d0 2024-05-27T00:25:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc312400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc30c073910 con 0x7fc30c1514d0 2024-05-27T00:25:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc312400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c072d90 con 0x7fc30c0c7a60 2024-05-27T00:25:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc311000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc30c1514d0 0x7fc30c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc311000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc30c1514d0 0x7fc30c06e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39050/0 (socket says 172.21.15.98:39050) 2024-05-27T00:25:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc311000640 1 -- 172.21.15.98:0/2327366471 learned_addr learned my addr 172.21.15.98:0/2327366471 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc30be00640 1 --2- 172.21.15.98:0/2327366471 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0c7a60 0x7fc30c06e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:01.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc311000640 1 -- 172.21.15.98:0/2327366471 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0c7a60 msgr2=0x7fc30c06e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:01.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc311000640 1 --2- 172.21.15.98:0/2327366471 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0c7a60 0x7fc30c06e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc311000640 1 -- 172.21.15.98:0/2327366471 >> v1:172.21.15.204:6789/0 conn(0x7fc30c0728d0 legacy=0x7fc30c15aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:01.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.628+0000 7fc311000640 1 -- 172.21.15.98:0/2327366471 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c072fe0 con 0x7fc30c1514d0 2024-05-27T00:25:01.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc311000640 1 --2- 172.21.15.98:0/2327366471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c1514d0 0x7fc30c06e090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc300009870 tx=0x7fc30001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922e362d67175324 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:01.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc30b400640 1 -- 172.21.15.98:0/2327366471 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3000b1070 con 0x7fc30c1514d0 2024-05-27T00:25:01.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc30b400640 1 -- 172.21.15.98:0/2327366471 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3000ac620 con 0x7fc30c1514d0 2024-05-27T00:25:01.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc30b400640 1 -- 172.21.15.98:0/2327366471 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3000ac9f0 con 0x7fc30c1514d0 2024-05-27T00:25:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc312400640 1 -- 172.21.15.98:0/2327366471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c1514d0 msgr2=0x7fc30c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc312400640 1 --2- 172.21.15.98:0/2327366471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c1514d0 0x7fc30c06e090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc300009870 tx=0x7fc30001c4d0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc312400640 1 -- 172.21.15.98:0/2327366471 shutdown_connections 2024-05-27T00:25:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc312400640 1 --2- 172.21.15.98:0/2327366471 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0c7a60 0x7fc30c06e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc312400640 1 --2- 172.21.15.98:0/2327366471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c1514d0 0x7fc30c06e090 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc312400640 1 -- 172.21.15.98:0/2327366471 >> 172.21.15.98:0/2327366471 conn(0x7fc30c0cd060 msgr2=0x7fc30c0cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc312400640 1 -- 172.21.15.98:0/2327366471 shutdown_connections 2024-05-27T00:25:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc312400640 1 -- 172.21.15.98:0/2327366471 wait complete. 2024-05-27T00:25:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc312400640 1 Processor -- start 2024-05-27T00:25:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.632+0000 7fc312400640 1 -- start start 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc312400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0728d0 0x7fc30c14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc312400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c0c7a60 0x7fc30c14bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc312400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc30c1514d0 0x7fc30c148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc312400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c151a30 con 0x7fc30c0c7a60 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc312400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c1518b0 con 0x7fc30c1514d0 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc312400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c1643a0 con 0x7fc30c0728d0 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc30be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c0c7a60 0x7fc30c14bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc30be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c0c7a60 0x7fc30c14bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39052/0 (socket says 172.21.15.98:39052) 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc30be00640 1 -- 172.21.15.98:0/3976694387 learned_addr learned my addr 172.21.15.98:0/3976694387 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc311000640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0728d0 0x7fc30c14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc311a00640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc30c1514d0 0x7fc30c148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:01.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc30be00640 1 -- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0728d0 msgr2=0x7fc30c14b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc30be00640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0728d0 0x7fc30c14b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc30be00640 1 -- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc30c1514d0 msgr2=0x7fc30c148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc30be00640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc30c1514d0 0x7fc30c148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc30be00640 1 -- 172.21.15.98:0/3976694387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c14c270 con 0x7fc30c0c7a60 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc311000640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0728d0 0x7fc30c14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc30be00640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c0c7a60 0x7fc30c14bd30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800d5f0 tx=0x7fc2f800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc308c00640 1 -- 172.21.15.98:0/3976694387 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f80125f0 con 0x7fc30c0c7a60 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc308c00640 1 -- 172.21.15.98:0/3976694387 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f8019070 con 0x7fc30c0c7a60 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc312400640 1 -- 172.21.15.98:0/3976694387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c1f75a0 con 0x7fc30c0c7a60 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc308c00640 1 -- 172.21.15.98:0/3976694387 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f80129b0 con 0x7fc30c0c7a60 2024-05-27T00:25:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc312400640 1 -- 172.21.15.98:0/3976694387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c1f3850 con 0x7fc30c0c7a60 2024-05-27T00:25:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc312400640 1 -- 172.21.15.98:0/3976694387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30c075c40 con 0x7fc30c0c7a60 2024-05-27T00:25:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.636+0000 7fc308c00640 1 -- 172.21.15.98:0/3976694387 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc2f8012b50 con 0x7fc30c0c7a60 2024-05-27T00:25:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.640+0000 7fc308c00640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2ec0678c0 0x7fc2ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:01.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.640+0000 7fc308c00640 1 -- 172.21.15.98:0/3976694387 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fc2f8086330 con 0x7fc30c0c7a60 2024-05-27T00:25:01.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.640+0000 7fc311000640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2ec0678c0 0x7fc2ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:01.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.640+0000 7fc311000640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2ec0678c0 0x7fc2ec069d80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc3000ae000 tx=0x7fc300008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:01.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.644+0000 7fc308c00640 1 -- 172.21.15.98:0/3976694387 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc2f8056ba0 con 0x7fc30c0c7a60 2024-05-27T00:25:01.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.832+0000 7fc312400640 1 -- 172.21.15.98:0/3976694387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc30c0cd960 con 0x7fc30c0c7a60 2024-05-27T00:25:01.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.832+0000 7fc308c00640 1 -- 172.21.15.98:0/3976694387 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6010 (secure 0 0 0) 0x7fc2f80564a0 con 0x7fc30c0c7a60 2024-05-27T00:25:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:01.839 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:01.461447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 43 2024-05-27T00:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 -- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2ec0678c0 msgr2=0x7fc2ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2ec0678c0 0x7fc2ec069d80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc3000ae000 tx=0x7fc300008040 comp rx=0 tx=0).stop 2024-05-27T00:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 -- 172.21.15.98:0/3976694387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c0c7a60 msgr2=0x7fc30c14bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:01.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c0c7a60 0x7fc30c14bd30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800d5f0 tx=0x7fc2f800dab0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 -- 172.21.15.98:0/3976694387 shutdown_connections 2024-05-27T00:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2ec0678c0 0x7fc2ec069d80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc30c1514d0 0x7fc30c148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc30c0c7a60 0x7fc30c14bd30 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 --2- 172.21.15.98:0/3976694387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc30c0728d0 0x7fc30c14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 -- 172.21.15.98:0/3976694387 >> 172.21.15.98:0/3976694387 conn(0x7fc30c0cd060 msgr2=0x7fc30c0cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 -- 172.21.15.98:0/3976694387 shutdown_connections 2024-05-27T00:25:01.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:01.836+0000 7fc312400640 1 -- 172.21.15.98:0/3976694387 wait complete. 2024-05-27T00:25:01.857 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:25:03.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:04.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.000+0000 7f4801200640 1 Processor -- start 2024-05-27T00:25:04.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.000+0000 7f4801200640 1 -- start start 2024-05-27T00:25:04.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f4801200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f47fc0735d0 0x7f47fc151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:04.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f4801200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 0x7f47fc151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:04.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f4801200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47fc0c17d0 con 0x7f47fc0731f0 2024-05-27T00:25:04.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f4801200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f47fc071990 con 0x7f47fc0735d0 2024-05-27T00:25:04.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f4801200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f47fc070e10 con 0x7f47fc0c79f0 2024-05-27T00:25:04.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f47f1400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f47fc0735d0 0x7f47fc151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:04.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f47fa800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 0x7f47fc151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:04.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f47f1400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f47fc0735d0 0x7f47fc151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:38954/0 (socket says 172.21.15.98:38954) 2024-05-27T00:25:04.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f47f1400640 1 -- 172.21.15.98:0/2380380577 learned_addr learned my addr 172.21.15.98:0/2380380577 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:04.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f47f1400640 1 -- 172.21.15.98:0/2380380577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 msgr2=0x7f47fc151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:04.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f47f1400640 1 --2- 172.21.15.98:0/2380380577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 0x7f47fc151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:04.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f47f1400640 1 -- 172.21.15.98:0/2380380577 >> v1:172.21.15.204:6789/0 conn(0x7f47fc0731f0 legacy=0x7f47fc0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:04.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f47f1400640 1 -- 172.21.15.98:0/2380380577 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc0710f0 con 0x7f47fc0735d0 2024-05-27T00:25:04.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.004+0000 7f47fa800640 1 --2- 172.21.15.98:0/2380380577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 0x7f47fc151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:25:04.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f47f1400640 1 --2- 172.21.15.98:0/2380380577 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f47fc0735d0 0x7f47fc151430 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009870 tx=0x7f47e401ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb51761f7501a4a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:04.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f47f9400640 1 -- 172.21.15.98:0/2380380577 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e40b5070 con 0x7f47fc0735d0 2024-05-27T00:25:04.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f47f9400640 1 -- 172.21.15.98:0/2380380577 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47e40a8a50 con 0x7f47fc0735d0 2024-05-27T00:25:04.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f47f9400640 1 -- 172.21.15.98:0/2380380577 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e40a8e20 con 0x7f47fc0735d0 2024-05-27T00:25:04.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f4801200640 1 -- 172.21.15.98:0/2380380577 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f47fc0735d0 msgr2=0x7f47fc151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:04.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f4801200640 1 --2- 172.21.15.98:0/2380380577 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f47fc0735d0 0x7f47fc151430 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009870 tx=0x7f47e401ed50 comp rx=0 tx=0).stop 2024-05-27T00:25:04.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f4801200640 1 -- 172.21.15.98:0/2380380577 shutdown_connections 2024-05-27T00:25:04.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f4801200640 1 --2- 172.21.15.98:0/2380380577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 0x7f47fc151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:04.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f4801200640 1 --2- 172.21.15.98:0/2380380577 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f47fc0735d0 0x7f47fc151430 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:04.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f4801200640 1 -- 172.21.15.98:0/2380380577 >> 172.21.15.98:0/2380380577 conn(0x7f47fc067530 msgr2=0x7f47fc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:04.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f4801200640 1 -- 172.21.15.98:0/2380380577 shutdown_connections 2024-05-27T00:25:04.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f4801200640 1 -- 172.21.15.98:0/2380380577 wait complete. 2024-05-27T00:25:04.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f4801200640 1 Processor -- start 2024-05-27T00:25:04.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.008+0000 7f4801200640 1 -- start start 2024-05-27T00:25:04.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f4801200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc0731f0 0x7f47fc15b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f4801200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f47fc0735d0 0x7f47fc1564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f4801200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 0x7f47fc158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f4801200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc0c8400 con 0x7f47fc0731f0 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f4801200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc0716b0 con 0x7f47fc0735d0 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f4801200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f47fc157970 con 0x7f47fc0c79f0 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc0731f0 0x7f47fc15b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc0731f0 0x7f47fc15b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51352/0 (socket says 172.21.15.98:51352) 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f9e00640 1 -- 172.21.15.98:0/686493519 learned_addr learned my addr 172.21.15.98:0/686493519 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47fa800640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 0x7f47fc158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f1400640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f47fc0735d0 0x7f47fc1564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f9e00640 1 -- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 msgr2=0x7f47fc158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:04.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f9e00640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 0x7f47fc158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f9e00640 1 -- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f47fc0735d0 msgr2=0x7f47fc1564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f9e00640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f47fc0735d0 0x7f47fc1564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f9e00640 1 -- 172.21.15.98:0/686493519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc15b740 con 0x7f47fc0731f0 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f9e00640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc0731f0 0x7f47fc15b200 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00c820 tx=0x7f47ec00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f2a00640 1 -- 172.21.15.98:0/686493519 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ec003b80 con 0x7f47fc0731f0 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f2a00640 1 -- 172.21.15.98:0/686493519 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47ec003d20 con 0x7f47fc0731f0 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f4801200640 1 -- 172.21.15.98:0/686493519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47fc1e5780 con 0x7f47fc0731f0 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f47f2a00640 1 -- 172.21.15.98:0/686493519 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ec00aa90 con 0x7f47fc0731f0 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f4801200640 1 -- 172.21.15.98:0/686493519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47fc1ee690 con 0x7f47fc0731f0 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.012+0000 7f4801200640 1 -- 172.21.15.98:0/686493519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47fc075cb0 con 0x7f47fc0731f0 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.016+0000 7f47f2a00640 1 -- 172.21.15.98:0/686493519 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f47ec00ac30 con 0x7f47fc0731f0 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.016+0000 7f47f2a00640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f47d80678c0 0x7f47d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.016+0000 7f47f2a00640 1 -- 172.21.15.98:0/686493519 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7f47ec085fb0 con 0x7f47fc0731f0 2024-05-27T00:25:04.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.016+0000 7f47f1400640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f47d80678c0 0x7f47d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.016+0000 7f47f1400640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f47d80678c0 0x7f47d8069d80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f47e40b2000 tx=0x7f47e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:04.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.020+0000 7f47f2a00640 1 -- 172.21.15.98:0/686493519 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f47ec056820 con 0x7f47fc0731f0 2024-05-27T00:25:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 -- 172.21.15.98:0/686493519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47fc158510 con 0x7f47fc0731f0 2024-05-27T00:25:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f47f2a00640 1 -- 172.21.15.98:0/686493519 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6010 (secure 0 0 0) 0x7f47ec056120 con 0x7f47fc0731f0 2024-05-27T00:25:04.210 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:04.210 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:01.461447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:25:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 43 2024-05-27T00:25:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 -- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f47d80678c0 msgr2=0x7f47d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f47d80678c0 0x7f47d8069d80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f47e40b2000 tx=0x7f47e4008040 comp rx=0 tx=0).stop 2024-05-27T00:25:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 -- 172.21.15.98:0/686493519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc0731f0 msgr2=0x7f47fc15b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc0731f0 0x7f47fc15b200 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00c820 tx=0x7f47ec00cce0 comp rx=0 tx=0).stop 2024-05-27T00:25:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 -- 172.21.15.98:0/686493519 shutdown_connections 2024-05-27T00:25:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47fc0c79f0 0x7f47fc158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f47d80678c0 0x7f47d8069d80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f47fc0735d0 0x7f47fc1564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 --2- 172.21.15.98:0/686493519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47fc0731f0 0x7f47fc15b200 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 -- 172.21.15.98:0/686493519 >> 172.21.15.98:0/686493519 conn(0x7f47fc067530 msgr2=0x7f47fc14fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 -- 172.21.15.98:0/686493519 shutdown_connections 2024-05-27T00:25:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:04.208+0000 7f4801200640 1 -- 172.21.15.98:0/686493519 wait complete. 2024-05-27T00:25:04.232 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 43, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 43, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5667": { "addr": "172.21.15.98:6834/1223646158", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 1223646158, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 1223646158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5667, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:25:01.461447+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5667 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:25:06.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:06.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.376+0000 7f98c5200640 1 Processor -- start 2024-05-27T00:25:06.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.376+0000 7f98c5200640 1 -- start start 2024-05-27T00:25:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98c5200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c00735d0 0x7f98c0151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98c5200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c00c17d0 con 0x7f98c00731f0 2024-05-27T00:25:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98c5200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c0070e10 con 0x7f98c00c79f0 2024-05-27T00:25:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98c5200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f98c0071990 con 0x7f98c00735d0 2024-05-27T00:25:06.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bde00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f98c00731f0 0x7f98c00c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44690/0 (socket says 172.21.15.98:44690) 2024-05-27T00:25:06.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bde00640 1 -- 172.21.15.98:0/4152196030 learned_addr learned my addr 172.21.15.98:0/4152196030 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:25:06.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98b7e00640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c00735d0 0x7f98c0151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:06.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3925195821 0 0) 0x7f98c00c17d0 con 0x7f98c00731f0 2024-05-27T00:25:06.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98a4005740 con 0x7f98c00731f0 2024-05-27T00:25:06.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2661294014 0 0) 0x7f98a4005740 con 0x7f98c00731f0 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c00735d0 msgr2=0x7f98c0151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c00735d0 0x7f98c0151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 >> v1:172.21.15.204:6789/0 conn(0x7f98c00c79f0 legacy=0x7f98c0157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c00710f0 con 0x7f98c00731f0 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f98b0001680 con 0x7f98c00731f0 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 >> v1:172.21.15.98:6789/0 conn(0x7f98c00731f0 legacy=0x7f98c00c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c00735d0 0x7f98a4005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a4006180 0x7f98a4006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98a4006f60 0x7f98a40073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98c0071990 con 0x7f98a4006180 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f98a400dd50 con 0x7f98a4006f60 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f98a400df50 con 0x7f98c00735d0 2024-05-27T00:25:06.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98b7e00640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a4006180 0x7f98a4006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98bde00640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c00735d0 0x7f98a4005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98be800640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98a4006f60 0x7f98a40073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98b7e00640 1 -- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c00735d0 msgr2=0x7f98a4005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98b7e00640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c00735d0 0x7f98a4005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98b7e00640 1 -- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98a4006f60 msgr2=0x7f98a40073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98b7e00640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98a4006f60 0x7f98a40073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.380+0000 7f98b7e00640 1 -- 172.21.15.98:0/4152196030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a4005740 con 0x7f98a4006180 2024-05-27T00:25:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98b7e00640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a4006180 0x7f98a4006a20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009ab0 tx=0x7f98a801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a364a70dbc96e731 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a80b2070 con 0x7f98a4006180 2024-05-27T00:25:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98a80a5640 con 0x7f98a4006180 2024-05-27T00:25:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98bd400640 1 -- 172.21.15.98:0/4152196030 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a80096b0 con 0x7f98a4006180 2024-05-27T00:25:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98c5200640 1 -- 172.21.15.98:0/4152196030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a4006180 msgr2=0x7f98a4006a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98c5200640 1 --2- 172.21.15.98:0/4152196030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98a4006180 0x7f98a4006a20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009ab0 tx=0x7f98a801ca50 comp rx=0 tx=0).stop 2024-05-27T00:25:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98be800640 1 -- 172.21.15.98:0/4152196030 reap_dead start 2024-05-27T00:25:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98c5200640 1 -- 172.21.15.98:0/4152196030 shutdown_connections 2024-05-27T00:25:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98c5200640 1 -- 172.21.15.98:0/4152196030 >> 172.21.15.98:0/4152196030 conn(0x7f98c0067530 msgr2=0x7f98c006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98c5200640 1 -- 172.21.15.98:0/4152196030 shutdown_connections 2024-05-27T00:25:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98c5200640 1 -- 172.21.15.98:0/4152196030 wait complete. 2024-05-27T00:25:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98c5200640 1 Processor -- start 2024-05-27T00:25:06.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.384+0000 7f98c5200640 1 -- start start 2024-05-27T00:25:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98c5200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c014e020 0x7f98c014d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98c5200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c014ac00 0x7f98c014e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98c5200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98c01475b0 0x7f98c0147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98c5200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98c00c8400 con 0x7f98c014ac00 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98c5200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f98c00716b0 con 0x7f98c01475b0 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98c5200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f98c0158080 con 0x7f98c014e020 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c014ac00 0x7f98c014e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c014ac00 0x7f98c014e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51364/0 (socket says 172.21.15.98:51364) 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b7e00640 1 -- 172.21.15.98:0/1714275821 learned_addr learned my addr 172.21.15.98:0/1714275821 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98bde00640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c014e020 0x7f98c014d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98be800640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98c01475b0 0x7f98c0147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b7e00640 1 -- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c014e020 msgr2=0x7f98c014d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b7e00640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c014e020 0x7f98c014d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b7e00640 1 -- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98c01475b0 msgr2=0x7f98c0147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b7e00640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98c01475b0 0x7f98c0147a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b7e00640 1 -- 172.21.15.98:0/1714275821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c01ed500 con 0x7f98c014ac00 2024-05-27T00:25:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b7e00640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c014ac00 0x7f98c014e400 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f98a801c880 tx=0x7f98a80197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b6000640 1 -- 172.21.15.98:0/1714275821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a80bd070 con 0x7f98c014ac00 2024-05-27T00:25:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b6000640 1 -- 172.21.15.98:0/1714275821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98a80ad440 con 0x7f98c014ac00 2024-05-27T00:25:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b6000640 1 -- 172.21.15.98:0/1714275821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a80b2040 con 0x7f98c014ac00 2024-05-27T00:25:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98c5200640 1 -- 172.21.15.98:0/1714275821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c01ec6a0 con 0x7f98c014ac00 2024-05-27T00:25:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98c5200640 1 -- 172.21.15.98:0/1714275821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c01ecc60 con 0x7f98c014ac00 2024-05-27T00:25:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98c5200640 1 -- 172.21.15.98:0/1714275821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c0075cb0 con 0x7f98c014ac00 2024-05-27T00:25:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.388+0000 7f98b6000640 1 -- 172.21.15.98:0/1714275821 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f98a80c0070 con 0x7f98c014ac00 2024-05-27T00:25:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.392+0000 7f98b6000640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f989c0678f0 0x7f989c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.392+0000 7f98b6000640 1 -- 172.21.15.98:0/1714275821 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7f98a811fbd0 con 0x7f98c014ac00 2024-05-27T00:25:06.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.392+0000 7f98bde00640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f989c0678f0 0x7f989c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:06.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.392+0000 7f98bde00640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f989c0678f0 0x7f989c069db0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f98b0004850 tx=0x7f98b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:06.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.396+0000 7f98b6000640 1 -- 172.21.15.98:0/1714275821 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f98a8016760 con 0x7f98c014ac00 2024-05-27T00:25:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 -- 172.21.15.98:0/1714275821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98c00c6d10 con 0x7f98c014ac00 2024-05-27T00:25:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98b6000640 1 -- 172.21.15.98:0/1714275821 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6010 (secure 0 0 0) 0x7f98a811f070 con 0x7f98c014ac00 2024-05-27T00:25:06.591 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:06.591 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:01.461447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:25:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 43 2024-05-27T00:25:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 -- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f989c0678f0 msgr2=0x7f989c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f989c0678f0 0x7f989c069db0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f98b0004850 tx=0x7f98b0009040 comp rx=0 tx=0).stop 2024-05-27T00:25:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 -- 172.21.15.98:0/1714275821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c014ac00 msgr2=0x7f98c014e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c014ac00 0x7f98c014e400 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f98a801c880 tx=0x7f98a80197f0 comp rx=0 tx=0).stop 2024-05-27T00:25:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 -- 172.21.15.98:0/1714275821 shutdown_connections 2024-05-27T00:25:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f989c0678f0 0x7f989c069db0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98c01475b0 0x7f98c0147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98c014ac00 0x7f98c014e400 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 --2- 172.21.15.98:0/1714275821 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98c014e020 0x7f98c014d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 -- 172.21.15.98:0/1714275821 >> 172.21.15.98:0/1714275821 conn(0x7f98c0067530 msgr2=0x7f98c01516e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 -- 172.21.15.98:0/1714275821 shutdown_connections 2024-05-27T00:25:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:06.588+0000 7f98c5200640 1 -- 172.21.15.98:0/1714275821 wait complete. 2024-05-27T00:25:06.612 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:25:08.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:08.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd0da00640 1 Processor -- start 2024-05-27T00:25:08.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd0da00640 1 -- start start 2024-05-27T00:25:08.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd0da00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08152a50 0x7efd080709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:08.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd0da00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0815be40 con 0x7efd080cb7e0 2024-05-27T00:25:08.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd0da00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd080c50e0 con 0x7efd08158e60 2024-05-27T00:25:08.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd0da00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efd080c8a00 con 0x7efd08152a50 2024-05-27T00:25:08.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd06000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efd080cb7e0 0x7efd08158750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44698/0 (socket says 172.21.15.98:44698) 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd06000640 1 -- 172.21.15.98:0/1592630811 learned_addr learned my addr 172.21.15.98:0/1592630811 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd06a00640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08152a50 0x7efd080709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 973963705 0 0) 0x7efd0815be40 con 0x7efd080cb7e0 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcec003530 con 0x7efd080cb7e0 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 17617751 0 0) 0x7efcec003530 con 0x7efd080cb7e0 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08152a50 msgr2=0x7efd080709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08152a50 0x7efd080709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 >> v1:172.21.15.204:6789/0 conn(0x7efd08158e60 legacy=0x7efd08162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd080c8c50 con 0x7efd080cb7e0 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7efcf40016a0 con 0x7efd080cb7e0 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 >> v1:172.21.15.98:6789/0 conn(0x7efd080cb7e0 legacy=0x7efd08158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcec0039f0 0x7efcec003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:08.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 0x7efcec004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08152a50 0x7efcec005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd080c8a00 con 0x7efcec004810 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efcec00bb60 con 0x7efcec0039f0 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efcec00bd60 con 0x7efd08152a50 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd06000640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 0x7efcec004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd06a00640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcec0039f0 0x7efcec003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd07400640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08152a50 0x7efcec005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd06000640 1 -- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08152a50 msgr2=0x7efcec005280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd06000640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08152a50 0x7efcec005280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd06000640 1 -- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcec0039f0 msgr2=0x7efcec003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd06000640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcec0039f0 0x7efcec003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:08.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.760+0000 7efd06000640 1 -- 172.21.15.98:0/1592630811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcec003530 con 0x7efcec004810 2024-05-27T00:25:08.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd06000640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 0x7efcec004c90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7efd0815c010 tx=0x7efcf401cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ea6577dbae16df server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:08.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4007c90 con 0x7efcec004810 2024-05-27T00:25:08.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf40a9070 con 0x7efcec004810 2024-05-27T00:25:08.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd05600640 1 -- 172.21.15.98:0/1592630811 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4005ce0 con 0x7efcec004810 2024-05-27T00:25:08.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd0da00640 1 -- 172.21.15.98:0/1592630811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 msgr2=0x7efcec004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:08.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd0da00640 1 --2- 172.21.15.98:0/1592630811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 0x7efcec004c90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7efd0815c010 tx=0x7efcf401cb70 comp rx=0 tx=0).stop 2024-05-27T00:25:08.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd07400640 1 -- 172.21.15.98:0/1592630811 reap_dead start 2024-05-27T00:25:08.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd0da00640 1 -- 172.21.15.98:0/1592630811 shutdown_connections 2024-05-27T00:25:08.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd0da00640 1 -- 172.21.15.98:0/1592630811 >> 172.21.15.98:0/1592630811 conn(0x7efd08072f90 msgr2=0x7efd080733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:08.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd0da00640 1 -- 172.21.15.98:0/1592630811 shutdown_connections 2024-05-27T00:25:08.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.764+0000 7efd0da00640 1 -- 172.21.15.98:0/1592630811 wait complete. 2024-05-27T00:25:08.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd0da00640 1 Processor -- start 2024-05-27T00:25:08.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd0da00640 1 -- start start 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd0da00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 0x7efd081f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd0da00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08155eb0 0x7efd080cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd0da00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd080c6fe0 0x7efd080c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd0da00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd08152e30 con 0x7efcec004810 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd0da00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd080cb570 con 0x7efd080c6fe0 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd0da00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efd080c9c10 con 0x7efd08155eb0 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 0x7efd081f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 0x7efd081f35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51386/0 (socket says 172.21.15.98:51386) 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06a00640 1 -- 172.21.15.98:0/559468630 learned_addr learned my addr 172.21.15.98:0/559468630 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06000640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08155eb0 0x7efd080cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd07400640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd080c6fe0 0x7efd080c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06a00640 1 -- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08155eb0 msgr2=0x7efd080cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06a00640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08155eb0 0x7efd080cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06a00640 1 -- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd080c6fe0 msgr2=0x7efd080c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06a00640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd080c6fe0 0x7efd080c7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06a00640 1 -- 172.21.15.98:0/559468630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd081562e0 con 0x7efcec004810 2024-05-27T00:25:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06000640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08155eb0 0x7efd080cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:25:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd06a00640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 0x7efd081f35b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efcf80115e0 tx=0x7efcf8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:08.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efcfea00640 1 -- 172.21.15.98:0/559468630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf800f8f0 con 0x7efcec004810 2024-05-27T00:25:08.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efcfea00640 1 -- 172.21.15.98:0/559468630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf800fa90 con 0x7efcec004810 2024-05-27T00:25:08.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd0da00640 1 -- 172.21.15.98:0/559468630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd080cc920 con 0x7efcec004810 2024-05-27T00:25:08.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efcfea00640 1 -- 172.21.15.98:0/559468630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf800fe40 con 0x7efcec004810 2024-05-27T00:25:08.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.768+0000 7efd0da00640 1 -- 172.21.15.98:0/559468630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd080c5f40 con 0x7efcec004810 2024-05-27T00:25:08.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.772+0000 7efd0da00640 1 -- 172.21.15.98:0/559468630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd08075cb0 con 0x7efcec004810 2024-05-27T00:25:08.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.772+0000 7efcfea00640 1 -- 172.21.15.98:0/559468630 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efcf8015460 con 0x7efcec004810 2024-05-27T00:25:08.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.772+0000 7efcfea00640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efce80678c0 0x7efce8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:08.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.772+0000 7efcfea00640 1 -- 172.21.15.98:0/559468630 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7efcf8086900 con 0x7efcec004810 2024-05-27T00:25:08.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.772+0000 7efd06000640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efce80678c0 0x7efce8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:08.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.772+0000 7efd06000640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efce80678c0 0x7efce8069d80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efd0815c010 tx=0x7efcf40b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:08.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.776+0000 7efcfea00640 1 -- 172.21.15.98:0/559468630 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7efcf8019030 con 0x7efcec004810 2024-05-27T00:25:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 -- 172.21.15.98:0/559468630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd08152150 con 0x7efcec004810 2024-05-27T00:25:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efcfea00640 1 -- 172.21.15.98:0/559468630 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6007 (secure 0 0 0) 0x7efcf8057160 con 0x7efcec004810 2024-05-27T00:25:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:08.970 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:06.572172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:25:08.970 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 44 2024-05-27T00:25:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 -- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efce80678c0 msgr2=0x7efce8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efce80678c0 0x7efce8069d80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efd0815c010 tx=0x7efcf40b1040 comp rx=0 tx=0).stop 2024-05-27T00:25:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 -- 172.21.15.98:0/559468630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 msgr2=0x7efd081f35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 0x7efd081f35b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efcf80115e0 tx=0x7efcf8011aa0 comp rx=0 tx=0).stop 2024-05-27T00:25:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 -- 172.21.15.98:0/559468630 shutdown_connections 2024-05-27T00:25:08.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efce80678c0 0x7efce8069d80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd080c6fe0 0x7efd080c7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd08155eb0 0x7efd080cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 --2- 172.21.15.98:0/559468630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcec004810 0x7efd081f35b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 -- 172.21.15.98:0/559468630 >> 172.21.15.98:0/559468630 conn(0x7efd08072f90 msgr2=0x7efd080c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 -- 172.21.15.98:0/559468630 shutdown_connections 2024-05-27T00:25:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:08.968+0000 7efd0da00640 1 -- 172.21.15.98:0/559468630 wait complete. 2024-05-27T00:25:08.991 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:25:10.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab15a00640 1 Processor -- start 2024-05-27T00:25:11.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab15a00640 1 -- start start 2024-05-27T00:25:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab15a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fab10152a50 0x7fab100709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:11.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab15a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab100cb7e0 0x7fab10070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab15a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab100b3ed0 0x7fab10158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab15a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fab100c8a00 con 0x7fab100b3ed0 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab15a00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fab1015be40 con 0x7fab10152a50 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab15a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fab100c50e0 con 0x7fab100cb7e0 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab0f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab100b3ed0 0x7fab10158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab0f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab100b3ed0 0x7fab10158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51390/0 (socket says 172.21.15.98:51390) 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab0f400640 1 -- 172.21.15.98:0/4080190230 learned_addr learned my addr 172.21.15.98:0/4080190230 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab0ea00640 1 --2- 172.21.15.98:0/4080190230 >> v2:172.21.15.204:3300/0 conn(0x7fab10152a50 0x7fab100709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab06000640 1 --2- 172.21.15.98:0/4080190230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab100cb7e0 0x7fab10070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab0f400640 1 -- 172.21.15.98:0/4080190230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab100cb7e0 msgr2=0x7fab10070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab0f400640 1 --2- 172.21.15.98:0/4080190230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab100cb7e0 0x7fab10070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab0f400640 1 -- 172.21.15.98:0/4080190230 >> v2:172.21.15.204:3300/0 conn(0x7fab10152a50 msgr2=0x7fab100709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab0f400640 1 --2- 172.21.15.98:0/4080190230 >> v2:172.21.15.204:3300/0 conn(0x7fab10152a50 0x7fab100709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.136+0000 7fab0f400640 1 -- 172.21.15.98:0/4080190230 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab100c8c50 con 0x7fab100b3ed0 2024-05-27T00:25:11.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab0f400640 1 --2- 172.21.15.98:0/4080190230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 0x7fab10158750 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fab100715f0 tx=0x7fab0801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6d642f65e3f727 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab0e000640 1 -- 172.21.15.98:0/4080190230 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab080b0070 con 0x7fab100b3ed0 2024-05-27T00:25:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab0e000640 1 -- 172.21.15.98:0/4080190230 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab080ab600 con 0x7fab100b3ed0 2024-05-27T00:25:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab0e000640 1 -- 172.21.15.98:0/4080190230 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab080ab9d0 con 0x7fab100b3ed0 2024-05-27T00:25:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab15a00640 1 -- 172.21.15.98:0/4080190230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 msgr2=0x7fab10158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab15a00640 1 --2- 172.21.15.98:0/4080190230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 0x7fab10158750 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fab100715f0 tx=0x7fab0801c460 comp rx=0 tx=0).stop 2024-05-27T00:25:11.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab15a00640 1 -- 172.21.15.98:0/4080190230 shutdown_connections 2024-05-27T00:25:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab15a00640 1 --2- 172.21.15.98:0/4080190230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 0x7fab10158750 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab15a00640 1 --2- 172.21.15.98:0/4080190230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab100cb7e0 0x7fab10070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab15a00640 1 --2- 172.21.15.98:0/4080190230 >> v2:172.21.15.204:3300/0 conn(0x7fab10152a50 0x7fab100709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab15a00640 1 -- 172.21.15.98:0/4080190230 >> 172.21.15.98:0/4080190230 conn(0x7fab10072f90 msgr2=0x7fab100733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab15a00640 1 -- 172.21.15.98:0/4080190230 shutdown_connections 2024-05-27T00:25:11.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab15a00640 1 -- 172.21.15.98:0/4080190230 wait complete. 2024-05-27T00:25:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.140+0000 7fab15a00640 1 Processor -- start 2024-05-27T00:25:11.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab15a00640 1 -- start start 2024-05-27T00:25:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab15a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 0x7fab100c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab15a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab100cb7e0 0x7fab101445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab15a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab10152a50 0x7fab10143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab15a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab10152e30 con 0x7fab100b3ed0 2024-05-27T00:25:11.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab15a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fab100cb570 con 0x7fab10152a50 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab15a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fab100c9c10 con 0x7fab100cb7e0 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab0ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 0x7fab100c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab0ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 0x7fab100c6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51398/0 (socket says 172.21.15.98:51398) 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab0ea00640 1 -- 172.21.15.98:0/2816068651 learned_addr learned my addr 172.21.15.98:0/2816068651 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab06000640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab100cb7e0 0x7fab101445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab0f400640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab10152a50 0x7fab10143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab0ea00640 1 -- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab100cb7e0 msgr2=0x7fab101445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab0ea00640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab100cb7e0 0x7fab101445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab0ea00640 1 -- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab10152a50 msgr2=0x7fab10143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab0ea00640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab10152a50 0x7fab10143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab0ea00640 1 -- 172.21.15.98:0/2816068651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab10147000 con 0x7fab100b3ed0 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab0ea00640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 0x7fab100c6bb0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fab0000d800 tx=0x7fab0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab07400640 1 -- 172.21.15.98:0/2816068651 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0000b6c0 con 0x7fab100b3ed0 2024-05-27T00:25:11.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab07400640 1 -- 172.21.15.98:0/2816068651 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab00019070 con 0x7fab100b3ed0 2024-05-27T00:25:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab07400640 1 -- 172.21.15.98:0/2816068651 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0000ba60 con 0x7fab100b3ed0 2024-05-27T00:25:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab15a00640 1 -- 172.21.15.98:0/2816068651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1014b5b0 con 0x7fab100b3ed0 2024-05-27T00:25:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab15a00640 1 -- 172.21.15.98:0/2816068651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1014bb70 con 0x7fab100b3ed0 2024-05-27T00:25:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.144+0000 7fab15a00640 1 -- 172.21.15.98:0/2816068651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab100b3cb0 con 0x7fab100b3ed0 2024-05-27T00:25:11.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.148+0000 7fab07400640 1 -- 172.21.15.98:0/2816068651 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fab0000bc00 con 0x7fab100b3ed0 2024-05-27T00:25:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.148+0000 7fab07400640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faaec0678c0 0x7faaec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.148+0000 7fab07400640 1 -- 172.21.15.98:0/2816068651 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fab00086400 con 0x7fab100b3ed0 2024-05-27T00:25:11.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.148+0000 7fab06000640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faaec0678c0 0x7faaec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:11.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.148+0000 7fab06000640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faaec0678c0 0x7faaec069d80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fab100714e0 tx=0x7faaf8003a40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:11.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.152+0000 7fab07400640 1 -- 172.21.15.98:0/2816068651 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fab00056c60 con 0x7fab100b3ed0 2024-05-27T00:25:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.340+0000 7fab15a00640 1 -- 172.21.15.98:0/2816068651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab10075c90 con 0x7fab100b3ed0 2024-05-27T00:25:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab07400640 1 -- 172.21.15.98:0/2816068651 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6007 (secure 0 0 0) 0x7fab00056560 con 0x7fab100b3ed0 2024-05-27T00:25:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:11.346 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:06.572172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:25:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 44 2024-05-27T00:25:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 -- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faaec0678c0 msgr2=0x7faaec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faaec0678c0 0x7faaec069d80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fab100714e0 tx=0x7faaf8003a40 comp rx=0 tx=0).stop 2024-05-27T00:25:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 -- 172.21.15.98:0/2816068651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 msgr2=0x7fab100c6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 0x7fab100c6bb0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fab0000d800 tx=0x7fab0000dcc0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 -- 172.21.15.98:0/2816068651 shutdown_connections 2024-05-27T00:25:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faaec0678c0 0x7faaec069d80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab10152a50 0x7fab10143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab100cb7e0 0x7fab101445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 --2- 172.21.15.98:0/2816068651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab100b3ed0 0x7fab100c6bb0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 -- 172.21.15.98:0/2816068651 >> 172.21.15.98:0/2816068651 conn(0x7fab10072f90 msgr2=0x7fab10073370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 -- 172.21.15.98:0/2816068651 shutdown_connections 2024-05-27T00:25:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:11.344+0000 7fab15a00640 1 -- 172.21.15.98:0/2816068651 wait complete. 2024-05-27T00:25:11.367 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:25:13.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:13.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.504+0000 7f0ebfc00640 1 Processor -- start 2024-05-27T00:25:13.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.504+0000 7f0ebfc00640 1 -- start start 2024-05-27T00:25:13.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.504+0000 7f0ebfc00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0eb80731f0 0x7f0eb80c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:13.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.504+0000 7f0ebfc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80c79f0 0x7f0eb8157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:13.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.504+0000 7f0ebfc00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb80c17d0 con 0x7f0eb80735d0 2024-05-27T00:25:13.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.504+0000 7f0ebfc00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f0eb8071990 con 0x7f0eb80731f0 2024-05-27T00:25:13.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.504+0000 7f0ebfc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb8070e10 con 0x7f0eb80c79f0 2024-05-27T00:25:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.508+0000 7f0ebdc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80c79f0 0x7f0eb8157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.508+0000 7f0ebd200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f0eb80731f0 0x7f0eb80c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.508+0000 7f0ebdc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80c79f0 0x7f0eb8157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:34892/0 (socket says 172.21.15.98:34892) 2024-05-27T00:25:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.508+0000 7f0ebdc00640 1 -- 172.21.15.98:0/1640822431 learned_addr learned my addr 172.21.15.98:0/1640822431 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.508+0000 7f0eb7e00640 1 --1- 172.21.15.98:0/1640822431 >> v1:172.21.15.204:6789/0 conn(0x7f0eb80735d0 0x7f0eb8151430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:35780/0 (socket says 172.21.15.98:35780) 2024-05-27T00:25:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.508+0000 7f0ebd200640 1 -- 172.21.15.98:0/1640822431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80c79f0 msgr2=0x7f0eb8157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.508+0000 7f0ebd200640 1 --2- 172.21.15.98:0/1640822431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80c79f0 0x7f0eb8157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.508+0000 7f0ebd200640 1 -- 172.21.15.98:0/1640822431 >> v1:172.21.15.204:6789/0 conn(0x7f0eb80735d0 legacy=0x7f0eb8151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:13.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.508+0000 7f0ebd200640 1 -- 172.21.15.98:0/1640822431 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb80710f0 con 0x7f0eb80731f0 2024-05-27T00:25:13.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.508+0000 7f0ebd200640 1 --2- 172.21.15.98:0/1640822431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eb80731f0 0x7f0eb80c0eb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009d90 tx=0x7f0ea801f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31e6506f47c0de0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:13.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0eb7400640 1 -- 172.21.15.98:0/1640822431 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea80b4070 con 0x7f0eb80731f0 2024-05-27T00:25:13.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0eb7400640 1 -- 172.21.15.98:0/1640822431 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ea80af600 con 0x7f0eb80731f0 2024-05-27T00:25:13.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0eb7400640 1 -- 172.21.15.98:0/1640822431 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea80af9d0 con 0x7f0eb80731f0 2024-05-27T00:25:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 -- 172.21.15.98:0/1640822431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eb80731f0 msgr2=0x7f0eb80c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 --2- 172.21.15.98:0/1640822431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eb80731f0 0x7f0eb80c0eb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009d90 tx=0x7f0ea801f150 comp rx=0 tx=0).stop 2024-05-27T00:25:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 -- 172.21.15.98:0/1640822431 shutdown_connections 2024-05-27T00:25:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 --2- 172.21.15.98:0/1640822431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80c79f0 0x7f0eb8157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 --2- 172.21.15.98:0/1640822431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eb80731f0 0x7f0eb80c0eb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:13.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 -- 172.21.15.98:0/1640822431 >> 172.21.15.98:0/1640822431 conn(0x7f0eb8067530 msgr2=0x7f0eb806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:13.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 -- 172.21.15.98:0/1640822431 shutdown_connections 2024-05-27T00:25:13.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 -- 172.21.15.98:0/1640822431 wait complete. 2024-05-27T00:25:13.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 Processor -- start 2024-05-27T00:25:13.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 -- start start 2024-05-27T00:25:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eb80731f0 0x7f0eb8149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80735d0 0x7f0eb8151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb80c79f0 0x7f0eb81587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb80c8400 con 0x7f0eb80c79f0 2024-05-27T00:25:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb80716b0 con 0x7f0eb80731f0 2024-05-27T00:25:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.512+0000 7f0ebfc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb80c13f0 con 0x7f0eb80735d0 2024-05-27T00:25:13.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebdc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb80c79f0 0x7f0eb81587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebdc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb80c79f0 0x7f0eb81587c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38818/0 (socket says 172.21.15.98:38818) 2024-05-27T00:25:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebdc00640 1 -- 172.21.15.98:0/2805714975 learned_addr learned my addr 172.21.15.98:0/2805714975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebd200640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eb80731f0 0x7f0eb8149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0eb7e00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80735d0 0x7f0eb8151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebdc00640 1 -- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80735d0 msgr2=0x7f0eb8151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebdc00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80735d0 0x7f0eb8151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:13.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebdc00640 1 -- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eb80731f0 msgr2=0x7f0eb8149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:13.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebdc00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eb80731f0 0x7f0eb8149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:13.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebdc00640 1 -- 172.21.15.98:0/2805714975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb814cd70 con 0x7f0eb80c79f0 2024-05-27T00:25:13.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebdc00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb80c79f0 0x7f0eb81587c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000e980 tx=0x7f0eb000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:13.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0eb4c00640 1 -- 172.21.15.98:0/2805714975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb000de00 con 0x7f0eb80c79f0 2024-05-27T00:25:13.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0eb4c00640 1 -- 172.21.15.98:0/2805714975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eb0010440 con 0x7f0eb80c79f0 2024-05-27T00:25:13.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebfc00640 1 -- 172.21.15.98:0/2805714975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb814a5d0 con 0x7f0eb80c79f0 2024-05-27T00:25:13.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0eb4c00640 1 -- 172.21.15.98:0/2805714975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0010830 con 0x7f0eb80c79f0 2024-05-27T00:25:13.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebfc00640 1 -- 172.21.15.98:0/2805714975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eb815af70 con 0x7f0eb80c79f0 2024-05-27T00:25:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0ebfc00640 1 -- 172.21.15.98:0/2805714975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eb8075cb0 con 0x7f0eb80c79f0 2024-05-27T00:25:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0eb4c00640 1 -- 172.21.15.98:0/2805714975 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0eb00109d0 con 0x7f0eb80c79f0 2024-05-27T00:25:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0eb4c00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0e980678f0 0x7f0e98069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:13.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.516+0000 7f0eb4c00640 1 -- 172.21.15.98:0/2805714975 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7f0eb0086720 con 0x7f0eb80c79f0 2024-05-27T00:25:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.520+0000 7f0ebd200640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0e980678f0 0x7f0e98069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:13.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.520+0000 7f0ebd200640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0e980678f0 0x7f0e98069db0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80b1000 tx=0x7f0ea80aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:13.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.524+0000 7f0eb4c00640 1 -- 172.21.15.98:0/2805714975 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f0eb814cd70 con 0x7f0eb80c79f0 2024-05-27T00:25:13.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 -- 172.21.15.98:0/2805714975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0eb8158510 con 0x7f0eb80c79f0 2024-05-27T00:25:13.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0eb4c00640 1 -- 172.21.15.98:0/2805714975 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6013 (secure 0 0 0) 0x7f0eb0014030 con 0x7f0eb80c79f0 2024-05-27T00:25:13.718 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:13.719 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:12.740737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:25:13.719 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 45 2024-05-27T00:25:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 -- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0e980678f0 msgr2=0x7f0e98069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0e980678f0 0x7f0e98069db0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80b1000 tx=0x7f0ea80aa000 comp rx=0 tx=0).stop 2024-05-27T00:25:13.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 -- 172.21.15.98:0/2805714975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb80c79f0 msgr2=0x7f0eb81587c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb80c79f0 0x7f0eb81587c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000e980 tx=0x7f0eb000ee40 comp rx=0 tx=0).stop 2024-05-27T00:25:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 -- 172.21.15.98:0/2805714975 shutdown_connections 2024-05-27T00:25:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb80c79f0 0x7f0eb81587c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eb80735d0 0x7f0eb8151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0e980678f0 0x7f0e98069db0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 --2- 172.21.15.98:0/2805714975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eb80731f0 0x7f0eb8149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 -- 172.21.15.98:0/2805714975 >> 172.21.15.98:0/2805714975 conn(0x7f0eb8067530 msgr2=0x7f0eb80c7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 -- 172.21.15.98:0/2805714975 shutdown_connections 2024-05-27T00:25:13.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:13.716+0000 7f0ebfc00640 1 -- 172.21.15.98:0/2805714975 wait complete. 2024-05-27T00:25:13.740 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:25:15.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:15.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.884+0000 7f106d600640 1 Processor -- start 2024-05-27T00:25:15.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.884+0000 7f106d600640 1 -- start start 2024-05-27T00:25:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f106d600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f10680731f0 0x7f10680c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f106d600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f10680c79f0 0x7f1068157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f106d600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10680c17d0 con 0x7f10680735d0 2024-05-27T00:25:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f106d600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1068071990 con 0x7f10680c79f0 2024-05-27T00:25:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f106d600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f1068070e10 con 0x7f10680731f0 2024-05-27T00:25:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f105e000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f10680735d0 0x7f1068151430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33788/0 (socket says 172.21.15.98:33788) 2024-05-27T00:25:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f105e000640 1 -- 172.21.15.98:0/1868828310 learned_addr learned my addr 172.21.15.98:0/1868828310 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:25:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f1067400640 1 --2- 172.21.15.98:0/1868828310 >> v2:172.21.15.98:3300/0 conn(0x7f10680c79f0 0x7f1068157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f1066a00640 1 --2- 172.21.15.98:0/1868828310 >> v2:172.21.15.204:3300/0 conn(0x7f10680731f0 0x7f10680c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f1067400640 1 -- 172.21.15.98:0/1868828310 >> v2:172.21.15.204:3300/0 conn(0x7f10680731f0 msgr2=0x7f10680c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f1067400640 1 --2- 172.21.15.98:0/1868828310 >> v2:172.21.15.204:3300/0 conn(0x7f10680731f0 0x7f10680c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f1067400640 1 -- 172.21.15.98:0/1868828310 >> v1:172.21.15.98:6789/0 conn(0x7f10680735d0 legacy=0x7f1068151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f1066000640 1 -- 172.21.15.98:0/1868828310 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827972450 0 0) 0x7f10680c17d0 con 0x7f10680735d0 2024-05-27T00:25:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.888+0000 7f1067400640 1 -- 172.21.15.98:0/1868828310 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10680710f0 con 0x7f10680c79f0 2024-05-27T00:25:15.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f1067400640 1 --2- 172.21.15.98:0/1868828310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680c79f0 0x7f1068157970 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f10680cca90 tx=0x7f1060020870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d54d39b42ced1b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:15.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f1066000640 1 -- 172.21.15.98:0/1868828310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10600b6070 con 0x7f10680c79f0 2024-05-27T00:25:15.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f1066000640 1 -- 172.21.15.98:0/1868828310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10600a9730 con 0x7f10680c79f0 2024-05-27T00:25:15.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f1066000640 1 -- 172.21.15.98:0/1868828310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10600a9b00 con 0x7f10680c79f0 2024-05-27T00:25:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f106d600640 1 -- 172.21.15.98:0/1868828310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680c79f0 msgr2=0x7f1068157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f106d600640 1 --2- 172.21.15.98:0/1868828310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680c79f0 0x7f1068157970 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f10680cca90 tx=0x7f1060020870 comp rx=0 tx=0).stop 2024-05-27T00:25:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f106d600640 1 -- 172.21.15.98:0/1868828310 shutdown_connections 2024-05-27T00:25:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f106d600640 1 --2- 172.21.15.98:0/1868828310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680c79f0 0x7f1068157970 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f106d600640 1 --2- 172.21.15.98:0/1868828310 >> v2:172.21.15.204:3300/0 conn(0x7f10680731f0 0x7f10680c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f106d600640 1 -- 172.21.15.98:0/1868828310 >> 172.21.15.98:0/1868828310 conn(0x7f1068067530 msgr2=0x7f106806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:15.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f106d600640 1 -- 172.21.15.98:0/1868828310 shutdown_connections 2024-05-27T00:25:15.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f106d600640 1 -- 172.21.15.98:0/1868828310 wait complete. 2024-05-27T00:25:15.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f106d600640 1 Processor -- start 2024-05-27T00:25:15.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.892+0000 7f106d600640 1 -- start start 2024-05-27T00:25:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f106d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10680731f0 0x7f1068146d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f106d600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680735d0 0x7f106813e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f106d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10680c79f0 0x7f106813e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f106d600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10680c13f0 con 0x7f10680735d0 2024-05-27T00:25:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f106d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f10680c8400 con 0x7f10680731f0 2024-05-27T00:25:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f106d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f10680c1570 con 0x7f10680c79f0 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105e000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680735d0 0x7f106813e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105e000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680735d0 0x7f106813e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38824/0 (socket says 172.21.15.98:38824) 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105e000640 1 -- 172.21.15.98:0/1264181975 learned_addr learned my addr 172.21.15.98:0/1264181975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f1066a00640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10680731f0 0x7f1068146d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f1067400640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10680c79f0 0x7f106813e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105e000640 1 -- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10680c79f0 msgr2=0x7f106813e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105e000640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10680c79f0 0x7f106813e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105e000640 1 -- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10680731f0 msgr2=0x7f1068146d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105e000640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10680731f0 0x7f1068146d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105e000640 1 -- 172.21.15.98:0/1264181975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1068146650 con 0x7f10680735d0 2024-05-27T00:25:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f1067400640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10680c79f0 0x7f106813e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:25:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105e000640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680735d0 0x7f106813e1d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f105000c4b0 tx=0x7f105000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105f400640 1 -- 172.21.15.98:0/1264181975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1050018070 con 0x7f10680735d0 2024-05-27T00:25:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105f400640 1 -- 172.21.15.98:0/1264181975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1050007500 con 0x7f10680735d0 2024-05-27T00:25:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f106d600640 1 -- 172.21.15.98:0/1264181975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1068159fb0 con 0x7f10680735d0 2024-05-27T00:25:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105f400640 1 -- 172.21.15.98:0/1264181975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1050004a30 con 0x7f10680735d0 2024-05-27T00:25:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f106d600640 1 -- 172.21.15.98:0/1264181975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10681587f0 con 0x7f10680735d0 2024-05-27T00:25:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f106d600640 1 -- 172.21.15.98:0/1264181975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10681589f0 con 0x7f10680735d0 2024-05-27T00:25:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.896+0000 7f105f400640 1 -- 172.21.15.98:0/1264181975 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f105001b070 con 0x7f10680735d0 2024-05-27T00:25:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.900+0000 7f105f400640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10440678c0 0x7f1044069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.900+0000 7f105f400640 1 -- 172.21.15.98:0/1264181975 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7f1050085240 con 0x7f10680735d0 2024-05-27T00:25:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.900+0000 7f1066a00640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10440678c0 0x7f1044069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.900+0000 7f1066a00640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10440678c0 0x7f1044069d80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1058003b90 tx=0x7f10580098d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:15.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:15.904+0000 7f105f400640 1 -- 172.21.15.98:0/1264181975 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f10681589f0 con 0x7f10680735d0 2024-05-27T00:25:16.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.092+0000 7f106d600640 1 -- 172.21.15.98:0/1264181975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1068075cb0 con 0x7f10680735d0 2024-05-27T00:25:16.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.092+0000 7f105f400640 1 -- 172.21.15.98:0/1264181975 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6013 (secure 0 0 0) 0x7f1050055510 con 0x7f10680735d0 2024-05-27T00:25:16.096 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:12.740737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:25:16.096 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 45 2024-05-27T00:25:16.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 -- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10440678c0 msgr2=0x7f1044069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10440678c0 0x7f1044069d80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1058003b90 tx=0x7f10580098d0 comp rx=0 tx=0).stop 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 -- 172.21.15.98:0/1264181975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680735d0 msgr2=0x7f106813e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680735d0 0x7f106813e1d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f105000c4b0 tx=0x7f105000c970 comp rx=0 tx=0).stop 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 -- 172.21.15.98:0/1264181975 shutdown_connections 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10680c79f0 0x7f106813e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10680735d0 0x7f106813e1d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f10440678c0 0x7f1044069d80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 --2- 172.21.15.98:0/1264181975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10680731f0 0x7f1068146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 -- 172.21.15.98:0/1264181975 >> 172.21.15.98:0/1264181975 conn(0x7f1068067530 msgr2=0x7f10680c71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 -- 172.21.15.98:0/1264181975 shutdown_connections 2024-05-27T00:25:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:16.096+0000 7f106d600640 1 -- 172.21.15.98:0/1264181975 wait complete. 2024-05-27T00:25:16.117 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 45, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5667": { "addr": "172.21.15.98:6834/1223646158", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 1223646158, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 1223646158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5667, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:clientreplay", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:25:12.740737+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5667 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:25:18.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:18.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9c00640 1 Processor -- start 2024-05-27T00:25:18.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9c00640 1 -- start start 2024-05-27T00:25:18.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe4a40c3a60 0x7fe4a415b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:18.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4a4158290 0x7fe4a40c8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:18.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4a40c90b0 con 0x7fe4a4151950 2024-05-27T00:25:18.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe4a415bad0 con 0x7fe4a4158290 2024-05-27T00:25:18.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fe4a40c8e60 con 0x7fe4a40c3a60 2024-05-27T00:25:18.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4a4158290 0x7fe4a40c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:18.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4a4158290 0x7fe4a40c8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38828/0 (socket says 172.21.15.98:38828) 2024-05-27T00:25:18.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9200640 1 -- 172.21.15.98:0/1947145504 learned_addr learned my addr 172.21.15.98:0/1947145504 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:18.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a3400640 1 --2- 172.21.15.98:0/1947145504 >> v2:172.21.15.204:3300/0 conn(0x7fe4a40c3a60 0x7fe4a415b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:18.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9200640 1 -- 172.21.15.98:0/1947145504 >> v2:172.21.15.204:3300/0 conn(0x7fe4a40c3a60 msgr2=0x7fe4a415b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:18.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9200640 1 --2- 172.21.15.98:0/1947145504 >> v2:172.21.15.204:3300/0 conn(0x7fe4a40c3a60 0x7fe4a415b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:18.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9200640 1 -- 172.21.15.98:0/1947145504 >> v1:172.21.15.204:6789/0 conn(0x7fe4a4151950 legacy=0x7fe4a40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:18.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.264+0000 7fe4a9200640 1 -- 172.21.15.98:0/1947145504 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a40c9350 con 0x7fe4a4158290 2024-05-27T00:25:18.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a9200640 1 --2- 172.21.15.98:0/1947145504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4158290 0x7fe4a40c8070 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe4a406c090 tx=0x7fe49c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13f2b746f82a4b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:18.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a2a00640 1 -- 172.21.15.98:0/1947145504 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49c0a57a0 con 0x7fe4a4158290 2024-05-27T00:25:18.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a2a00640 1 -- 172.21.15.98:0/1947145504 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe49c0a5940 con 0x7fe4a4158290 2024-05-27T00:25:18.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a2a00640 1 -- 172.21.15.98:0/1947145504 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49c0a5d10 con 0x7fe4a4158290 2024-05-27T00:25:18.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a9c00640 1 -- 172.21.15.98:0/1947145504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4158290 msgr2=0x7fe4a40c8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:18.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a9c00640 1 --2- 172.21.15.98:0/1947145504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4158290 0x7fe4a40c8070 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe4a406c090 tx=0x7fe49c01c9f0 comp rx=0 tx=0).stop 2024-05-27T00:25:18.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a9c00640 1 -- 172.21.15.98:0/1947145504 shutdown_connections 2024-05-27T00:25:18.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a9c00640 1 --2- 172.21.15.98:0/1947145504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4158290 0x7fe4a40c8070 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:18.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a9c00640 1 --2- 172.21.15.98:0/1947145504 >> v2:172.21.15.204:3300/0 conn(0x7fe4a40c3a60 0x7fe4a415b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:18.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a9c00640 1 -- 172.21.15.98:0/1947145504 >> 172.21.15.98:0/1947145504 conn(0x7fe4a406dd00 msgr2=0x7fe4a406b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:18.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a9c00640 1 -- 172.21.15.98:0/1947145504 shutdown_connections 2024-05-27T00:25:18.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a9c00640 1 -- 172.21.15.98:0/1947145504 wait complete. 2024-05-27T00:25:18.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.268+0000 7fe4a9c00640 1 Processor -- start 2024-05-27T00:25:18.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9c00640 1 -- start start 2024-05-27T00:25:18.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4a40c3a60 0x7fe4a4144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4151950 0x7fe4a40730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4a4158290 0x7fe4a41f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a40c6f60 con 0x7fe4a4151950 2024-05-27T00:25:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a40c9e00 con 0x7fe4a40c3a60 2024-05-27T00:25:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe4a40c70e0 con 0x7fe4a4158290 2024-05-27T00:25:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4151950 0x7fe4a40730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:18.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4151950 0x7fe4a40730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38832/0 (socket says 172.21.15.98:38832) 2024-05-27T00:25:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a3400640 1 -- 172.21.15.98:0/2177757439 learned_addr learned my addr 172.21.15.98:0/2177757439 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9200640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4a4158290 0x7fe4a41f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a3e00640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4a40c3a60 0x7fe4a4144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a3400640 1 -- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4a4158290 msgr2=0x7fe4a41f2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a3400640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4a4158290 0x7fe4a41f2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a3400640 1 -- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4a40c3a60 msgr2=0x7fe4a4144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a3400640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4a40c3a60 0x7fe4a4144820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:18.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a3400640 1 -- 172.21.15.98:0/2177757439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a40c8c30 con 0x7fe4a4151950 2024-05-27T00:25:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a3400640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4151950 0x7fe4a40730b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe49000d7d0 tx=0x7fe49000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe49be00640 1 -- 172.21.15.98:0/2177757439 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490004020 con 0x7fe4a4151950 2024-05-27T00:25:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe49be00640 1 -- 172.21.15.98:0/2177757439 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4900041c0 con 0x7fe4a4151950 2024-05-27T00:25:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe49be00640 1 -- 172.21.15.98:0/2177757439 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4900043b0 con 0x7fe4a4151950 2024-05-27T00:25:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9c00640 1 -- 172.21.15.98:0/2177757439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a40cd2e0 con 0x7fe4a4151950 2024-05-27T00:25:18.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9c00640 1 -- 172.21.15.98:0/2177757439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a40ca600 con 0x7fe4a4151950 2024-05-27T00:25:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.272+0000 7fe4a9c00640 1 -- 172.21.15.98:0/2177757439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a4075c40 con 0x7fe4a4151950 2024-05-27T00:25:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.276+0000 7fe49be00640 1 -- 172.21.15.98:0/2177757439 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe49001b020 con 0x7fe4a4151950 2024-05-27T00:25:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.276+0000 7fe49be00640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4840678c0 0x7fe484069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.276+0000 7fe49be00640 1 -- 172.21.15.98:0/2177757439 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fe490085ce0 con 0x7fe4a4151950 2024-05-27T00:25:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.276+0000 7fe4a3e00640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4840678c0 0x7fe484069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:18.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.276+0000 7fe4a3e00640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4840678c0 0x7fe484069d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe494001650 tx=0x7fe494009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:18.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.280+0000 7fe49be00640 1 -- 172.21.15.98:0/2177757439 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fe490056550 con 0x7fe4a4151950 2024-05-27T00:25:18.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.468+0000 7fe4a9c00640 1 -- 172.21.15.98:0/2177757439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4a40c9b50 con 0x7fe4a4151950 2024-05-27T00:25:18.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.468+0000 7fe49be00640 1 -- 172.21.15.98:0/2177757439 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6013 (secure 0 0 0) 0x7fe490055e50 con 0x7fe4a4151950 2024-05-27T00:25:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:18.472 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:12.740737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:25:18.472 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 45 2024-05-27T00:25:18.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 -- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4840678c0 msgr2=0x7fe484069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:18.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4840678c0 0x7fe484069d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe494001650 tx=0x7fe494009040 comp rx=0 tx=0).stop 2024-05-27T00:25:18.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 -- 172.21.15.98:0/2177757439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4151950 msgr2=0x7fe4a40730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4151950 0x7fe4a40730b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe49000d7d0 tx=0x7fe49000dc90 comp rx=0 tx=0).stop 2024-05-27T00:25:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 -- 172.21.15.98:0/2177757439 shutdown_connections 2024-05-27T00:25:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4a4158290 0x7fe4a41f2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4a4151950 0x7fe4a40730b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4840678c0 0x7fe484069d80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 --2- 172.21.15.98:0/2177757439 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4a40c3a60 0x7fe4a4144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 -- 172.21.15.98:0/2177757439 >> 172.21.15.98:0/2177757439 conn(0x7fe4a406dd00 msgr2=0x7fe4a40c5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 -- 172.21.15.98:0/2177757439 shutdown_connections 2024-05-27T00:25:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:18.472+0000 7fe4a9c00640 1 -- 172.21.15.98:0/2177757439 wait complete. 2024-05-27T00:25:18.494 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:25:18.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:25:18.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:25:20.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa564e00640 1 Processor -- start 2024-05-27T00:25:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa564e00640 1 -- start start 2024-05-27T00:25:20.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa564e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa5600c6a60 0x7fa560073340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:20.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa564e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5600c6e40 0x7fa560158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:20.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa564e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa560151f00 0x7fa560152350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:20.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa564e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa560152890 con 0x7fa560151f00 2024-05-27T00:25:20.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa564e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa560073880 con 0x7fa5600c6a60 2024-05-27T00:25:20.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa564e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa560159300 con 0x7fa5600c6e40 2024-05-27T00:25:20.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa560151f00 0x7fa560152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:20.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa560151f00 0x7fa560152350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38846/0 (socket says 172.21.15.98:38846) 2024-05-27T00:25:20.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55fe00640 1 -- 172.21.15.98:0/831986329 learned_addr learned my addr 172.21.15.98:0/831986329 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:20.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55f400640 1 --2- 172.21.15.98:0/831986329 >> v2:172.21.15.204:3300/0 conn(0x7fa5600c6a60 0x7fa560073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:20.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55ea00640 1 --2- 172.21.15.98:0/831986329 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5600c6e40 0x7fa560158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:20.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55fe00640 1 -- 172.21.15.98:0/831986329 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5600c6e40 msgr2=0x7fa560158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:20.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55fe00640 1 --2- 172.21.15.98:0/831986329 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5600c6e40 0x7fa560158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55fe00640 1 -- 172.21.15.98:0/831986329 >> v2:172.21.15.204:3300/0 conn(0x7fa5600c6a60 msgr2=0x7fa560073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:20.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55fe00640 1 --2- 172.21.15.98:0/831986329 >> v2:172.21.15.204:3300/0 conn(0x7fa5600c6a60 0x7fa560073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55fe00640 1 -- 172.21.15.98:0/831986329 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56015bb10 con 0x7fa560151f00 2024-05-27T00:25:20.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.640+0000 7fa55f400640 1 --2- 172.21.15.98:0/831986329 >> v2:172.21.15.204:3300/0 conn(0x7fa5600c6a60 0x7fa560073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:25:20.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa55fe00640 1 --2- 172.21.15.98:0/831986329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 0x7fa560152350 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa56006f8e0 tx=0x7fa55801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3c83850acccbbd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:20.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa55e000640 1 -- 172.21.15.98:0/831986329 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5580b0070 con 0x7fa560151f00 2024-05-27T00:25:20.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa55e000640 1 -- 172.21.15.98:0/831986329 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5580ab600 con 0x7fa560151f00 2024-05-27T00:25:20.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa55e000640 1 -- 172.21.15.98:0/831986329 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5580ab9d0 con 0x7fa560151f00 2024-05-27T00:25:20.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa564e00640 1 -- 172.21.15.98:0/831986329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 msgr2=0x7fa560152350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:20.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa564e00640 1 --2- 172.21.15.98:0/831986329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 0x7fa560152350 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa56006f8e0 tx=0x7fa55801c520 comp rx=0 tx=0).stop 2024-05-27T00:25:20.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa564e00640 1 -- 172.21.15.98:0/831986329 shutdown_connections 2024-05-27T00:25:20.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa564e00640 1 --2- 172.21.15.98:0/831986329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 0x7fa560152350 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa564e00640 1 --2- 172.21.15.98:0/831986329 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5600c6e40 0x7fa560158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa564e00640 1 --2- 172.21.15.98:0/831986329 >> v2:172.21.15.204:3300/0 conn(0x7fa5600c6a60 0x7fa560073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa564e00640 1 -- 172.21.15.98:0/831986329 >> 172.21.15.98:0/831986329 conn(0x7fa5600cbe00 msgr2=0x7fa5600cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:20.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa564e00640 1 -- 172.21.15.98:0/831986329 shutdown_connections 2024-05-27T00:25:20.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.644+0000 7fa564e00640 1 -- 172.21.15.98:0/831986329 wait complete. 2024-05-27T00:25:20.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 Processor -- start 2024-05-27T00:25:20.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 -- start start 2024-05-27T00:25:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5600c6a60 0x7fa5601480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5600c6e40 0x7fa560159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 0x7fa56015a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5600c76f0 con 0x7fa560151f00 2024-05-27T00:25:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa5600c7570 con 0x7fa5600c6e40 2024-05-27T00:25:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa5600c7870 con 0x7fa5600c6a60 2024-05-27T00:25:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 0x7fa56015a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 0x7fa56015a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38856/0 (socket says 172.21.15.98:38856) 2024-05-27T00:25:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55fe00640 1 -- 172.21.15.98:0/2975252738 learned_addr learned my addr 172.21.15.98:0/2975252738 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55ea00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5600c6e40 0x7fa560159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55f400640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5600c6a60 0x7fa5601480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55fe00640 1 -- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5600c6a60 msgr2=0x7fa5601480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55fe00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5600c6a60 0x7fa5601480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55fe00640 1 -- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5600c6e40 msgr2=0x7fa560159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55fe00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5600c6e40 0x7fa560159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55fe00640 1 -- 172.21.15.98:0/2975252738 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56015b490 con 0x7fa560151f00 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55ea00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5600c6e40 0x7fa560159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa55fe00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 0x7fa56015a760 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa55801cdc0 tx=0x7fa5580192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa557400640 1 -- 172.21.15.98:0/2975252738 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5580b9070 con 0x7fa560151f00 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa557400640 1 -- 172.21.15.98:0/2975252738 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5580abb90 con 0x7fa560151f00 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 -- 172.21.15.98:0/2975252738 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5601475e0 con 0x7fa560151f00 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa557400640 1 -- 172.21.15.98:0/2975252738 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5580b0040 con 0x7fa560151f00 2024-05-27T00:25:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 -- 172.21.15.98:0/2975252738 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa560147ba0 con 0x7fa560151f00 2024-05-27T00:25:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.648+0000 7fa564e00640 1 -- 172.21.15.98:0/2975252738 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa560075c40 con 0x7fa560151f00 2024-05-27T00:25:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.652+0000 7fa557400640 1 -- 172.21.15.98:0/2975252738 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa5580bc070 con 0x7fa560151f00 2024-05-27T00:25:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.652+0000 7fa557400640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa5400678f0 0x7fa540069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.652+0000 7fa557400640 1 -- 172.21.15.98:0/2975252738 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fa55811eae0 con 0x7fa560151f00 2024-05-27T00:25:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.652+0000 7fa55f400640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa5400678f0 0x7fa540069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.652+0000 7fa55f400640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa5400678f0 0x7fa540069db0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa550009a00 tx=0x7fa5500039a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:20.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.656+0000 7fa557400640 1 -- 172.21.15.98:0/2975252738 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa558016110 con 0x7fa560151f00 2024-05-27T00:25:20.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.844+0000 7fa564e00640 1 -- 172.21.15.98:0/2975252738 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa560159a80 con 0x7fa560151f00 2024-05-27T00:25:20.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa557400640 1 -- 172.21.15.98:0/2975252738 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6014 (secure 0 0 0) 0x7fa5580eec60 con 0x7fa560151f00 2024-05-27T00:25:20.850 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:20.850 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:20.599986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":19,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":1}]} 2024-05-27T00:25:20.850 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 46 2024-05-27T00:25:20.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 -- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa5400678f0 msgr2=0x7fa540069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa5400678f0 0x7fa540069db0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa550009a00 tx=0x7fa5500039a0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 -- 172.21.15.98:0/2975252738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 msgr2=0x7fa56015a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 0x7fa56015a760 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa55801cdc0 tx=0x7fa5580192c0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 -- 172.21.15.98:0/2975252738 shutdown_connections 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa560151f00 0x7fa56015a760 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa5400678f0 0x7fa540069db0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5600c6e40 0x7fa560159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 --2- 172.21.15.98:0/2975252738 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5600c6a60 0x7fa5601480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 -- 172.21.15.98:0/2975252738 >> 172.21.15.98:0/2975252738 conn(0x7fa5600cbe00 msgr2=0x7fa5601615f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 -- 172.21.15.98:0/2975252738 shutdown_connections 2024-05-27T00:25:20.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:20.848+0000 7fa564e00640 1 -- 172.21.15.98:0/2975252738 wait complete. 2024-05-27T00:25:20.870 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5502 2024-05-27T00:25:20.870 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.c 2024-05-27T00:25:21.871 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-05-27T00:25:21.872 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-27T00:25:21.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-27T00:25:21.875 INFO:tasks.ceph.mds.c:Started 2024-05-27T00:25:21.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:21.964 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2024-05-27T00:25:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f247fc00640 1 Processor -- start 2024-05-27T00:25:22.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f247fc00640 1 -- start start 2024-05-27T00:25:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f247fc00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f24780731f0 0x7f24780c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f247fc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24780c79f0 0x7f2478157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f247fc00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24780c17d0 con 0x7f24780735d0 2024-05-27T00:25:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f247fc00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f2478071990 con 0x7f24780731f0 2024-05-27T00:25:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f247fc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2478070e10 con 0x7f24780c79f0 2024-05-27T00:25:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f2477e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f24780735d0 0x7f2478151430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33798/0 (socket says 172.21.15.98:33798) 2024-05-27T00:25:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f2477e00640 1 -- 172.21.15.98:0/882562123 learned_addr learned my addr 172.21.15.98:0/882562123 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:25:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f247d200640 1 --2- 172.21.15.98:0/882562123 >> v2:172.21.15.204:3300/0 conn(0x7f24780731f0 0x7f24780c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:22.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.016+0000 7f247dc00640 1 --2- 172.21.15.98:0/882562123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24780c79f0 0x7f2478157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.020+0000 7f2477400640 1 -- 172.21.15.98:0/882562123 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2987551176 0 0) 0x7f24780c17d0 con 0x7f24780735d0 2024-05-27T00:25:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.020+0000 7f2477400640 1 -- 172.21.15.98:0/882562123 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f245c003530 con 0x7f24780735d0 2024-05-27T00:25:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.020+0000 7f247d200640 1 -- 172.21.15.98:0/882562123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24780c79f0 msgr2=0x7f2478157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.020+0000 7f247d200640 1 --2- 172.21.15.98:0/882562123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24780c79f0 0x7f2478157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.020+0000 7f2477400640 1 -- 172.21.15.98:0/882562123 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1929592789 0 0) 0x7f245c003530 con 0x7f24780735d0 2024-05-27T00:25:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.020+0000 7f247d200640 1 -- 172.21.15.98:0/882562123 >> v1:172.21.15.98:6789/0 conn(0x7f24780735d0 legacy=0x7f2478151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:22.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.020+0000 7f247d200640 1 -- 172.21.15.98:0/882562123 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24780710f0 con 0x7f24780731f0 2024-05-27T00:25:22.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.020+0000 7f247d200640 1 --2- 172.21.15.98:0/882562123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24780731f0 0x7f24780c0eb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2468009d90 tx=0x7f246801f080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ef242c04a3124b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:22.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.020+0000 7f2477400640 1 -- 172.21.15.98:0/882562123 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24680b5070 con 0x7f24780731f0 2024-05-27T00:25:22.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f2477400640 1 -- 172.21.15.98:0/882562123 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24680a8bd0 con 0x7f24780731f0 2024-05-27T00:25:22.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f2477400640 1 -- 172.21.15.98:0/882562123 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2464001680 con 0x7f24780731f0 2024-05-27T00:25:22.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 -- 172.21.15.98:0/882562123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24780731f0 msgr2=0x7f24780c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:22.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 --2- 172.21.15.98:0/882562123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24780731f0 0x7f24780c0eb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2468009d90 tx=0x7f246801f080 comp rx=0 tx=0).stop 2024-05-27T00:25:22.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 -- 172.21.15.98:0/882562123 shutdown_connections 2024-05-27T00:25:22.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 --2- 172.21.15.98:0/882562123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24780c79f0 0x7f2478157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:22.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 --2- 172.21.15.98:0/882562123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24780731f0 0x7f24780c0eb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:22.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 -- 172.21.15.98:0/882562123 >> 172.21.15.98:0/882562123 conn(0x7f2478067530 msgr2=0x7f247806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:22.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 -- 172.21.15.98:0/882562123 shutdown_connections 2024-05-27T00:25:22.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 -- 172.21.15.98:0/882562123 wait complete. 2024-05-27T00:25:22.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 Processor -- start 2024-05-27T00:25:22.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 -- start start 2024-05-27T00:25:22.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24780731f0 0x7f247815ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24780735d0 0x7f2478145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24780c79f0 0x7f24781457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24780c8400 con 0x7f24780c79f0 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f24780716b0 con 0x7f24780735d0 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247fc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f24780c13f0 con 0x7f24780731f0 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247dc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24780c79f0 0x7f24781457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247dc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24780c79f0 0x7f24781457b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38898/0 (socket says 172.21.15.98:38898) 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247dc00640 1 -- 172.21.15.98:0/3330623077 learned_addr learned my addr 172.21.15.98:0/3330623077 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f247d200640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24780731f0 0x7f247815ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.024+0000 7f2477e00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24780735d0 0x7f2478145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f247dc00640 1 -- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24780731f0 msgr2=0x7f247815ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:22.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f247dc00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24780731f0 0x7f247815ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:22.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f247dc00640 1 -- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24780735d0 msgr2=0x7f2478145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:22.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f247dc00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24780735d0 0x7f2478145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:22.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f247dc00640 1 -- 172.21.15.98:0/3330623077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24781470a0 con 0x7f24780c79f0 2024-05-27T00:25:22.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f2477e00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24780735d0 0x7f2478145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:25:22.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f247dc00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24780c79f0 0x7f24781457b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f247000e820 tx=0x7f247000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:22.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f2474c00640 1 -- 172.21.15.98:0/3330623077 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2470017070 con 0x7f24780c79f0 2024-05-27T00:25:22.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f2474c00640 1 -- 172.21.15.98:0/3330623077 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f247000c940 con 0x7f24780c79f0 2024-05-27T00:25:22.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f2474c00640 1 -- 172.21.15.98:0/3330623077 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247000cd10 con 0x7f24780c79f0 2024-05-27T00:25:22.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f247fc00640 1 -- 172.21.15.98:0/3330623077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24781f4430 con 0x7f24780c79f0 2024-05-27T00:25:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f247fc00640 1 -- 172.21.15.98:0/3330623077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f247815a460 con 0x7f24780c79f0 2024-05-27T00:25:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f247fc00640 1 -- 172.21.15.98:0/3330623077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2478075cb0 con 0x7f24780c79f0 2024-05-27T00:25:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f2474c00640 1 -- 172.21.15.98:0/3330623077 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f247001a070 con 0x7f24780c79f0 2024-05-27T00:25:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f2474c00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24580678c0 0x7f2458069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f2474c00640 1 -- 172.21.15.98:0/3330623077 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7f2470085dc0 con 0x7f24780c79f0 2024-05-27T00:25:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.028+0000 7f247d200640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24580678c0 0x7f2458069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.032+0000 7f247d200640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24580678c0 0x7f2458069d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f24680b2000 tx=0x7f24680aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.032+0000 7f2474c00640 1 -- 172.21.15.98:0/3330623077 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f2470055eb0 con 0x7f24780c79f0 2024-05-27T00:25:22.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.224+0000 7f247fc00640 1 -- 172.21.15.98:0/3330623077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2478158510 con 0x7f24780c79f0 2024-05-27T00:25:22.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f2474c00640 1 -- 172.21.15.98:0/3330623077 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6014 (secure 0 0 0) 0x7f247005a1b0 con 0x7f24780c79f0 2024-05-27T00:25:22.230 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:22.230 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:20.599986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":19,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":1}]} 2024-05-27T00:25:22.230 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 46 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 -- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24580678c0 msgr2=0x7f2458069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24580678c0 0x7f2458069d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f24680b2000 tx=0x7f24680aa070 comp rx=0 tx=0).stop 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 -- 172.21.15.98:0/3330623077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24780c79f0 msgr2=0x7f24781457b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24780c79f0 0x7f24781457b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f247000e820 tx=0x7f247000ece0 comp rx=0 tx=0).stop 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 -- 172.21.15.98:0/3330623077 shutdown_connections 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24780c79f0 0x7f24781457b0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f24580678c0 0x7f2458069d80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24780735d0 0x7f2478145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 --2- 172.21.15.98:0/3330623077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24780731f0 0x7f247815ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 -- 172.21.15.98:0/3330623077 >> 172.21.15.98:0/3330623077 conn(0x7f2478067530 msgr2=0x7f247814f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 -- 172.21.15.98:0/3330623077 shutdown_connections 2024-05-27T00:25:22.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:22.228+0000 7f247fc00640 1 -- 172.21.15.98:0/3330623077 wait complete. 2024-05-27T00:25:22.250 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-05-27T00:25:24.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:25:24.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece3200640 1 Processor -- start 2024-05-27T00:25:24.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece3200640 1 -- start start 2024-05-27T00:25:24.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece3200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece415b9b0 0x7fece415bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:24.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece3200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fece406e870 0x7fece415c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:24.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece3200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece40c3c10 con 0x7fece40b3e30 2024-05-27T00:25:24.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece3200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fece40c39c0 con 0x7fece406e870 2024-05-27T00:25:24.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece3200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fece40c3ef0 con 0x7fece415b9b0 2024-05-27T00:25:24.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece2800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fece406e870 0x7fece415c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:24.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece2800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fece406e870 0x7fece415c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45244/0 (socket says 172.21.15.98:45244) 2024-05-27T00:25:24.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece2800640 1 -- 172.21.15.98:0/1358628150 learned_addr learned my addr 172.21.15.98:0/1358628150 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fecd9400640 1 --1- 172.21.15.98:0/1358628150 >> v1:172.21.15.98:6789/0 conn(0x7fece40b3e30 0x7fece406e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33198/0 (socket says 172.21.15.98:33198) 2024-05-27T00:25:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece1e00640 1 --2- 172.21.15.98:0/1358628150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece415b9b0 0x7fece415bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece2800640 1 -- 172.21.15.98:0/1358628150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece415b9b0 msgr2=0x7fece415bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:25:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece2800640 1 --2- 172.21.15.98:0/1358628150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece415b9b0 0x7fece415bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece2800640 1 -- 172.21.15.98:0/1358628150 >> v1:172.21.15.98:6789/0 conn(0x7fece40b3e30 legacy=0x7fece406e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.396+0000 7fece2800640 1 -- 172.21.15.98:0/1358628150 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece406d680 con 0x7fece406e870 2024-05-27T00:25:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece2800640 1 --2- 172.21.15.98:0/1358628150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 0x7fece415c2d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009870 tx=0x7fecdc01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3872a1eafd6acc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece1400640 1 -- 172.21.15.98:0/1358628150 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc0b0070 con 0x7fece406e870 2024-05-27T00:25:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece1400640 1 -- 172.21.15.98:0/1358628150 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecdc0ab600 con 0x7fece406e870 2024-05-27T00:25:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece1400640 1 -- 172.21.15.98:0/1358628150 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc0ab9d0 con 0x7fece406e870 2024-05-27T00:25:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece3200640 1 -- 172.21.15.98:0/1358628150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 msgr2=0x7fece415c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece3200640 1 --2- 172.21.15.98:0/1358628150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 0x7fece415c2d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009870 tx=0x7fecdc01c330 comp rx=0 tx=0).stop 2024-05-27T00:25:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece3200640 1 -- 172.21.15.98:0/1358628150 shutdown_connections 2024-05-27T00:25:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece3200640 1 --2- 172.21.15.98:0/1358628150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 0x7fece415c2d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece3200640 1 --2- 172.21.15.98:0/1358628150 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece415b9b0 0x7fece415bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece3200640 1 -- 172.21.15.98:0/1358628150 >> 172.21.15.98:0/1358628150 conn(0x7fece4070a10 msgr2=0x7fece4070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece3200640 1 -- 172.21.15.98:0/1358628150 shutdown_connections 2024-05-27T00:25:24.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece3200640 1 -- 172.21.15.98:0/1358628150 wait complete. 2024-05-27T00:25:24.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece3200640 1 Processor -- start 2024-05-27T00:25:24.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.400+0000 7fece3200640 1 -- start start 2024-05-27T00:25:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece3200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 0x7fece4161390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece3200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece40b3e30 0x7fece41f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece415b9b0 0x7fece4160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece3200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fece4071530 con 0x7fece406e870 2024-05-27T00:25:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fece4070110 con 0x7fece415b9b0 2024-05-27T00:25:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece3200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fece414f5f0 con 0x7fece40b3e30 2024-05-27T00:25:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece1e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 0x7fece4161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece1e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 0x7fece4161390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45260/0 (socket says 172.21.15.98:45260) 2024-05-27T00:25:24.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece1e00640 1 -- 172.21.15.98:0/3152374391 learned_addr learned my addr 172.21.15.98:0/3152374391 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:25:24.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fecd9400640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece40b3e30 0x7fece41f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:24.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece2800640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece415b9b0 0x7fece4160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:24.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece1e00640 1 -- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece40b3e30 msgr2=0x7fece41f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:24.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece1e00640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece40b3e30 0x7fece41f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:24.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece1e00640 1 -- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece415b9b0 msgr2=0x7fece4160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:24.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece1e00640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece415b9b0 0x7fece4160ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:24.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece1e00640 1 -- 172.21.15.98:0/3152374391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece415f340 con 0x7fece406e870 2024-05-27T00:25:24.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece2800640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece415b9b0 0x7fece4160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:25:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece1e00640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 0x7fece4161390 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fecd400d620 tx=0x7fecd400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fecdaa00640 1 -- 172.21.15.98:0/3152374391 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd400b620 con 0x7fece406e870 2024-05-27T00:25:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fecdaa00640 1 -- 172.21.15.98:0/3152374391 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecd4019070 con 0x7fece406e870 2024-05-27T00:25:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fecdaa00640 1 -- 172.21.15.98:0/3152374391 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd400b9e0 con 0x7fece406e870 2024-05-27T00:25:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece3200640 1 -- 172.21.15.98:0/3152374391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece415f5e0 con 0x7fece406e870 2024-05-27T00:25:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece3200640 1 -- 172.21.15.98:0/3152374391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece41f45e0 con 0x7fece406e870 2024-05-27T00:25:24.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fece3200640 1 -- 172.21.15.98:0/3152374391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece4075c40 con 0x7fece406e870 2024-05-27T00:25:24.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.404+0000 7fecdaa00640 1 -- 172.21.15.98:0/3152374391 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fecd400bb80 con 0x7fece406e870 2024-05-27T00:25:24.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.408+0000 7fecdaa00640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fecc00678f0 0x7fecc0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:25:24.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.408+0000 7fecdaa00640 1 -- 172.21.15.98:0/3152374391 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fecd4086400 con 0x7fece406e870 2024-05-27T00:25:24.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.408+0000 7fecd9400640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fecc00678f0 0x7fecc0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:25:24.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.408+0000 7fecd9400640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fecc00678f0 0x7fecc0069db0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feccc0012f0 tx=0x7feccc005b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:25:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.412+0000 7fecdaa00640 1 -- 172.21.15.98:0/3152374391 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fecd4056bd0 con 0x7fece406e870 2024-05-27T00:25:24.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.604+0000 7fece3200640 1 -- 172.21.15.98:0/3152374391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fece40b3a00 con 0x7fece406e870 2024-05-27T00:25:24.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.604+0000 7fecdaa00640 1 -- 172.21.15.98:0/3152374391 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6839 (secure 0 0 0) 0x7fecd40564d0 con 0x7fece406e870 2024-05-27T00:25:24.607 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:25:24.607 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:20.599986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":19,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":1}]} 2024-05-27T00:25:24.608 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 47 2024-05-27T00:25:24.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 -- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fecc00678f0 msgr2=0x7fecc0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fecc00678f0 0x7fecc0069db0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feccc0012f0 tx=0x7feccc005b80 comp rx=0 tx=0).stop 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 -- 172.21.15.98:0/3152374391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 msgr2=0x7fece4161390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 0x7fece4161390 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fecd400d620 tx=0x7fecd400dae0 comp rx=0 tx=0).stop 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 -- 172.21.15.98:0/3152374391 shutdown_connections 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fecc00678f0 0x7fecc0069db0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece415b9b0 0x7fece4160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece40b3e30 0x7fece41f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 --2- 172.21.15.98:0/3152374391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fece406e870 0x7fece4161390 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 -- 172.21.15.98:0/3152374391 >> 172.21.15.98:0/3152374391 conn(0x7fece4070a10 msgr2=0x7fece406e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 -- 172.21.15.98:0/3152374391 shutdown_connections 2024-05-27T00:25:24.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:25:24.608+0000 7fece3200640 1 -- 172.21.15.98:0/3152374391 wait complete. 2024-05-27T00:25:24.630 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-05-27T00:25:24.630 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 82 secs before thrashing 2024-05-27T00:25:26.394 INFO:tasks.ceph.mds.a.smithi098.stderr:2024-05-27T00:25:26.392+0000 7fc996a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:25:27.477 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 63% (397/622) Updating files: 64% (399/622) Updating files: 65% (405/622) Updating files: 66% (411/622) Updating files: 67% (417/622) Updating files: 68% (423/622) Updating files: 69% (430/622) Updating files: 70% (436/622) Updating files: 71% (442/622) Updating files: 72% (448/622) Updating files: 73% (455/622) Updating files: 73% (458/622) Updating files: 74% (461/622) Updating files: 75% (467/622) Updating files: 76% (473/622) Updating files: 77% (479/622) Updating files: 78% (486/622) Updating files: 78% (488/622) Updating files: 79% (492/622) Updating files: 80% (498/622) Updating files: 81% (504/622) Updating files: 82% (511/622) Updating files: 83% (517/622) Updating files: 84% (523/622) Updating files: 85% (529/622) Updating files: 86% (535/622) Updating files: 87% (542/622) Updating files: 88% (548/622) Updating files: 89% (554/622) Updating files: 90% (560/622) Updating files: 91% (567/622) Updating files: 92% (573/622) Updating files: 93% (579/622) Updating files: 94% (585/622) Updating files: 95% (591/622) Updating files: 96% (598/622) Updating files: 97% (604/622) Updating files: 98% (610/622) Updating files: 99% (616/622) Updating files: 100% (622/622) Updating files: 100% (622/622), done. 2024-05-27T00:25:27.696 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 3fa121bf0a8 v0.25 2024-05-27T00:25:27.699 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.25 2024-05-27T00:25:27.709 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 26 -eq 48 ] 2024-05-27T00:25:27.709 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.26 2024-05-27T00:25:27.709 INFO:tasks.workunit.client.0.smithi098.stdout:v0.26 2024-05-27T00:25:27.710 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.26 2024-05-27T00:25:27.710 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.26 2024-05-27T00:25:28.906 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 9981ff90968 v0.26 2024-05-27T00:25:28.911 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.26 2024-05-27T00:25:29.086 INFO:tasks.workunit.client.0.smithi098.stdout:v0.27 2024-05-27T00:25:29.087 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 27 -eq 48 ] 2024-05-27T00:25:29.087 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.27 2024-05-27T00:25:29.087 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.27 2024-05-27T00:25:29.087 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.27 2024-05-27T00:25:30.234 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 793034c62c8 v0.27 2024-05-27T00:25:30.238 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.27 2024-05-27T00:25:30.408 INFO:tasks.workunit.client.0.smithi098.stdout:v0.28 2024-05-27T00:25:30.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 28 -eq 48 ] 2024-05-27T00:25:30.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.28 2024-05-27T00:25:30.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.28 2024-05-27T00:25:30.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.28 2024-05-27T00:25:38.068 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2024-05-27T00:25:38.239 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 071881d7e55 v0.28 2024-05-27T00:25:38.245 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.28 2024-05-27T00:25:38.261 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 29 -eq 48 ] 2024-05-27T00:25:38.261 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.29 2024-05-27T00:25:38.261 INFO:tasks.workunit.client.0.smithi098.stdout:v0.29 2024-05-27T00:25:38.261 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.29 2024-05-27T00:25:38.262 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.29 2024-05-27T00:25:38.859 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 8e69c39f699 v0.29 2024-05-27T00:25:38.866 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.29 2024-05-27T00:25:39.039 INFO:tasks.workunit.client.0.smithi098.stdout:v0.30 2024-05-27T00:25:39.039 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 30 -eq 48 ] 2024-05-27T00:25:39.039 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.30 2024-05-27T00:25:39.039 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.30 2024-05-27T00:25:39.039 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.30 2024-05-27T00:25:40.029 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 64b1b2c70f0 v0.30 2024-05-27T00:25:40.036 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.30 2024-05-27T00:25:40.150 INFO:tasks.workunit.client.0.smithi098.stdout:v0.31 2024-05-27T00:25:40.151 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 31 -eq 48 ] 2024-05-27T00:25:40.151 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.31 2024-05-27T00:25:40.151 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.31 2024-05-27T00:25:40.151 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.31 2024-05-27T00:25:41.676 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 71% (284/398) Updating files: 72% (287/398) Updating files: 73% (291/398) Updating files: 74% (295/398) Updating files: 75% (299/398) Updating files: 76% (303/398) Updating files: 77% (307/398) Updating files: 78% (311/398) Updating files: 79% (315/398) Updating files: 80% (319/398) Updating files: 81% (323/398) Updating files: 82% (327/398) Updating files: 83% (331/398) Updating files: 84% (335/398) Updating files: 85% (339/398) Updating files: 86% (343/398) Updating files: 87% (347/398) Updating files: 88% (351/398) Updating files: 89% (355/398) Updating files: 90% (359/398) Updating files: 91% (363/398) Updating files: 92% (367/398) Updating files: 93% (371/398) Updating files: 94% (375/398) Updating files: 95% (379/398) Updating files: 96% (383/398) Updating files: 97% (387/398) Updating files: 98% (391/398) Updating files: 99% (395/398) Updating files: 100% (398/398) Updating files: 100% (398/398), done. 2024-05-27T00:25:41.724 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 9019c6ce640 v0.31 2024-05-27T00:25:41.732 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.31 2024-05-27T00:25:41.846 INFO:tasks.workunit.client.0.smithi098.stdout:v0.32 2024-05-27T00:25:41.846 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 32 -eq 48 ] 2024-05-27T00:25:41.847 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.32 2024-05-27T00:25:41.847 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.32 2024-05-27T00:25:41.847 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.32 2024-05-27T00:25:42.722 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at c08d08baa6a v0.32 2024-05-27T00:25:42.731 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.32 2024-05-27T00:25:42.790 INFO:tasks.workunit.client.0.smithi098.stdout:v0.33 2024-05-27T00:25:42.791 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 33 -eq 48 ] 2024-05-27T00:25:42.791 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.33 2024-05-27T00:25:42.791 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.33 2024-05-27T00:25:42.791 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.33 2024-05-27T00:25:50.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:25:50.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:25:56.892 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 15% (24/160) Updating files: 16% (26/160) Updating files: 17% (28/160) Updating files: 18% (29/160) Updating files: 19% (31/160) Updating files: 20% (32/160) Updating files: 21% (34/160) Updating files: 22% (36/160) Updating files: 23% (37/160) Updating files: 24% (39/160) Updating files: 25% (40/160) Updating files: 26% (42/160) Updating files: 27% (44/160) Updating files: 28% (45/160) Updating files: 29% (47/160) Updating files: 30% (48/160) Updating files: 31% (50/160) Updating files: 32% (52/160) Updating files: 33% (53/160) Updating files: 34% (55/160) Updating files: 35% (56/160) Updating files: 36% (58/160) Updating files: 37% (60/160) Updating files: 38% (61/160) Updating files: 39% (63/160) Updating files: 40% (64/160) Updating files: 41% (66/160) Updating files: 42% (68/160) Updating files: 43% (69/160) Updating files: 44% (71/160) Updating files: 45% (72/160) Updating files: 46% (74/160) Updating files: 47% (76/160) Updating files: 48% (77/160) Updating files: 49% (79/160) Updating files: 50% (80/160) Updating files: 51% (82/160) Updating files: 52% (84/160) Updating files: 53% (85/160) Updating files: 54% (87/160) Updating files: 55% (88/160) Updating files: 56% (90/160) Updating files: 57% (92/160) Updating files: 58% (93/160) Updating files: 59% (95/160) Updating files: 60% (96/160) Updating files: 61% (98/160) Updating files: 62% (100/160) Updating files: 63% (101/160) Updating files: 64% (103/160) Updating files: 65% (104/160) Updating files: 66% (106/160) Updating files: 67% (108/160) Updating files: 68% (109/160) Updating files: 69% (111/160) Updating files: 70% (112/160) Updating files: 71% (114/160) Updating files: 72% (116/160) Updating files: 73% (117/160) Updating files: 74% (119/160) Updating files: 75% (120/160) Updating files: 76% (122/160) Updating files: 77% (124/160) Updating files: 78% (125/160) Updating files: 79% (127/160) Updating files: 80% (128/160) Updating files: 81% (130/160) Updating files: 82% (132/160) Updating files: 83% (133/160) Updating files: 84% (135/160) Updating files: 85% (136/160) Updating files: 86% (138/160) Updating files: 87% (140/160) Updating files: 88% (141/160) Updating files: 89% (143/160) Updating files: 90% (144/160) Updating files: 91% (146/160) Updating files: 92% (148/160) Updating files: 93% (149/160) Updating files: 94% (151/160) Updating files: 95% (152/160) Updating files: 96% (154/160) Updating files: 97% (156/160) Updating files: 98% (157/160) Updating files: 99% (159/160) Updating files: 100% (160/160) Updating files: 100% (160/160), done. 2024-05-27T00:25:56.936 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 340b4348bf6 v0.33 2024-05-27T00:25:56.940 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.33 2024-05-27T00:25:57.044 INFO:tasks.workunit.client.0.smithi098.stdout:v0.34 2024-05-27T00:25:57.044 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 34 -eq 48 ] 2024-05-27T00:25:57.044 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.34 2024-05-27T00:25:57.044 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.34 2024-05-27T00:25:57.045 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.34 2024-05-27T00:25:57.908 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 2f039eeeb74 v0.34 2024-05-27T00:25:57.912 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.34 2024-05-27T00:25:58.023 INFO:tasks.workunit.client.0.smithi098.stdout:v0.35 2024-05-27T00:25:58.024 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 35 -eq 48 ] 2024-05-27T00:25:58.024 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.35 2024-05-27T00:25:58.024 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.35 2024-05-27T00:25:58.024 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.35 2024-05-27T00:25:59.397 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 97% (327/334) Updating files: 98% (328/334) Updating files: 99% (331/334) Updating files: 100% (334/334) Updating files: 100% (334/334), done. 2024-05-27T00:25:59.439 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 0afda378124 v0.35 2024-05-27T00:25:59.444 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.35 2024-05-27T00:25:59.537 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 36 -eq 48 ] 2024-05-27T00:25:59.538 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.36 2024-05-27T00:25:59.538 INFO:tasks.workunit.client.0.smithi098.stdout:v0.36 2024-05-27T00:25:59.538 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.36 2024-05-27T00:25:59.538 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.36 2024-05-27T00:26:00.567 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 877cacbbdbb v0.36 2024-05-27T00:26:00.570 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.36 2024-05-27T00:26:00.756 INFO:tasks.workunit.client.0.smithi098.stdout:v0.37 2024-05-27T00:26:00.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 37 -eq 48 ] 2024-05-27T00:26:00.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.37 2024-05-27T00:26:00.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.37 2024-05-27T00:26:00.757 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.37 2024-05-27T00:26:15.950 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 34% (62/181) Updating files: 35% (64/181) Updating files: 36% (66/181) Updating files: 37% (67/181) Updating files: 38% (69/181) Updating files: 39% (71/181) Updating files: 40% (73/181) Updating files: 41% (75/181) Updating files: 42% (77/181) Updating files: 43% (78/181) Updating files: 44% (80/181) Updating files: 45% (82/181) Updating files: 46% (84/181) Updating files: 47% (86/181) Updating files: 48% (87/181) Updating files: 49% (89/181) Updating files: 50% (91/181) Updating files: 51% (93/181) Updating files: 52% (95/181) Updating files: 53% (96/181) Updating files: 54% (98/181) Updating files: 55% (100/181) Updating files: 56% (102/181) Updating files: 57% (104/181) Updating files: 58% (105/181) Updating files: 59% (107/181) Updating files: 60% (109/181) Updating files: 61% (111/181) Updating files: 62% (113/181) Updating files: 63% (115/181) Updating files: 64% (116/181) Updating files: 65% (118/181) Updating files: 66% (120/181) Updating files: 67% (122/181) Updating files: 68% (124/181) Updating files: 69% (125/181) Updating files: 70% (127/181) Updating files: 71% (129/181) Updating files: 72% (131/181) Updating files: 73% (133/181) Updating files: 74% (134/181) Updating files: 75% (136/181) Updating files: 76% (138/181) Updating files: 77% (140/181) Updating files: 78% (142/181) Updating files: 79% (143/181) Updating files: 80% (145/181) Updating files: 81% (147/181) Updating files: 82% (149/181) Updating files: 83% (151/181) Updating files: 84% (153/181) Updating files: 85% (154/181) Updating files: 86% (156/181) Updating files: 87% (158/181) Updating files: 88% (160/181) Updating files: 89% (162/181) Updating files: 90% (163/181) Updating files: 91% (165/181) Updating files: 92% (167/181) Updating files: 93% (169/181) Updating files: 94% (171/181) Updating files: 95% (172/181) Updating files: 96% (174/181) Updating files: 97% (176/181) Updating files: 98% (178/181) Updating files: 99% (180/181) Updating files: 100% (181/181) Updating files: 100% (181/181), done. 2024-05-27T00:26:16.028 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at a6f3bbb744a v0.37 2024-05-27T00:26:16.031 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.37 2024-05-27T00:26:16.053 INFO:tasks.workunit.client.0.smithi098.stdout:v0.38 2024-05-27T00:26:16.053 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 38 -eq 48 ] 2024-05-27T00:26:16.053 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.38 2024-05-27T00:26:16.053 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.38 2024-05-27T00:26:16.053 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.38 2024-05-27T00:26:17.022 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at b600ec2ac7c v0.38 2024-05-27T00:26:17.025 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.38 2024-05-27T00:26:17.259 INFO:tasks.workunit.client.0.smithi098.stdout:v0.39 2024-05-27T00:26:17.259 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 39 -eq 48 ] 2024-05-27T00:26:17.259 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.39 2024-05-27T00:26:17.259 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.39 2024-05-27T00:26:17.259 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.39 2024-05-27T00:26:18.581 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 321ecdaba2c v0.39 2024-05-27T00:26:18.583 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.39 2024-05-27T00:26:18.693 INFO:tasks.workunit.client.0.smithi098.stdout:v0.40 2024-05-27T00:26:18.693 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 40 -eq 48 ] 2024-05-27T00:26:18.693 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.40 2024-05-27T00:26:18.693 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.40 2024-05-27T00:26:18.693 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.40 2024-05-27T00:26:19.759 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 7eea40ea37f v0.40 2024-05-27T00:26:19.763 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.40 2024-05-27T00:26:19.894 INFO:tasks.workunit.client.0.smithi098.stdout:v0.41 2024-05-27T00:26:19.895 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 41 -eq 48 ] 2024-05-27T00:26:19.895 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.41 2024-05-27T00:26:19.895 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.41 2024-05-27T00:26:19.895 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.41 2024-05-27T00:26:20.684 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at c1345f7136a v0.41 2024-05-27T00:26:20.687 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.41 2024-05-27T00:26:20.852 INFO:tasks.workunit.client.0.smithi098.stdout:v0.42 2024-05-27T00:26:20.853 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 42 -eq 48 ] 2024-05-27T00:26:20.853 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.42 2024-05-27T00:26:20.853 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.42 2024-05-27T00:26:20.853 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.42 2024-05-27T00:26:21.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:26:21.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:26:32.289 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 43% (109/250) Updating files: 44% (110/250) Updating files: 45% (113/250) Updating files: 46% (115/250) Updating files: 47% (118/250) Updating files: 48% (120/250) Updating files: 49% (123/250) Updating files: 50% (125/250) Updating files: 51% (128/250) Updating files: 52% (130/250) Updating files: 53% (133/250) Updating files: 54% (135/250) Updating files: 55% (138/250) Updating files: 56% (140/250) Updating files: 57% (143/250) Updating files: 58% (145/250) Updating files: 59% (148/250) Updating files: 60% (150/250) Updating files: 61% (153/250) Updating files: 62% (155/250) Updating files: 63% (158/250) Updating files: 64% (160/250) Updating files: 65% (163/250) Updating files: 66% (165/250) Updating files: 67% (168/250) Updating files: 68% (170/250) Updating files: 69% (173/250) Updating files: 70% (175/250) Updating files: 71% (178/250) Updating files: 72% (180/250) Updating files: 73% (183/250) Updating files: 74% (185/250) Updating files: 75% (188/250) Updating files: 76% (190/250) Updating files: 77% (193/250) Updating files: 78% (195/250) Updating files: 79% (198/250) Updating files: 80% (200/250) Updating files: 81% (203/250) Updating files: 82% (205/250) Updating files: 83% (208/250) Updating files: 84% (210/250) Updating files: 85% (213/250) Updating files: 86% (215/250) Updating files: 87% (218/250) Updating files: 88% (220/250) Updating files: 89% (223/250) Updating files: 90% (225/250) Updating files: 91% (228/250) Updating files: 92% (230/250) Updating files: 93% (233/250) Updating files: 94% (235/250) Updating files: 95% (238/250) Updating files: 96% (240/250) Updating files: 97% (243/250) Updating files: 98% (245/250) Updating files: 99% (248/250) Updating files: 100% (250/250) Updating files: 100% (250/250), done. 2024-05-27T00:26:32.376 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at b205c64c687 v0.42 2024-05-27T00:26:32.380 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.42 2024-05-27T00:26:32.625 INFO:tasks.workunit.client.0.smithi098.stdout:v0.43 2024-05-27T00:26:32.625 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 43 -eq 48 ] 2024-05-27T00:26:32.626 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.43 2024-05-27T00:26:32.626 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.43 2024-05-27T00:26:32.626 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.43 2024-05-27T00:26:33.359 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 9fa8781c014 v0.43 2024-05-27T00:26:33.363 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.43 2024-05-27T00:26:33.495 INFO:tasks.workunit.client.0.smithi098.stdout:v0.44 2024-05-27T00:26:33.496 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 44 -eq 48 ] 2024-05-27T00:26:33.496 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.44 2024-05-27T00:26:33.496 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.44 2024-05-27T00:26:33.496 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.44 2024-05-27T00:26:36.348 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 26% (49/186) Updating files: 27% (51/186) Updating files: 28% (53/186) Updating files: 29% (54/186) Updating files: 30% (56/186) Updating files: 31% (58/186) Updating files: 32% (60/186) Updating files: 33% (62/186) Updating files: 34% (64/186) Updating files: 35% (66/186) Updating files: 36% (67/186) Updating files: 37% (69/186) Updating files: 38% (71/186) Updating files: 39% (73/186) Updating files: 40% (75/186) Updating files: 41% (77/186) Updating files: 42% (79/186) Updating files: 43% (80/186) Updating files: 44% (82/186) Updating files: 45% (84/186) Updating files: 46% (86/186) Updating files: 47% (88/186) Updating files: 48% (90/186) Updating files: 49% (92/186) Updating files: 50% (93/186) Updating files: 51% (95/186) Updating files: 52% (97/186) Updating files: 53% (99/186) Updating files: 54% (101/186) Updating files: 55% (103/186) Updating files: 56% (105/186) Updating files: 57% (107/186) Updating files: 58% (108/186) Updating files: 59% (110/186) Updating files: 60% (112/186) Updating files: 60% (113/186) Updating files: 61% (114/186) Updating files: 62% (116/186) Updating files: 63% (118/186) Updating files: 64% (120/186) Updating files: 65% (121/186) Updating files: 66% (123/186) Updating files: 67% (125/186) Updating files: 68% (127/186) Updating files: 69% (129/186) Updating files: 70% (131/186) Updating files: 71% (133/186) Updating files: 72% (134/186) Updating files: 73% (136/186) Updating files: 74% (138/186) Updating files: 75% (140/186) Updating files: 76% (142/186) Updating files: 77% (144/186) Updating files: 78% (146/186) Updating files: 79% (147/186) Updating files: 80% (149/186) Updating files: 81% (151/186) Updating files: 82% (153/186) Updating files: 83% (155/186) Updating files: 84% (157/186) Updating files: 85% (159/186) Updating files: 86% (160/186) Updating files: 87% (162/186) Updating files: 88% (164/186) Updating files: 89% (166/186) Updating files: 90% (168/186) Updating files: 91% (170/186) Updating files: 92% (172/186) Updating files: 93% (173/186) Updating files: 94% (175/186) Updating files: 95% (177/186) Updating files: 96% (179/186) Updating files: 97% (181/186) Updating files: 98% (183/186) Updating files: 99% (185/186) Updating files: 100% (186/186) Updating files: 100% (186/186), done. 2024-05-27T00:26:36.402 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at cdd5298d276 v0.44 2024-05-27T00:26:36.407 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.44 2024-05-27T00:26:36.534 INFO:tasks.workunit.client.0.smithi098.stdout:v0.45 2024-05-27T00:26:36.536 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 45 -eq 48 ] 2024-05-27T00:26:36.536 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.45 2024-05-27T00:26:36.537 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.45 2024-05-27T00:26:36.537 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.45 2024-05-27T00:26:37.355 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 0aea1cb1df5 v0.45 2024-05-27T00:26:37.358 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.45 2024-05-27T00:26:37.506 INFO:tasks.workunit.client.0.smithi098.stdout:v0.46 2024-05-27T00:26:37.506 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 46 -eq 48 ] 2024-05-27T00:26:37.507 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.46 2024-05-27T00:26:37.507 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.46 2024-05-27T00:26:37.507 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.46 2024-05-27T00:26:46.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:26:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.763+0000 7fbb21400640 1 Processor -- start 2024-05-27T00:26:46.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb21400640 1 -- start start 2024-05-27T00:26:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb21400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb1c157cd0 0x7fbb1c0c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb21400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fbb1c1580b0 0x7fbb1c15e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb21400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb1c0c59d0 con 0x7fbb1c0b3cb0 2024-05-27T00:26:46.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb21400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbb1c0c51e0 con 0x7fbb1c157cd0 2024-05-27T00:26:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb21400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fbb1c0cabb0 con 0x7fbb1c1580b0 2024-05-27T00:26:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb1be00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb1c157cd0 0x7fbb1c0c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb20a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbb1c0b3cb0 0x7fbb1c0c9a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47942/0 (socket says 172.21.15.98:47942) 2024-05-27T00:26:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb1be00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb1c157cd0 0x7fbb1c0c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39480/0 (socket says 172.21.15.98:39480) 2024-05-27T00:26:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb20a00640 1 -- 172.21.15.98:0/1315061526 learned_addr learned my addr 172.21.15.98:0/1315061526 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:26:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb13e00640 1 --2- 172.21.15.98:0/1315061526 >> v2:172.21.15.204:3300/0 conn(0x7fbb1c1580b0 0x7fbb1c15e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb1be00640 1 -- 172.21.15.98:0/1315061526 >> v2:172.21.15.204:3300/0 conn(0x7fbb1c1580b0 msgr2=0x7fbb1c15e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:46.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb1be00640 1 --2- 172.21.15.98:0/1315061526 >> v2:172.21.15.204:3300/0 conn(0x7fbb1c1580b0 0x7fbb1c15e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:46.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb1be00640 1 -- 172.21.15.98:0/1315061526 >> v1:172.21.15.98:6789/0 conn(0x7fbb1c0b3cb0 legacy=0x7fbb1c0c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:46.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.767+0000 7fbb1be00640 1 -- 172.21.15.98:0/1315061526 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c0cae50 con 0x7fbb1c157cd0 2024-05-27T00:26:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb1be00640 1 --2- 172.21.15.98:0/1315061526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 0x7fbb1c0c5490 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0cbb50 tx=0x7fbb0c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469a86e2fbb275a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:46.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb1b400640 1 -- 172.21.15.98:0/1315061526 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c0b0070 con 0x7fbb1c157cd0 2024-05-27T00:26:46.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb1b400640 1 -- 172.21.15.98:0/1315061526 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb0c0ab600 con 0x7fbb1c157cd0 2024-05-27T00:26:46.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb1b400640 1 -- 172.21.15.98:0/1315061526 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c0ab9d0 con 0x7fbb1c157cd0 2024-05-27T00:26:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb21400640 1 -- 172.21.15.98:0/1315061526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 msgr2=0x7fbb1c0c5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb21400640 1 --2- 172.21.15.98:0/1315061526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 0x7fbb1c0c5490 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0cbb50 tx=0x7fbb0c01c3f0 comp rx=0 tx=0).stop 2024-05-27T00:26:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb21400640 1 -- 172.21.15.98:0/1315061526 shutdown_connections 2024-05-27T00:26:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb21400640 1 --2- 172.21.15.98:0/1315061526 >> v2:172.21.15.204:3300/0 conn(0x7fbb1c1580b0 0x7fbb1c15e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb21400640 1 --2- 172.21.15.98:0/1315061526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 0x7fbb1c0c5490 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb21400640 1 -- 172.21.15.98:0/1315061526 >> 172.21.15.98:0/1315061526 conn(0x7fbb1c070b30 msgr2=0x7fbb1c070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb21400640 1 -- 172.21.15.98:0/1315061526 shutdown_connections 2024-05-27T00:26:46.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb21400640 1 -- 172.21.15.98:0/1315061526 wait complete. 2024-05-27T00:26:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb21400640 1 Processor -- start 2024-05-27T00:26:46.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.771+0000 7fbb21400640 1 -- start start 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb21400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb1c0b3cb0 0x7fbb1c14f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb21400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 0x7fbb1c145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb21400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb1c1580b0 0x7fbb1c1448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb21400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c0cb410 con 0x7fbb1c157cd0 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb21400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c15eb80 con 0x7fbb1c1580b0 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb21400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1c15a830 con 0x7fbb1c0b3cb0 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb13e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 0x7fbb1c145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb13e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 0x7fbb1c145640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39496/0 (socket says 172.21.15.98:39496) 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb13e00640 1 -- 172.21.15.98:0/508208193 learned_addr learned my addr 172.21.15.98:0/508208193 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb1be00640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb1c0b3cb0 0x7fbb1c14f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb20a00640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb1c1580b0 0x7fbb1c1448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:46.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb13e00640 1 -- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb1c0b3cb0 msgr2=0x7fbb1c14f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:46.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb13e00640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb1c0b3cb0 0x7fbb1c14f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:46.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb13e00640 1 -- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb1c1580b0 msgr2=0x7fbb1c1448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:46.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb13e00640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb1c1580b0 0x7fbb1c1448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:46.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb13e00640 1 -- 172.21.15.98:0/508208193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c14a330 con 0x7fbb1c157cd0 2024-05-27T00:26:46.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb13e00640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 0x7fbb1c145640 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400d5f0 tx=0x7fbb0400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:46.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb18c00640 1 -- 172.21.15.98:0/508208193 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb040125f0 con 0x7fbb1c157cd0 2024-05-27T00:26:46.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb18c00640 1 -- 172.21.15.98:0/508208193 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb04019070 con 0x7fbb1c157cd0 2024-05-27T00:26:46.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb18c00640 1 -- 172.21.15.98:0/508208193 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb040129b0 con 0x7fbb1c157cd0 2024-05-27T00:26:46.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb21400640 1 -- 172.21.15.98:0/508208193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1c14f1f0 con 0x7fbb1c157cd0 2024-05-27T00:26:46.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb21400640 1 -- 172.21.15.98:0/508208193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb1c149a90 con 0x7fbb1c157cd0 2024-05-27T00:26:46.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb21400640 1 -- 172.21.15.98:0/508208193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb1c075c40 con 0x7fbb1c157cd0 2024-05-27T00:26:46.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.775+0000 7fbb18c00640 1 -- 172.21.15.98:0/508208193 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbb04012b50 con 0x7fbb1c157cd0 2024-05-27T00:26:46.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.779+0000 7fbb18c00640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbaf80678c0 0x7fbaf8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:46.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.779+0000 7fbb18c00640 1 -- 172.21.15.98:0/508208193 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fbb040863c0 con 0x7fbb1c157cd0 2024-05-27T00:26:46.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.779+0000 7fbb1be00640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbaf80678c0 0x7fbaf8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:46.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.779+0000 7fbb1be00640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbaf80678c0 0x7fbaf8069d80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c0ad000 tx=0x7fbb0c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:46.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.783+0000 7fbb18c00640 1 -- 172.21.15.98:0/508208193 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fbb04056c10 con 0x7fbb1c157cd0 2024-05-27T00:26:46.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.971+0000 7fbb21400640 1 -- 172.21.15.98:0/508208193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb1c0ca570 con 0x7fbb1c157cd0 2024-05-27T00:26:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.971+0000 7fbb18c00640 1 -- 172.21.15.98:0/508208193 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6839 (secure 0 0 0) 0x7fbb04056510 con 0x7fbb1c157cd0 2024-05-27T00:26:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:26:47.460 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:20.599986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":19,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":1}]} 2024-05-27T00:26:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 47 2024-05-27T00:26:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 -- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbaf80678c0 msgr2=0x7fbaf8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbaf80678c0 0x7fbaf8069d80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c0ad000 tx=0x7fbb0c008040 comp rx=0 tx=0).stop 2024-05-27T00:26:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 -- 172.21.15.98:0/508208193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 msgr2=0x7fbb1c145640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 0x7fbb1c145640 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400d5f0 tx=0x7fbb0400dab0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 -- 172.21.15.98:0/508208193 shutdown_connections 2024-05-27T00:26:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbaf80678c0 0x7fbaf8069d80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb1c1580b0 0x7fbb1c1448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb1c157cd0 0x7fbb1c145640 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 --2- 172.21.15.98:0/508208193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb1c0b3cb0 0x7fbb1c14f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 -- 172.21.15.98:0/508208193 >> 172.21.15.98:0/508208193 conn(0x7fbb1c070b30 msgr2=0x7fbb1c0c7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 -- 172.21.15.98:0/508208193 shutdown_connections 2024-05-27T00:26:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:46.975+0000 7fbb21400640 1 -- 172.21.15.98:0/508208193 wait complete. 2024-05-27T00:26:47.463 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2024-05-27T00:26:47.463 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-27T00:26:47.463 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:26:47.498 INFO:tasks.ceph.mds.a:Stopped 2024-05-27T00:26:47.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:26:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f0352400640 1 Processor -- start 2024-05-27T00:26:47.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f0352400640 1 -- start start 2024-05-27T00:26:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f0352400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f034c0735d0 0x7f034c151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f0352400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034c0c17d0 con 0x7f034c0c79f0 2024-05-27T00:26:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f0352400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034c070e10 con 0x7f034c0731f0 2024-05-27T00:26:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f0352400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f034c071990 con 0x7f034c0735d0 2024-05-27T00:26:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034be00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f034c0c79f0 0x7f034c157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47948/0 (socket says 172.21.15.98:47948) 2024-05-27T00:26:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034be00640 1 -- 172.21.15.98:0/3201719257 learned_addr learned my addr 172.21.15.98:0/3201719257 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:26:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034aa00640 1 --2- 172.21.15.98:0/3201719257 >> v2:172.21.15.204:3300/0 conn(0x7f034c0735d0 0x7f034c151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1920263268 0 0) 0x7f034c0c17d0 con 0x7f034c0c79f0 2024-05-27T00:26:47.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0330003530 con 0x7f034c0c79f0 2024-05-27T00:26:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 231583956 0 0) 0x7f0330003530 con 0x7f034c0c79f0 2024-05-27T00:26:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 >> v2:172.21.15.204:3300/0 conn(0x7f034c0735d0 msgr2=0x7f034c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 --2- 172.21.15.98:0/3201719257 >> v2:172.21.15.204:3300/0 conn(0x7f034c0735d0 0x7f034c151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 >> v1:172.21.15.204:6789/0 conn(0x7f034c0731f0 legacy=0x7f034c0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c0710f0 con 0x7f034c0c79f0 2024-05-27T00:26:47.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f0344001a00 con 0x7f034c0c79f0 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 >> v1:172.21.15.98:6789/0 conn(0x7f034c0c79f0 legacy=0x7f034c157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03300039c0 0x7f0330003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0330004810 0x7f0330004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03300051d0 0x7f033000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f034c071990 con 0x7f03300039c0 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f033000c0a0 con 0x7f03300051d0 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f033000e4b0 con 0x7f0330004810 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034b400640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03300039c0 0x7f0330003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034aa00640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0330004810 0x7f0330004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034be00640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03300051d0 0x7f033000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034b400640 1 -- 172.21.15.98:0/3201719257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0330004810 msgr2=0x7f0330004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034b400640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0330004810 0x7f0330004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034b400640 1 -- 172.21.15.98:0/3201719257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03300051d0 msgr2=0x7f033000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034b400640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03300051d0 0x7f033000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034b400640 1 -- 172.21.15.98:0/3201719257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0330003530 con 0x7f03300039c0 2024-05-27T00:26:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034be00640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03300051d0 0x7f033000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:26:47.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.675+0000 7f034be00640 1 -- 172.21.15.98:0/3201719257 reap_dead start 2024-05-27T00:26:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f034b400640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03300039c0 0x7f0330003e00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f033c001780 tx=0x7f033c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08471042c2068ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c0013a0 con 0x7f03300039c0 2024-05-27T00:26:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f033c0052f0 con 0x7f03300039c0 2024-05-27T00:26:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f034a000640 1 -- 172.21.15.98:0/3201719257 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c003840 con 0x7f03300039c0 2024-05-27T00:26:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f0352400640 1 -- 172.21.15.98:0/3201719257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03300039c0 msgr2=0x7f0330003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f0352400640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03300039c0 0x7f0330003e00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f033c001780 tx=0x7f033c01c3d0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f0352400640 1 -- 172.21.15.98:0/3201719257 shutdown_connections 2024-05-27T00:26:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f0352400640 1 --2- 172.21.15.98:0/3201719257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03300039c0 0x7f0330003e00 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f0352400640 1 -- 172.21.15.98:0/3201719257 >> 172.21.15.98:0/3201719257 conn(0x7f034c067530 msgr2=0x7f034c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:47.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f0352400640 1 -- 172.21.15.98:0/3201719257 shutdown_connections 2024-05-27T00:26:47.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.679+0000 7f0352400640 1 -- 172.21.15.98:0/3201719257 wait complete. 2024-05-27T00:26:47.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0352400640 1 Processor -- start 2024-05-27T00:26:47.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0352400640 1 -- start start 2024-05-27T00:26:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0352400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f03300039c0 0x7f034c1e3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0352400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c145660 0x7f034c1e8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0352400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f034c145ac0 0x7f034c1e8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0352400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f034c1515b0 con 0x7f034c145660 2024-05-27T00:26:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0352400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f034c151430 con 0x7f034c145ac0 2024-05-27T00:26:47.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0352400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f034c151730 con 0x7f03300039c0 2024-05-27T00:26:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c145660 0x7f034c1e8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c145660 0x7f034c1e8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39510/0 (socket says 172.21.15.98:39510) 2024-05-27T00:26:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034aa00640 1 -- 172.21.15.98:0/1018530317 learned_addr learned my addr 172.21.15.98:0/1018530317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:26:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034b400640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f03300039c0 0x7f034c1e3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034be00640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f034c145ac0 0x7f034c1e8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034aa00640 1 -- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f03300039c0 msgr2=0x7f034c1e3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034aa00640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f03300039c0 0x7f034c1e3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034aa00640 1 -- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f034c145ac0 msgr2=0x7f034c1e8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034aa00640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f034c145ac0 0x7f034c1e8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034aa00640 1 -- 172.21.15.98:0/1018530317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c1f3060 con 0x7f034c145660 2024-05-27T00:26:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f034aa00640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c145660 0x7f034c1e8680 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0338011a10 tx=0x7f0338011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0343400640 1 -- 172.21.15.98:0/1018530317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0338019070 con 0x7f034c145660 2024-05-27T00:26:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0343400640 1 -- 172.21.15.98:0/1018530317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0338004240 con 0x7f034c145660 2024-05-27T00:26:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0352400640 1 -- 172.21.15.98:0/1018530317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034c1f3320 con 0x7f034c145660 2024-05-27T00:26:47.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0343400640 1 -- 172.21.15.98:0/1018530317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033800f8e0 con 0x7f034c145660 2024-05-27T00:26:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.683+0000 7f0352400640 1 -- 172.21.15.98:0/1018530317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034c1f38e0 con 0x7f034c145660 2024-05-27T00:26:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.687+0000 7f0352400640 1 -- 172.21.15.98:0/1018530317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034c075cb0 con 0x7f034c145660 2024-05-27T00:26:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.687+0000 7f0343400640 1 -- 172.21.15.98:0/1018530317 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f033801c070 con 0x7f034c145660 2024-05-27T00:26:47.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.687+0000 7f0343400640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f032c0678c0 0x7f032c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.687+0000 7f0343400640 1 -- 172.21.15.98:0/1018530317 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7f0338086390 con 0x7f034c145660 2024-05-27T00:26:47.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.687+0000 7f034b400640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f032c0678c0 0x7f032c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:47.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.687+0000 7f034b400640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f032c0678c0 0x7f032c069d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f033c0ac000 tx=0x7f033c001130 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:47.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.691+0000 7f0343400640 1 -- 172.21.15.98:0/1018530317 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f0338056510 con 0x7f034c145660 2024-05-27T00:26:47.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.879+0000 7f0352400640 1 -- 172.21.15.98:0/1018530317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f034c158510 con 0x7f034c145660 2024-05-27T00:26:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.879+0000 7f0343400640 1 -- 172.21.15.98:0/1018530317 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6839 (secure 0 0 0) 0x7f033805a810 con 0x7f034c145660 2024-05-27T00:26:47.884 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:26:47.884 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:20.599986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":19,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":1}]} 2024-05-27T00:26:47.884 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 47 2024-05-27T00:26:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 -- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f032c0678c0 msgr2=0x7f032c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f032c0678c0 0x7f032c069d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f033c0ac000 tx=0x7f033c001130 comp rx=0 tx=0).stop 2024-05-27T00:26:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 -- 172.21.15.98:0/1018530317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c145660 msgr2=0x7f034c1e8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c145660 0x7f034c1e8680 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0338011a10 tx=0x7f0338011ed0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 -- 172.21.15.98:0/1018530317 shutdown_connections 2024-05-27T00:26:47.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f032c0678c0 0x7f032c069d80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f034c145ac0 0x7f034c1e8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c145660 0x7f034c1e8680 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 --2- 172.21.15.98:0/1018530317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f03300039c0 0x7f034c1e3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:47.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 -- 172.21.15.98:0/1018530317 >> 172.21.15.98:0/1018530317 conn(0x7f034c067530 msgr2=0x7f034c0c6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:47.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 -- 172.21.15.98:0/1018530317 shutdown_connections 2024-05-27T00:26:47.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:47.883+0000 7f0352400640 1 -- 172.21.15.98:0/1018530317 wait complete. 2024-05-27T00:26:47.905 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:26:49.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:26:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.051+0000 7fd26e000640 1 Processor -- start 2024-05-27T00:26:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.051+0000 7fd26e000640 1 -- start start 2024-05-27T00:26:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.051+0000 7fd26e000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd2680cbe10 0x7fd2681592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.051+0000 7fd26e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2680cc1f0 0x7fd2680c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd26e000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd268073570 0x7fd2681559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd26e000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd268159810 con 0x7fd268073570 2024-05-27T00:26:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd26e000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd2680c6dd0 con 0x7fd2680cbe10 2024-05-27T00:26:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd26e000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd2680c70b0 con 0x7fd2680cc1f0 2024-05-27T00:26:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd267e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd268073570 0x7fd2681559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd267e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd268073570 0x7fd2681559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39516/0 (socket says 172.21.15.98:39516) 2024-05-27T00:26:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd267e00640 1 -- 172.21.15.98:0/2882928154 learned_addr learned my addr 172.21.15.98:0/2882928154 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:26:50.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd267400640 1 --2- 172.21.15.98:0/2882928154 >> v2:172.21.15.204:3300/0 conn(0x7fd2680cbe10 0x7fd2681592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:50.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd266a00640 1 --2- 172.21.15.98:0/2882928154 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2680cc1f0 0x7fd2680c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:50.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd267e00640 1 -- 172.21.15.98:0/2882928154 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2680cc1f0 msgr2=0x7fd2680c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:50.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd267e00640 1 --2- 172.21.15.98:0/2882928154 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2680cc1f0 0x7fd2680c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd267e00640 1 -- 172.21.15.98:0/2882928154 >> v2:172.21.15.204:3300/0 conn(0x7fd2680cbe10 msgr2=0x7fd2681592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:50.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd267e00640 1 --2- 172.21.15.98:0/2882928154 >> v2:172.21.15.204:3300/0 conn(0x7fd2680cbe10 0x7fd2681592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd267e00640 1 -- 172.21.15.98:0/2882928154 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2680c7390 con 0x7fd268073570 2024-05-27T00:26:50.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd267e00640 1 --2- 172.21.15.98:0/2882928154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd268073570 0x7fd2681559e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd2680cc760 tx=0x7fd26001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcc056283bb5630 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:50.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd266000640 1 -- 172.21.15.98:0/2882928154 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2600b5070 con 0x7fd268073570 2024-05-27T00:26:50.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd266000640 1 -- 172.21.15.98:0/2882928154 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2600a89d0 con 0x7fd268073570 2024-05-27T00:26:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.055+0000 7fd266000640 1 -- 172.21.15.98:0/2882928154 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2600a8da0 con 0x7fd268073570 2024-05-27T00:26:50.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 -- 172.21.15.98:0/2882928154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd268073570 msgr2=0x7fd2681559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:50.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 --2- 172.21.15.98:0/2882928154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd268073570 0x7fd2681559e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd2680cc760 tx=0x7fd26001ed10 comp rx=0 tx=0).stop 2024-05-27T00:26:50.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 -- 172.21.15.98:0/2882928154 shutdown_connections 2024-05-27T00:26:50.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 --2- 172.21.15.98:0/2882928154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd268073570 0x7fd2681559e0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 --2- 172.21.15.98:0/2882928154 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2680cc1f0 0x7fd2680c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 --2- 172.21.15.98:0/2882928154 >> v2:172.21.15.204:3300/0 conn(0x7fd2680cbe10 0x7fd2681592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 -- 172.21.15.98:0/2882928154 >> 172.21.15.98:0/2882928154 conn(0x7fd268067530 msgr2=0x7fd26806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 -- 172.21.15.98:0/2882928154 shutdown_connections 2024-05-27T00:26:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 -- 172.21.15.98:0/2882928154 wait complete. 2024-05-27T00:26:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 Processor -- start 2024-05-27T00:26:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 -- start start 2024-05-27T00:26:50.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268073570 0x7fd2681eed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:50.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680cbe10 0x7fd268158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2680cc1f0 0x7fd268151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2680c79d0 con 0x7fd2680cc1f0 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd2680c1330 con 0x7fd2680cbe10 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd26e000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd2680c6990 con 0x7fd268073570 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd267e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2680cc1f0 0x7fd268151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd267e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2680cc1f0 0x7fd268151c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39520/0 (socket says 172.21.15.98:39520) 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd267e00640 1 -- 172.21.15.98:0/289202566 learned_addr learned my addr 172.21.15.98:0/289202566 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd267400640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268073570 0x7fd2681eed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd266a00640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680cbe10 0x7fd268158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd267e00640 1 -- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268073570 msgr2=0x7fd2681eed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd267e00640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268073570 0x7fd2681eed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd267e00640 1 -- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680cbe10 msgr2=0x7fd268158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.059+0000 7fd267e00640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680cbe10 0x7fd268158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd267e00640 1 -- 172.21.15.98:0/289202566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd268158b90 con 0x7fd2680cc1f0 2024-05-27T00:26:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd266a00640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680cbe10 0x7fd268158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:26:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd267e00640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2680cc1f0 0x7fd268151c50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd26001d3c0 tx=0x7fd26001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd25f400640 1 -- 172.21.15.98:0/289202566 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2600b5040 con 0x7fd2680cc1f0 2024-05-27T00:26:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd25f400640 1 -- 172.21.15.98:0/289202566 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2600be070 con 0x7fd2680cc1f0 2024-05-27T00:26:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd25f400640 1 -- 172.21.15.98:0/289202566 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2600b0940 con 0x7fd2680cc1f0 2024-05-27T00:26:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd26e000640 1 -- 172.21.15.98:0/289202566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd268156840 con 0x7fd2680cc1f0 2024-05-27T00:26:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd26e000640 1 -- 172.21.15.98:0/289202566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd268156e00 con 0x7fd2680cc1f0 2024-05-27T00:26:50.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd26e000640 1 -- 172.21.15.98:0/289202566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd268075cb0 con 0x7fd2680cc1f0 2024-05-27T00:26:50.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd25f400640 1 -- 172.21.15.98:0/289202566 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd2600b0ae0 con 0x7fd2680cc1f0 2024-05-27T00:26:50.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd25f400640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2480678c0 0x7fd248069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:50.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd25f400640 1 -- 172.21.15.98:0/289202566 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fd260123480 con 0x7fd2680cc1f0 2024-05-27T00:26:50.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.063+0000 7fd267400640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2480678c0 0x7fd248069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:50.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.067+0000 7fd267400640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2480678c0 0x7fd248069d80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd258003ab0 tx=0x7fd258009830 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:50.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.067+0000 7fd25f400640 1 -- 172.21.15.98:0/289202566 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd260018870 con 0x7fd2680cc1f0 2024-05-27T00:26:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.255+0000 7fd26e000640 1 -- 172.21.15.98:0/289202566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2680c8c80 con 0x7fd2680cc1f0 2024-05-27T00:26:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.255+0000 7fd25f400640 1 -- 172.21.15.98:0/289202566 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6839 (secure 0 0 0) 0x7fd2600f3bc0 con 0x7fd2680cc1f0 2024-05-27T00:26:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:26:50.260 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:20.599986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":19,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":1}]} 2024-05-27T00:26:50.260 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 47 2024-05-27T00:26:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 -- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2480678c0 msgr2=0x7fd248069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2480678c0 0x7fd248069d80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd258003ab0 tx=0x7fd258009830 comp rx=0 tx=0).stop 2024-05-27T00:26:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 -- 172.21.15.98:0/289202566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2680cc1f0 msgr2=0x7fd268151c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2680cc1f0 0x7fd268151c50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd26001d3c0 tx=0x7fd26001bab0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 -- 172.21.15.98:0/289202566 shutdown_connections 2024-05-27T00:26:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2680cc1f0 0x7fd268151c50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2480678c0 0x7fd248069d80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680cbe10 0x7fd268158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 --2- 172.21.15.98:0/289202566 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268073570 0x7fd2681eed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:50.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 -- 172.21.15.98:0/289202566 >> 172.21.15.98:0/289202566 conn(0x7fd268067530 msgr2=0x7fd2681533c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:50.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 -- 172.21.15.98:0/289202566 shutdown_connections 2024-05-27T00:26:50.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:50.259+0000 7fd26e000640 1 -- 172.21.15.98:0/289202566 wait complete. 2024-05-27T00:26:50.282 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:26:51.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:26:51.105 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:26:51.145 INFO:tasks.ceph.mds.e.smithi098.stderr:2024-05-27T00:26:51.139+0000 7f6e64000640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 42340) UID: 0 2024-05-27T00:26:51.145 INFO:tasks.ceph.mds.c.smithi098.stderr:2024-05-27T00:26:51.143+0000 7f03cac00640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 42340) UID: 0 2024-05-27T00:26:52.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:26:52.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.415+0000 7fd80f200640 1 Processor -- start 2024-05-27T00:26:52.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.415+0000 7fd80f200640 1 -- start start 2024-05-27T00:26:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80f200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd808154800 0x7fd8080c8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80f200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd8080c37d0 0x7fd8080c3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80f200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8080c4140 con 0x7fd80815e5b0 2024-05-27T00:26:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80f200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd8080715d0 con 0x7fd808154800 2024-05-27T00:26:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80f200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd80815a8c0 con 0x7fd8080c37d0 2024-05-27T00:26:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80de00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd808154800 0x7fd8080c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80d400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd80815e5b0 0x7fd8080cdeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47958/0 (socket says 172.21.15.98:47958) 2024-05-27T00:26:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80d400640 1 -- 172.21.15.98:0/765060904 learned_addr learned my addr 172.21.15.98:0/765060904 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:26:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80e800640 1 --2- 172.21.15.98:0/765060904 >> v2:172.21.15.204:3300/0 conn(0x7fd8080c37d0 0x7fd8080c3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:52.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80de00640 1 -- 172.21.15.98:0/765060904 >> v2:172.21.15.204:3300/0 conn(0x7fd8080c37d0 msgr2=0x7fd8080c3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:52.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80de00640 1 --2- 172.21.15.98:0/765060904 >> v2:172.21.15.204:3300/0 conn(0x7fd8080c37d0 0x7fd8080c3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:52.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80de00640 1 -- 172.21.15.98:0/765060904 >> v1:172.21.15.98:6789/0 conn(0x7fd80815e5b0 legacy=0x7fd8080cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:52.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.419+0000 7fd80de00640 1 -- 172.21.15.98:0/765060904 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80815aba0 con 0x7fd808154800 2024-05-27T00:26:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80de00640 1 --2- 172.21.15.98:0/765060904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 0x7fd8080c8920 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd8080c31b0 tx=0x7fd7fc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae85bdace2f1fc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80ca00640 1 -- 172.21.15.98:0/765060904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc0b5070 con 0x7fd808154800 2024-05-27T00:26:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80ca00640 1 -- 172.21.15.98:0/765060904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7fc0a8970 con 0x7fd808154800 2024-05-27T00:26:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80ca00640 1 -- 172.21.15.98:0/765060904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc0a8d40 con 0x7fd808154800 2024-05-27T00:26:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 -- 172.21.15.98:0/765060904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 msgr2=0x7fd8080c8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 --2- 172.21.15.98:0/765060904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 0x7fd8080c8920 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd8080c31b0 tx=0x7fd7fc01ec70 comp rx=0 tx=0).stop 2024-05-27T00:26:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 -- 172.21.15.98:0/765060904 shutdown_connections 2024-05-27T00:26:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 --2- 172.21.15.98:0/765060904 >> v2:172.21.15.204:3300/0 conn(0x7fd8080c37d0 0x7fd8080c3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 --2- 172.21.15.98:0/765060904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 0x7fd8080c8920 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:52.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 -- 172.21.15.98:0/765060904 >> 172.21.15.98:0/765060904 conn(0x7fd808072f90 msgr2=0x7fd8080733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:52.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 -- 172.21.15.98:0/765060904 shutdown_connections 2024-05-27T00:26:52.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 -- 172.21.15.98:0/765060904 wait complete. 2024-05-27T00:26:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 Processor -- start 2024-05-27T00:26:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 -- start start 2024-05-27T00:26:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8080c37d0 0x7fd8081f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 0x7fd8081e5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd80815e5b0 0x7fd8081f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd80815b440 con 0x7fd808154800 2024-05-27T00:26:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd80815b160 con 0x7fd80815e5b0 2024-05-27T00:26:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.423+0000 7fd80f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd8080c3290 con 0x7fd8080c37d0 2024-05-27T00:26:52.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 0x7fd8081e5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:52.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 0x7fd8081e5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39536/0 (socket says 172.21.15.98:39536) 2024-05-27T00:26:52.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80d400640 1 -- 172.21.15.98:0/713203914 learned_addr learned my addr 172.21.15.98:0/713203914 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:26:52.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80e800640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd80815e5b0 0x7fd8081f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:52.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80de00640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8080c37d0 0x7fd8081f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80d400640 1 -- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8080c37d0 msgr2=0x7fd8081f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80d400640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8080c37d0 0x7fd8081f3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80d400640 1 -- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd80815e5b0 msgr2=0x7fd8081f10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80d400640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd80815e5b0 0x7fd8081f10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80d400640 1 -- 172.21.15.98:0/713203914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8081e5e60 con 0x7fd808154800 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80e800640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd80815e5b0 0x7fd8081f10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80d400640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 0x7fd8081e5820 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800c420 tx=0x7fd7f800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd802000640 1 -- 172.21.15.98:0/713203914 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8017070 con 0x7fd808154800 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd802000640 1 -- 172.21.15.98:0/713203914 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7f8005cf0 con 0x7fd808154800 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80f200640 1 -- 172.21.15.98:0/713203914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8081e6140 con 0x7fd808154800 2024-05-27T00:26:52.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd802000640 1 -- 172.21.15.98:0/713203914 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80072f0 con 0x7fd808154800 2024-05-27T00:26:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80f200640 1 -- 172.21.15.98:0/713203914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8081f08a0 con 0x7fd808154800 2024-05-27T00:26:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.427+0000 7fd80f200640 1 -- 172.21.15.98:0/713203914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd808075c40 con 0x7fd808154800 2024-05-27T00:26:52.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.431+0000 7fd802000640 1 -- 172.21.15.98:0/713203914 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd7f801a070 con 0x7fd808154800 2024-05-27T00:26:52.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.431+0000 7fd802000640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd7ec0678c0 0x7fd7ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:52.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.431+0000 7fd802000640 1 -- 172.21.15.98:0/713203914 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fd7f8085460 con 0x7fd808154800 2024-05-27T00:26:52.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.431+0000 7fd80de00640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd7ec0678c0 0x7fd7ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:52.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.431+0000 7fd80de00640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd7ec0678c0 0x7fd7ec069d80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0b2000 tx=0x7fd7fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.435+0000 7fd802000640 1 -- 172.21.15.98:0/713203914 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd7f8055ba0 con 0x7fd808154800 2024-05-27T00:26:52.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 -- 172.21.15.98:0/713203914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd808071860 con 0x7fd808154800 2024-05-27T00:26:52.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd802000640 1 -- 172.21.15.98:0/713203914 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6839 (secure 0 0 0) 0x7fd7f80554a0 con 0x7fd808154800 2024-05-27T00:26:52.631 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:26:52.631 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:20.599986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":19,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":1}]} 2024-05-27T00:26:52.631 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 47 2024-05-27T00:26:52.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 -- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd7ec0678c0 msgr2=0x7fd7ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:52.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd7ec0678c0 0x7fd7ec069d80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0b2000 tx=0x7fd7fc008040 comp rx=0 tx=0).stop 2024-05-27T00:26:52.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 -- 172.21.15.98:0/713203914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 msgr2=0x7fd8081e5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:52.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 0x7fd8081e5820 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800c420 tx=0x7fd7f800c8e0 comp rx=0 tx=0).stop 2024-05-27T00:26:52.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 -- 172.21.15.98:0/713203914 shutdown_connections 2024-05-27T00:26:52.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd7ec0678c0 0x7fd7ec069d80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:52.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd80815e5b0 0x7fd8081f10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:52.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd808154800 0x7fd8081e5820 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:52.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 --2- 172.21.15.98:0/713203914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8080c37d0 0x7fd8081f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:52.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 -- 172.21.15.98:0/713203914 >> 172.21.15.98:0/713203914 conn(0x7fd808072f90 msgr2=0x7fd808153c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:52.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 -- 172.21.15.98:0/713203914 shutdown_connections 2024-05-27T00:26:52.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:52.627+0000 7fd80f200640 1 -- 172.21.15.98:0/713203914 wait complete. 2024-05-27T00:26:52.653 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:26:54.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:26:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.787+0000 7ffa6b000640 1 Processor -- start 2024-05-27T00:26:54.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.787+0000 7ffa6b000640 1 -- start start 2024-05-27T00:26:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa6b000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64070fe0 0x7ffa64154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa6b000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6415a950 con 0x7ffa64070c00 2024-05-27T00:26:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa6b000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6415e060 con 0x7ffa640cb570 2024-05-27T00:26:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa6b000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6415e340 con 0x7ffa64070fe0 2024-05-27T00:26:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ffa64070c00 0x7ffa640c8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48098/0 (socket says 172.21.15.98:48098) 2024-05-27T00:26:54.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63e00640 1 -- 172.21.15.98:0/808913889 learned_addr learned my addr 172.21.15.98:0/808913889 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:26:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa5be00640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64070fe0 0x7ffa64154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 807044517 0 0) 0x7ffa6415a950 con 0x7ffa64070c00 2024-05-27T00:26:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa48003530 con 0x7ffa64070c00 2024-05-27T00:26:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4062274697 0 0) 0x7ffa48003530 con 0x7ffa64070c00 2024-05-27T00:26:54.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64070fe0 msgr2=0x7ffa64154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64070fe0 0x7ffa64154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 >> v1:172.21.15.204:6789/0 conn(0x7ffa640cb570 legacy=0x7ffa6415ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6415e620 con 0x7ffa64070c00 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7ffa54001680 con 0x7ffa64070c00 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 >> v1:172.21.15.98:6789/0 conn(0x7ffa64070c00 legacy=0x7ffa640c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa480039f0 0x7ffa48003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64070fe0 0x7ffa48004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa48004d50 0x7ffa480051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6415e340 con 0x7ffa48004d50 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4800bb40 con 0x7ffa480039f0 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffa4800bd40 con 0x7ffa64070fe0 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa69000640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa48004d50 0x7ffa480051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa63e00640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa480039f0 0x7ffa48003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:54.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa5be00640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64070fe0 0x7ffa48004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa69000640 1 -- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64070fe0 msgr2=0x7ffa48004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa69000640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64070fe0 0x7ffa48004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa69000640 1 -- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa480039f0 msgr2=0x7ffa48003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa69000640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa480039f0 0x7ffa48003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:54.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.791+0000 7ffa69000640 1 -- 172.21.15.98:0/808913889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa48003530 con 0x7ffa48004d50 2024-05-27T00:26:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa69000640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa48004d50 0x7ffa480051d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c001e40 tx=0x7ffa5c01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f2ba3d837b1bab server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c001a90 con 0x7ffa48004d50 2024-05-27T00:26:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa5c003b80 con 0x7ffa48004d50 2024-05-27T00:26:54.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa63400640 1 -- 172.21.15.98:0/808913889 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c005080 con 0x7ffa48004d50 2024-05-27T00:26:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa6b000640 1 -- 172.21.15.98:0/808913889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa48004d50 msgr2=0x7ffa480051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:54.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa6b000640 1 --2- 172.21.15.98:0/808913889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa48004d50 0x7ffa480051d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c001e40 tx=0x7ffa5c01c760 comp rx=0 tx=0).stop 2024-05-27T00:26:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa69000640 1 -- 172.21.15.98:0/808913889 reap_dead start 2024-05-27T00:26:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa6b000640 1 -- 172.21.15.98:0/808913889 shutdown_connections 2024-05-27T00:26:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa6b000640 1 -- 172.21.15.98:0/808913889 >> 172.21.15.98:0/808913889 conn(0x7ffa64072f90 msgr2=0x7ffa640733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa6b000640 1 -- 172.21.15.98:0/808913889 shutdown_connections 2024-05-27T00:26:54.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.795+0000 7ffa6b000640 1 -- 172.21.15.98:0/808913889 wait complete. 2024-05-27T00:26:54.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa6b000640 1 Processor -- start 2024-05-27T00:26:54.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa6b000640 1 -- start start 2024-05-27T00:26:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa6b000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa48004d50 0x7ffa641f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa6b000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa641f3df0 0x7ffa64158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa6b000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64159150 0x7ffa64149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa6b000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffa640c9110 con 0x7ffa641f3df0 2024-05-27T00:26:54.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa6b000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6415ebe0 con 0x7ffa48004d50 2024-05-27T00:26:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa6b000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6415b560 con 0x7ffa64159150 2024-05-27T00:26:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa5be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa641f3df0 0x7ffa64158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa5be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa641f3df0 0x7ffa64158c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33184/0 (socket says 172.21.15.98:33184) 2024-05-27T00:26:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa5be00640 1 -- 172.21.15.98:0/3085171842 learned_addr learned my addr 172.21.15.98:0/3085171842 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:26:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa63e00640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa48004d50 0x7ffa641f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:54.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa69000640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64159150 0x7ffa64149860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa5be00640 1 -- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64159150 msgr2=0x7ffa64149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa5be00640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64159150 0x7ffa64149860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa5be00640 1 -- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa48004d50 msgr2=0x7ffa641f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa5be00640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa48004d50 0x7ffa641f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa5be00640 1 -- 172.21.15.98:0/3085171842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa641595e0 con 0x7ffa641f3df0 2024-05-27T00:26:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa5be00640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa641f3df0 0x7ffa64158c10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c011970 tx=0x7ffa4c011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa60c00640 1 -- 172.21.15.98:0/3085171842 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c019070 con 0x7ffa641f3df0 2024-05-27T00:26:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa60c00640 1 -- 172.21.15.98:0/3085171842 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4c004240 con 0x7ffa641f3df0 2024-05-27T00:26:54.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa60c00640 1 -- 172.21.15.98:0/3085171842 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c00f8e0 con 0x7ffa641f3df0 2024-05-27T00:26:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa6b000640 1 -- 172.21.15.98:0/3085171842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa641f42f0 con 0x7ffa641f3df0 2024-05-27T00:26:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.799+0000 7ffa6b000640 1 -- 172.21.15.98:0/3085171842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa640ce310 con 0x7ffa641f3df0 2024-05-27T00:26:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.803+0000 7ffa6b000640 1 -- 172.21.15.98:0/3085171842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa64075cb0 con 0x7ffa641f3df0 2024-05-27T00:26:54.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.803+0000 7ffa60c00640 1 -- 172.21.15.98:0/3085171842 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffa4c01c070 con 0x7ffa641f3df0 2024-05-27T00:26:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.803+0000 7ffa60c00640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffa400678f0 0x7ffa40069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.803+0000 7ffa60c00640 1 -- 172.21.15.98:0/3085171842 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7ffa4c086460 con 0x7ffa641f3df0 2024-05-27T00:26:54.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.803+0000 7ffa63e00640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffa400678f0 0x7ffa40069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:54.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.803+0000 7ffa63e00640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffa400678f0 0x7ffa40069db0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffa54001ec0 tx=0x7ffa54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.807+0000 7ffa60c00640 1 -- 172.21.15.98:0/3085171842 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7ffa4c056c30 con 0x7ffa641f3df0 2024-05-27T00:26:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.999+0000 7ffa6b000640 1 -- 172.21.15.98:0/3085171842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa640c9720 con 0x7ffa641f3df0 2024-05-27T00:26:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:54.999+0000 7ffa60c00640 1 -- 172.21.15.98:0/3085171842 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6839 (secure 0 0 0) 0x7ffa4c056530 con 0x7ffa641f3df0 2024-05-27T00:26:55.004 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:26:55.004 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:20.599986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":19,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":1}]} 2024-05-27T00:26:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 47 2024-05-27T00:26:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 -- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffa400678f0 msgr2=0x7ffa40069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffa400678f0 0x7ffa40069db0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffa54001ec0 tx=0x7ffa54009040 comp rx=0 tx=0).stop 2024-05-27T00:26:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 -- 172.21.15.98:0/3085171842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa641f3df0 msgr2=0x7ffa64158c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa641f3df0 0x7ffa64158c10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c011970 tx=0x7ffa4c011e30 comp rx=0 tx=0).stop 2024-05-27T00:26:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 -- 172.21.15.98:0/3085171842 shutdown_connections 2024-05-27T00:26:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa64159150 0x7ffa64149860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffa641f3df0 0x7ffa64158c10 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffa400678f0 0x7ffa40069db0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 --2- 172.21.15.98:0/3085171842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa48004d50 0x7ffa641f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 -- 172.21.15.98:0/3085171842 >> 172.21.15.98:0/3085171842 conn(0x7ffa64072f90 msgr2=0x7ffa640c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 -- 172.21.15.98:0/3085171842 shutdown_connections 2024-05-27T00:26:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:55.003+0000 7ffa6b000640 1 -- 172.21.15.98:0/3085171842 wait complete. 2024-05-27T00:26:55.026 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:26:57.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:26:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.159+0000 7fd321000640 1 Processor -- start 2024-05-27T00:26:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.159+0000 7fd321000640 1 -- start start 2024-05-27T00:26:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd321000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd31c0b2f20 0x7fd31c06cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd321000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd31c0700c0 0x7fd31c15a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd321000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c15aec0 0x7fd31c0ce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd321000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd31c06bad0 con 0x7fd31c0700c0 2024-05-27T00:26:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd321000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd31c06b510 con 0x7fd31c0b2f20 2024-05-27T00:26:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd321000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd31c06b7f0 con 0x7fd31c15aec0 2024-05-27T00:26:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd319400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd31c0700c0 0x7fd31c15a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd319400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd31c0700c0 0x7fd31c15a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33196/0 (socket says 172.21.15.98:33196) 2024-05-27T00:26:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd319400640 1 -- 172.21.15.98:0/1772619265 learned_addr learned my addr 172.21.15.98:0/1772619265 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:26:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd31a800640 1 --2- 172.21.15.98:0/1772619265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c15aec0 0x7fd31c0ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd319e00640 1 --2- 172.21.15.98:0/1772619265 >> v2:172.21.15.204:3300/0 conn(0x7fd31c0b2f20 0x7fd31c06cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd319400640 1 -- 172.21.15.98:0/1772619265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c15aec0 msgr2=0x7fd31c0ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd319400640 1 --2- 172.21.15.98:0/1772619265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c15aec0 0x7fd31c0ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd319400640 1 -- 172.21.15.98:0/1772619265 >> v2:172.21.15.204:3300/0 conn(0x7fd31c0b2f20 msgr2=0x7fd31c06cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd319400640 1 --2- 172.21.15.98:0/1772619265 >> v2:172.21.15.204:3300/0 conn(0x7fd31c0b2f20 0x7fd31c06cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd319400640 1 -- 172.21.15.98:0/1772619265 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c0c91f0 con 0x7fd31c0700c0 2024-05-27T00:26:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd31a800640 1 --2- 172.21.15.98:0/1772619265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c15aec0 0x7fd31c0ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:26:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.163+0000 7fd319e00640 1 --2- 172.21.15.98:0/1772619265 >> v2:172.21.15.204:3300/0 conn(0x7fd31c0b2f20 0x7fd31c06cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:26:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd319400640 1 --2- 172.21.15.98:0/1772619265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c0700c0 0x7fd31c15a980 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0cbc00 tx=0x7fd30801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46651aeb96019961 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd318a00640 1 -- 172.21.15.98:0/1772619265 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3080ab5f0 con 0x7fd31c0700c0 2024-05-27T00:26:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd318a00640 1 -- 172.21.15.98:0/1772619265 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3080b0070 con 0x7fd31c0700c0 2024-05-27T00:26:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd318a00640 1 -- 172.21.15.98:0/1772619265 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3080ab9b0 con 0x7fd31c0700c0 2024-05-27T00:26:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 -- 172.21.15.98:0/1772619265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c0700c0 msgr2=0x7fd31c15a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 --2- 172.21.15.98:0/1772619265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c0700c0 0x7fd31c15a980 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0cbc00 tx=0x7fd30801c520 comp rx=0 tx=0).stop 2024-05-27T00:26:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 -- 172.21.15.98:0/1772619265 shutdown_connections 2024-05-27T00:26:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 --2- 172.21.15.98:0/1772619265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c15aec0 0x7fd31c0ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 --2- 172.21.15.98:0/1772619265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c0700c0 0x7fd31c15a980 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 --2- 172.21.15.98:0/1772619265 >> v2:172.21.15.204:3300/0 conn(0x7fd31c0b2f20 0x7fd31c06cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 -- 172.21.15.98:0/1772619265 >> 172.21.15.98:0/1772619265 conn(0x7fd31c06d680 msgr2=0x7fd31c06da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 -- 172.21.15.98:0/1772619265 shutdown_connections 2024-05-27T00:26:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 -- 172.21.15.98:0/1772619265 wait complete. 2024-05-27T00:26:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 Processor -- start 2024-05-27T00:26:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.167+0000 7fd321000640 1 -- start start 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd321000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd31c0700c0 0x7fd31c159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd321000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c0b2f20 0x7fd31c1f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd321000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c15aec0 0x7fd31c14abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd321000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c15e700 con 0x7fd31c15aec0 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd321000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c0b3300 con 0x7fd31c0700c0 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd321000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd31c15e880 con 0x7fd31c0b2f20 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd31a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c15aec0 0x7fd31c14abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd31a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c15aec0 0x7fd31c14abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33198/0 (socket says 172.21.15.98:33198) 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd31a800640 1 -- 172.21.15.98:0/2895317651 learned_addr learned my addr 172.21.15.98:0/2895317651 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd319e00640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd31c0700c0 0x7fd31c159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd319400640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c0b2f20 0x7fd31c1f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd31a800640 1 -- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c0b2f20 msgr2=0x7fd31c1f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:57.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd31a800640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c0b2f20 0x7fd31c1f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd31a800640 1 -- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd31c0700c0 msgr2=0x7fd31c159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd31a800640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd31c0700c0 0x7fd31c159840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd31a800640 1 -- 172.21.15.98:0/2895317651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c0ce2a0 con 0x7fd31c15aec0 2024-05-27T00:26:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd319e00640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd31c0700c0 0x7fd31c159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:26:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd31a800640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c15aec0 0x7fd31c14abc0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd314007e20 tx=0x7fd31400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd312000640 1 -- 172.21.15.98:0/2895317651 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31401a070 con 0x7fd31c15aec0 2024-05-27T00:26:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd312000640 1 -- 172.21.15.98:0/2895317651 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd314003ba0 con 0x7fd31c15aec0 2024-05-27T00:26:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd321000640 1 -- 172.21.15.98:0/2895317651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31c152540 con 0x7fd31c15aec0 2024-05-27T00:26:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd312000640 1 -- 172.21.15.98:0/2895317651 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31400de80 con 0x7fd31c15aec0 2024-05-27T00:26:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd321000640 1 -- 172.21.15.98:0/2895317651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31c1580e0 con 0x7fd31c15aec0 2024-05-27T00:26:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd321000640 1 -- 172.21.15.98:0/2895317651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd31c075cb0 con 0x7fd31c15aec0 2024-05-27T00:26:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.171+0000 7fd312000640 1 -- 172.21.15.98:0/2895317651 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd31401d070 con 0x7fd31c15aec0 2024-05-27T00:26:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.175+0000 7fd312000640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2fc0678c0 0x7fd2fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.175+0000 7fd312000640 1 -- 172.21.15.98:0/2895317651 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fd3140860d0 con 0x7fd31c15aec0 2024-05-27T00:26:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.175+0000 7fd319e00640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2fc0678c0 0x7fd2fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.175+0000 7fd319e00640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2fc0678c0 0x7fd2fc069d80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00c940 tx=0x7fd30c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:57.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.179+0000 7fd312000640 1 -- 172.21.15.98:0/2895317651 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd31c152540 con 0x7fd31c15aec0 2024-05-27T00:26:57.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 -- 172.21.15.98:0/2895317651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd31c06fea0 con 0x7fd31c15aec0 2024-05-27T00:26:57.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd312000640 1 -- 172.21.15.98:0/2895317651 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6839 (secure 0 0 0) 0x7fd31c0ce2a0 con 0x7fd31c15aec0 2024-05-27T00:26:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:26:57.374 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:20.599986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":19,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":1}]} 2024-05-27T00:26:57.375 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 47 2024-05-27T00:26:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 -- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2fc0678c0 msgr2=0x7fd2fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:57.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2fc0678c0 0x7fd2fc069d80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00c940 tx=0x7fd30c008040 comp rx=0 tx=0).stop 2024-05-27T00:26:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 -- 172.21.15.98:0/2895317651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c15aec0 msgr2=0x7fd31c14abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c15aec0 0x7fd31c14abc0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fd314007e20 tx=0x7fd31400d4a0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 -- 172.21.15.98:0/2895317651 shutdown_connections 2024-05-27T00:26:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd31c15aec0 0x7fd31c14abc0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd31c0b2f20 0x7fd31c1f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2fc0678c0 0x7fd2fc069d80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 --2- 172.21.15.98:0/2895317651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd31c0700c0 0x7fd31c159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 -- 172.21.15.98:0/2895317651 >> 172.21.15.98:0/2895317651 conn(0x7fd31c06d680 msgr2=0x7fd31c1566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 -- 172.21.15.98:0/2895317651 shutdown_connections 2024-05-27T00:26:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:57.371+0000 7fd321000640 1 -- 172.21.15.98:0/2895317651 wait complete. 2024-05-27T00:26:57.397 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:26:59.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:26:59.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.531+0000 7fc8d7000640 1 Processor -- start 2024-05-27T00:26:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.531+0000 7fc8d7000640 1 -- start start 2024-05-27T00:26:59.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8d7000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc8d0152520 0x7fc8d00c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:59.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8d7000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d006cc70 con 0x7fc8d00c70e0 2024-05-27T00:26:59.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8d7000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d0072fa0 con 0x7fc8d0073760 2024-05-27T00:26:59.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8d7000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fc8d0152900 con 0x7fc8d0152520 2024-05-27T00:26:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8d5000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc8d00c70e0 0x7fc8d006b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48110/0 (socket says 172.21.15.98:48110) 2024-05-27T00:26:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8d5000640 1 -- 172.21.15.98:0/2731311416 learned_addr learned my addr 172.21.15.98:0/2731311416 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:26:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cfe00640 1 --2- 172.21.15.98:0/2731311416 >> v2:172.21.15.204:3300/0 conn(0x7fc8d0152520 0x7fc8d00c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449153893 0 0) 0x7fc8d006cc70 con 0x7fc8d00c70e0 2024-05-27T00:26:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8b4003530 con 0x7fc8d00c70e0 2024-05-27T00:26:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2668387470 0 0) 0x7fc8b4003530 con 0x7fc8d00c70e0 2024-05-27T00:26:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 >> v2:172.21.15.204:3300/0 conn(0x7fc8d0152520 msgr2=0x7fc8d00c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 --2- 172.21.15.98:0/2731311416 >> v2:172.21.15.204:3300/0 conn(0x7fc8d0152520 0x7fc8d00c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 >> v1:172.21.15.204:6789/0 conn(0x7fc8d0073760 legacy=0x7fc8d0071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d006c6b0 con 0x7fc8d00c70e0 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fc8c8001a00 con 0x7fc8d00c70e0 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 >> v1:172.21.15.98:6789/0 conn(0x7fc8d00c70e0 legacy=0x7fc8d006b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8b40039c0 0x7fc8b4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b4004810 0x7fc8b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8b40051d0 0x7fc8b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0152900 con 0x7fc8b4004810 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b400c070 con 0x7fc8b40051d0 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b400c270 con 0x7fc8b40039c0 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8c7e00640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b4004810 0x7fc8b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8d5000640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8b40051d0 0x7fc8b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cfe00640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8b40039c0 0x7fc8b4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8c7e00640 1 -- 172.21.15.98:0/2731311416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8b40039c0 msgr2=0x7fc8b4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8c7e00640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8b40039c0 0x7fc8b4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8c7e00640 1 -- 172.21.15.98:0/2731311416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8b40051d0 msgr2=0x7fc8b400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:26:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8c7e00640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8b40051d0 0x7fc8b400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8c7e00640 1 -- 172.21.15.98:0/2731311416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b4003530 con 0x7fc8b4004810 2024-05-27T00:26:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8d5000640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8b40051d0 0x7fc8b400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:26:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8d5000640 1 -- 172.21.15.98:0/2731311416 reap_dead start 2024-05-27T00:26:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.535+0000 7fc8cfe00640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8b40039c0 0x7fc8b4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:26:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8c7e00640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b4004810 0x7fc8b4004c90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8001780 tx=0x7fc8b801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe983dfb7fe2e53 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80013a0 con 0x7fc8b4004810 2024-05-27T00:26:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b80052f0 con 0x7fc8b4004810 2024-05-27T00:26:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8cf400640 1 -- 172.21.15.98:0/2731311416 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8003840 con 0x7fc8b4004810 2024-05-27T00:26:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8d7000640 1 -- 172.21.15.98:0/2731311416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b4004810 msgr2=0x7fc8b4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8d7000640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b4004810 0x7fc8b4004c90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8001780 tx=0x7fc8b801c400 comp rx=0 tx=0).stop 2024-05-27T00:26:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8d7000640 1 -- 172.21.15.98:0/2731311416 shutdown_connections 2024-05-27T00:26:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8d7000640 1 --2- 172.21.15.98:0/2731311416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b4004810 0x7fc8b4004c90 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8d7000640 1 -- 172.21.15.98:0/2731311416 >> 172.21.15.98:0/2731311416 conn(0x7fc8d00c9b70 msgr2=0x7fc8d00c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8d7000640 1 -- 172.21.15.98:0/2731311416 shutdown_connections 2024-05-27T00:26:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.539+0000 7fc8d7000640 1 -- 172.21.15.98:0/2731311416 wait complete. 2024-05-27T00:26:59.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d7000640 1 Processor -- start 2024-05-27T00:26:59.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d7000640 1 -- start start 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d7000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8b4004810 0x7fc8d01e3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d7000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d00c4c70 0x7fc8d0157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d7000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01e4890 0x7fc8d0147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d7000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d00c78a0 con 0x7fc8d00c4c70 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d7000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0073240 con 0x7fc8d01e4890 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d7000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d00c7a20 con 0x7fc8b4004810 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8c7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d00c4c70 0x7fc8d0157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8c7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d00c4c70 0x7fc8d0157bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33208/0 (socket says 172.21.15.98:33208) 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8c7e00640 1 -- 172.21.15.98:0/2555492429 learned_addr learned my addr 172.21.15.98:0/2555492429 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8cfe00640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8b4004810 0x7fc8d01e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d5000640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01e4890 0x7fc8d0147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8c7e00640 1 -- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8b4004810 msgr2=0x7fc8d01e3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8c7e00640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8b4004810 0x7fc8d01e3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8c7e00640 1 -- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01e4890 msgr2=0x7fc8d0147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8c7e00640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01e4890 0x7fc8d0147230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8c7e00640 1 -- 172.21.15.98:0/2555492429 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d01f0360 con 0x7fc8d00c4c70 2024-05-27T00:26:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8c7e00640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d00c4c70 0x7fc8d0157bc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8005040 tx=0x7fc8b4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8ccc00640 1 -- 172.21.15.98:0/2555492429 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80013a0 con 0x7fc8d00c4c70 2024-05-27T00:26:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8ccc00640 1 -- 172.21.15.98:0/2555492429 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8b80ba070 con 0x7fc8d00c4c70 2024-05-27T00:26:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8ccc00640 1 -- 172.21.15.98:0/2555492429 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8007a40 con 0x7fc8d00c4c70 2024-05-27T00:26:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d7000640 1 -- 172.21.15.98:0/2555492429 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d00cc5c0 con 0x7fc8d00c4c70 2024-05-27T00:26:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.543+0000 7fc8d7000640 1 -- 172.21.15.98:0/2555492429 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d01f05d0 con 0x7fc8d00c4c70 2024-05-27T00:26:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.547+0000 7fc8d7000640 1 -- 172.21.15.98:0/2555492429 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d0075cb0 con 0x7fc8d00c4c70 2024-05-27T00:26:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.547+0000 7fc8ccc00640 1 -- 172.21.15.98:0/2555492429 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc8b80052f0 con 0x7fc8d00c4c70 2024-05-27T00:26:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.547+0000 7fc8ccc00640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc8ac0678c0 0x7fc8ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:26:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.547+0000 7fc8ccc00640 1 -- 172.21.15.98:0/2555492429 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6427+0+0 (secure 0 0 0) 0x7fc8b8120460 con 0x7fc8d00c4c70 2024-05-27T00:26:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.547+0000 7fc8cfe00640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc8ac0678c0 0x7fc8ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:26:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.547+0000 7fc8cfe00640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc8ac0678c0 0x7fc8ac069d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0069d30 tx=0x7fc8c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:26:59.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.555+0000 7fc8ccc00640 1 -- 172.21.15.98:0/2555492429 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc8b8016110 con 0x7fc8d00c4c70 2024-05-27T00:26:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 -- 172.21.15.98:0/2555492429 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8d006b5c0 con 0x7fc8d00c4c70 2024-05-27T00:26:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8ccc00640 1 -- 172.21.15.98:0/2555492429 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6839 (secure 0 0 0) 0x7fc8b80f0c30 con 0x7fc8d00c4c70 2024-05-27T00:26:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:26:59.830 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:25:20.599986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5667},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5667":{"gid":5667,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":19,"addr":"172.21.15.98:6834/1223646158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1223646158},{"type":"v1","addr":"172.21.15.98:6834","nonce":1223646158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5667,"qdb_cluster":[5667]},"id":1}]} 2024-05-27T00:26:59.831 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 47 2024-05-27T00:26:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 -- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc8ac0678c0 msgr2=0x7fc8ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc8ac0678c0 0x7fc8ac069d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0069d30 tx=0x7fc8c0008040 comp rx=0 tx=0).stop 2024-05-27T00:26:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 -- 172.21.15.98:0/2555492429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d00c4c70 msgr2=0x7fc8d0157bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:26:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d00c4c70 0x7fc8d0157bc0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8005040 tx=0x7fc8b4005840 comp rx=0 tx=0).stop 2024-05-27T00:26:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 -- 172.21.15.98:0/2555492429 shutdown_connections 2024-05-27T00:26:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc8ac0678c0 0x7fc8ac069d80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8d01e4890 0x7fc8d0147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8d00c4c70 0x7fc8d0157bc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 --2- 172.21.15.98:0/2555492429 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8b4004810 0x7fc8d01e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:26:59.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 -- 172.21.15.98:0/2555492429 >> 172.21.15.98:0/2555492429 conn(0x7fc8d00c9b70 msgr2=0x7fc8d014f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:26:59.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 -- 172.21.15.98:0/2555492429 shutdown_connections 2024-05-27T00:26:59.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:26:59.827+0000 7fc8d7000640 1 -- 172.21.15.98:0/2555492429 wait complete. 2024-05-27T00:26:59.852 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:27:01.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:01.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f1676200640 1 Processor -- start 2024-05-27T00:27:01.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f1676200640 1 -- start start 2024-05-27T00:27:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f1676200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1670070c00 0x7f16700c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f1676200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f167015a950 con 0x7f1670070fe0 2024-05-27T00:27:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f1676200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f167015e060 con 0x7f1670154e90 2024-05-27T00:27:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f1676200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f167015e340 con 0x7f1670070c00 2024-05-27T00:27:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f166ea00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1670070fe0 0x7f1670154780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48122/0 (socket says 172.21.15.98:48122) 2024-05-27T00:27:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f166ea00640 1 -- 172.21.15.98:0/1662202786 learned_addr learned my addr 172.21.15.98:0/1662202786 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f166f400640 1 --2- 172.21.15.98:0/1662202786 >> v2:172.21.15.204:3300/0 conn(0x7f1670070c00 0x7f16700c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1671266562 0 0) 0x7f167015a950 con 0x7f1670070fe0 2024-05-27T00:27:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.987+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1654003530 con 0x7f1670070fe0 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3514384866 0 0) 0x7f1654003530 con 0x7f1670070fe0 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 >> v2:172.21.15.204:3300/0 conn(0x7f1670070c00 msgr2=0x7f16700c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 --2- 172.21.15.98:0/1662202786 >> v2:172.21.15.204:3300/0 conn(0x7f1670070c00 0x7f16700c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 >> v1:172.21.15.204:6789/0 conn(0x7f1670154e90 legacy=0x7f167015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167015e620 con 0x7f1670070fe0 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f165c001680 con 0x7f1670070fe0 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 >> v1:172.21.15.98:6789/0 conn(0x7f1670070fe0 legacy=0x7f1670154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16540039c0 0x7f1654003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f1654004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16540051d0 0x7f165400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f167015e340 con 0x7f1654004810 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f165400c080 con 0x7f16540039c0 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f165400c280 con 0x7f16540051d0 2024-05-27T00:27:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166ea00640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f1654004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166f400640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16540039c0 0x7f1654003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166fe00640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16540051d0 0x7f165400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166ea00640 1 -- 172.21.15.98:0/1662202786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16540051d0 msgr2=0x7f165400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166ea00640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16540051d0 0x7f165400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166ea00640 1 -- 172.21.15.98:0/1662202786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16540039c0 msgr2=0x7f1654003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166ea00640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16540039c0 0x7f1654003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166ea00640 1 -- 172.21.15.98:0/1662202786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1654003530 con 0x7f1654004810 2024-05-27T00:27:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166fe00640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16540051d0 0x7f165400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:27:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166fe00640 1 -- 172.21.15.98:0/1662202786 reap_dead start 2024-05-27T00:27:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.991+0000 7f166ea00640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f1654004c90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c210 tx=0x7f165c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ab509017c255e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c007d20 con 0x7f1654004810 2024-05-27T00:27:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f165c0a9070 con 0x7f1654004810 2024-05-27T00:27:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f166e000640 1 -- 172.21.15.98:0/1662202786 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c0a5e00 con 0x7f1654004810 2024-05-27T00:27:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 -- 172.21.15.98:0/1662202786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 msgr2=0x7f1654004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f1654004c90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c210 tx=0x7f165c01cc60 comp rx=0 tx=0).stop 2024-05-27T00:27:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 -- 172.21.15.98:0/1662202786 shutdown_connections 2024-05-27T00:27:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 --2- 172.21.15.98:0/1662202786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f1654004c90 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 -- 172.21.15.98:0/1662202786 >> 172.21.15.98:0/1662202786 conn(0x7f1670072f90 msgr2=0x7f16700733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 -- 172.21.15.98:0/1662202786 shutdown_connections 2024-05-27T00:27:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 -- 172.21.15.98:0/1662202786 wait complete. 2024-05-27T00:27:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 Processor -- start 2024-05-27T00:27:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 -- start start 2024-05-27T00:27:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f16700c4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f167014fe60 0x7f1670150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16701f1540 0x7f16701f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f16700cb570 con 0x7f1654004810 2024-05-27T00:27:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f167015ebe0 con 0x7f167014fe60 2024-05-27T00:27:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f1676200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f16700cb6f0 con 0x7f16701f1540 2024-05-27T00:27:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f166f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f16700c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f166f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f16700c4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33222/0 (socket says 172.21.15.98:33222) 2024-05-27T00:27:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f166f400640 1 -- 172.21.15.98:0/3011192774 learned_addr learned my addr 172.21.15.98:0/3011192774 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f166fe00640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16701f1540 0x7f16701f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.995+0000 7f166ea00640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f167014fe60 0x7f1670150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f166f400640 1 -- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16701f1540 msgr2=0x7f16701f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f166f400640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16701f1540 0x7f16701f19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f166f400640 1 -- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f167014fe60 msgr2=0x7f1670150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f166f400640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f167014fe60 0x7f1670150280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f166f400640 1 -- 172.21.15.98:0/3011192774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16701f1f00 con 0x7f1654004810 2024-05-27T00:27:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f166f400640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f16700c4620 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1660011ad0 tx=0x7f1660011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f1667400640 1 -- 172.21.15.98:0/3011192774 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166000fb50 con 0x7f1654004810 2024-05-27T00:27:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f1667400640 1 -- 172.21.15.98:0/3011192774 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f166000fcf0 con 0x7f1654004810 2024-05-27T00:27:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f1667400640 1 -- 172.21.15.98:0/3011192774 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166000fea0 con 0x7f1654004810 2024-05-27T00:27:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f1676200640 1 -- 172.21.15.98:0/3011192774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16700c7ed0 con 0x7f1654004810 2024-05-27T00:27:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f1676200640 1 -- 172.21.15.98:0/3011192774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16700c8170 con 0x7f1654004810 2024-05-27T00:27:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f1676200640 1 -- 172.21.15.98:0/3011192774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1670075cb0 con 0x7f1654004810 2024-05-27T00:27:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f1667400640 1 -- 172.21.15.98:0/3011192774 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1660012040 con 0x7f1654004810 2024-05-27T00:27:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f1667400640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16500678c0 0x7f1650069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:01.999+0000 7f1667400640 1 -- 172.21.15.98:0/3011192774 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7f1660086720 con 0x7f1654004810 2024-05-27T00:27:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.003+0000 7f166ea00640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16500678c0 0x7f1650069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.003+0000 7f166ea00640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16500678c0 0x7f1650069d80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f165c0ab000 tx=0x7f165c0b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.003+0000 7f1667400640 1 -- 172.21.15.98:0/3011192774 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f16701f1f00 con 0x7f1654004810 2024-05-27T00:27:02.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 -- 172.21.15.98:0/3011192774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16700c9720 con 0x7f1654004810 2024-05-27T00:27:02.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1667400640 1 -- 172.21.15.98:0/3011192774 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6006 (secure 0 0 0) 0x7f1660019030 con 0x7f1654004810 2024-05-27T00:27:02.199 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:02.199 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:00.260048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 48 2024-05-27T00:27:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 -- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16500678c0 msgr2=0x7f1650069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16500678c0 0x7f1650069d80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f165c0ab000 tx=0x7f165c0b0040 comp rx=0 tx=0).stop 2024-05-27T00:27:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 -- 172.21.15.98:0/3011192774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 msgr2=0x7f16700c4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f16700c4620 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1660011ad0 tx=0x7f1660011f90 comp rx=0 tx=0).stop 2024-05-27T00:27:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 -- 172.21.15.98:0/3011192774 shutdown_connections 2024-05-27T00:27:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f16701f1540 0x7f16701f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f16500678c0 0x7f1650069d80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f167014fe60 0x7f1670150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 --2- 172.21.15.98:0/3011192774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1654004810 0x7f16700c4620 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 -- 172.21.15.98:0/3011192774 >> 172.21.15.98:0/3011192774 conn(0x7f1670072f90 msgr2=0x7f16700c3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 -- 172.21.15.98:0/3011192774 shutdown_connections 2024-05-27T00:27:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.195+0000 7f1676200640 1 -- 172.21.15.98:0/3011192774 wait complete. 2024-05-27T00:27:02.220 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-05-27T00:27:02.221 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:27:02.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.355+0000 7fdc20a00640 1 Processor -- start 2024-05-27T00:27:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.355+0000 7fdc20a00640 1 -- start start 2024-05-27T00:27:02.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.355+0000 7fdc20a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc1c0cc4f0 0x7fdc1c0c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:02.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.355+0000 7fdc20a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fdc1c0c67c0 0x7fdc1c0cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:02.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.355+0000 7fdc20a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc1c0c70b0 con 0x7fdc1c0629d0 2024-05-27T00:27:02.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.355+0000 7fdc20a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdc1c0c6330 con 0x7fdc1c0cc4f0 2024-05-27T00:27:02.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.355+0000 7fdc20a00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fdc1c0c6e60 con 0x7fdc1c0c67c0 2024-05-27T00:27:02.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.355+0000 7fdc1b400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc1c0cc4f0 0x7fdc1c0c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:02.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.355+0000 7fdc1be00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdc1c0629d0 0x7fdc1c15bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48132/0 (socket says 172.21.15.98:48132) 2024-05-27T00:27:02.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.355+0000 7fdc1be00640 1 -- 172.21.15.98:0/2192995757 learned_addr learned my addr 172.21.15.98:0/2192995757 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:02.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.359+0000 7fdc13e00640 1 --2- 172.21.15.98:0/2192995757 >> v2:172.21.15.204:3300/0 conn(0x7fdc1c0c67c0 0x7fdc1c0cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:02.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.359+0000 7fdc1b400640 1 -- 172.21.15.98:0/2192995757 >> v2:172.21.15.204:3300/0 conn(0x7fdc1c0c67c0 msgr2=0x7fdc1c0cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:02.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.359+0000 7fdc1b400640 1 --2- 172.21.15.98:0/2192995757 >> v2:172.21.15.204:3300/0 conn(0x7fdc1c0c67c0 0x7fdc1c0cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.359+0000 7fdc1b400640 1 -- 172.21.15.98:0/2192995757 >> v1:172.21.15.98:6789/0 conn(0x7fdc1c0629d0 legacy=0x7fdc1c15bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:02.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.359+0000 7fdc1b400640 1 -- 172.21.15.98:0/2192995757 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c0c8930 con 0x7fdc1c0cc4f0 2024-05-27T00:27:02.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.359+0000 7fdc1aa00640 1 -- 172.21.15.98:0/2192995757 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2137301597 0 0) 0x7fdc1c0c70b0 con 0x7fdc1c0629d0 2024-05-27T00:27:02.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc1b400640 1 --2- 172.21.15.98:0/2192995757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0cc4f0 0x7fdc1c0c7720 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c0c5cc0 tx=0x7fdc0c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a5b1e63ff1de3b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:02.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc1aa00640 1 -- 172.21.15.98:0/2192995757 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0b0070 con 0x7fdc1c0cc4f0 2024-05-27T00:27:02.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc1aa00640 1 -- 172.21.15.98:0/2192995757 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc0c0ab600 con 0x7fdc1c0cc4f0 2024-05-27T00:27:02.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc1aa00640 1 -- 172.21.15.98:0/2192995757 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0ab9d0 con 0x7fdc1c0cc4f0 2024-05-27T00:27:02.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc20a00640 1 -- 172.21.15.98:0/2192995757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0cc4f0 msgr2=0x7fdc1c0c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:02.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc20a00640 1 --2- 172.21.15.98:0/2192995757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0cc4f0 0x7fdc1c0c7720 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c0c5cc0 tx=0x7fdc0c01c480 comp rx=0 tx=0).stop 2024-05-27T00:27:02.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc20a00640 1 -- 172.21.15.98:0/2192995757 shutdown_connections 2024-05-27T00:27:02.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc20a00640 1 --2- 172.21.15.98:0/2192995757 >> v2:172.21.15.204:3300/0 conn(0x7fdc1c0c67c0 0x7fdc1c0cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc20a00640 1 --2- 172.21.15.98:0/2192995757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0cc4f0 0x7fdc1c0c7720 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc20a00640 1 -- 172.21.15.98:0/2192995757 >> 172.21.15.98:0/2192995757 conn(0x7fdc1c06dd00 msgr2=0x7fdc1c06b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:02.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc20a00640 1 -- 172.21.15.98:0/2192995757 shutdown_connections 2024-05-27T00:27:02.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc20a00640 1 -- 172.21.15.98:0/2192995757 wait complete. 2024-05-27T00:27:02.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc20a00640 1 Processor -- start 2024-05-27T00:27:02.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.363+0000 7fdc20a00640 1 -- start start 2024-05-27T00:27:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc20a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0629d0 0x7fdc1c1f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc20a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc1c0c67c0 0x7fdc1c1f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc20a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc1c0cc4f0 0x7fdc1c148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc20a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c165aa0 con 0x7fdc1c0629d0 2024-05-27T00:27:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc20a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c165920 con 0x7fdc1c0c67c0 2024-05-27T00:27:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc20a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1c165c20 con 0x7fdc1c0cc4f0 2024-05-27T00:27:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0629d0 0x7fdc1c1f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0629d0 0x7fdc1c1f79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33236/0 (socket says 172.21.15.98:33236) 2024-05-27T00:27:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1b400640 1 -- 172.21.15.98:0/62524320 learned_addr learned my addr 172.21.15.98:0/62524320 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc13e00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc1c0c67c0 0x7fdc1c1f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1be00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc1c0cc4f0 0x7fdc1c148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1b400640 1 -- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc1c0cc4f0 msgr2=0x7fdc1c148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1b400640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc1c0cc4f0 0x7fdc1c148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1b400640 1 -- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc1c0c67c0 msgr2=0x7fdc1c1f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1b400640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc1c0c67c0 0x7fdc1c1f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1b400640 1 -- 172.21.15.98:0/62524320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c0717d0 con 0x7fdc1c0629d0 2024-05-27T00:27:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc13e00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc1c0c67c0 0x7fdc1c1f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:27:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1be00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc1c0cc4f0 0x7fdc1c148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:27:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc1b400640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0629d0 0x7fdc1c1f79c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0ae040 tx=0x7fdc0c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc13400640 1 -- 172.21.15.98:0/62524320 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0b0050 con 0x7fdc1c0629d0 2024-05-27T00:27:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc13400640 1 -- 172.21.15.98:0/62524320 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc0c0bc070 con 0x7fdc1c0629d0 2024-05-27T00:27:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc13400640 1 -- 172.21.15.98:0/62524320 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0b5830 con 0x7fdc1c0629d0 2024-05-27T00:27:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc20a00640 1 -- 172.21.15.98:0/62524320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c071ab0 con 0x7fdc1c0629d0 2024-05-27T00:27:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc20a00640 1 -- 172.21.15.98:0/62524320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c1f71c0 con 0x7fdc1c0629d0 2024-05-27T00:27:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.367+0000 7fdc20a00640 1 -- 172.21.15.98:0/62524320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc1c075c40 con 0x7fdc1c0629d0 2024-05-27T00:27:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.371+0000 7fdc13400640 1 -- 172.21.15.98:0/62524320 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdc0c0b59d0 con 0x7fdc1c0629d0 2024-05-27T00:27:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.371+0000 7fdc13400640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbf80678f0 0x7fdbf8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.371+0000 7fdc13400640 1 -- 172.21.15.98:0/62524320 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7fdc0c120780 con 0x7fdc1c0629d0 2024-05-27T00:27:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.371+0000 7fdc13e00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbf80678f0 0x7fdbf8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.371+0000 7fdc13e00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbf80678f0 0x7fdbf8069db0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c1f1410 tx=0x7fdc04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.375+0000 7fdc13400640 1 -- 172.21.15.98:0/62524320 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fdc0c016070 con 0x7fdc1c0629d0 2024-05-27T00:27:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.563+0000 7fdc20a00640 1 -- 172.21.15.98:0/62524320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc1c162cd0 con 0x7fdc1c0629d0 2024-05-27T00:27:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.563+0000 7fdc13400640 1 -- 172.21.15.98:0/62524320 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6006 (secure 0 0 0) 0x7fdc0c0f1020 con 0x7fdc1c0629d0 2024-05-27T00:27:02.569 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:02.569 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:00.260048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 48 2024-05-27T00:27:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 -- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbf80678f0 msgr2=0x7fdbf8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbf80678f0 0x7fdbf8069db0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c1f1410 tx=0x7fdc04008040 comp rx=0 tx=0).stop 2024-05-27T00:27:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 -- 172.21.15.98:0/62524320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0629d0 msgr2=0x7fdc1c1f79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0629d0 0x7fdc1c1f79c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0ae040 tx=0x7fdc0c01acc0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 -- 172.21.15.98:0/62524320 shutdown_connections 2024-05-27T00:27:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc1c0cc4f0 0x7fdc1c148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbf80678f0 0x7fdbf8069db0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc1c0c67c0 0x7fdc1c1f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 --2- 172.21.15.98:0/62524320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc1c0629d0 0x7fdc1c1f79c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:02.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 -- 172.21.15.98:0/62524320 >> 172.21.15.98:0/62524320 conn(0x7fdc1c06dd00 msgr2=0x7fdc1c0cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:02.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 -- 172.21.15.98:0/62524320 shutdown_connections 2024-05-27T00:27:02.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:02.567+0000 7fdc20a00640 1 -- 172.21.15.98:0/62524320 wait complete. 2024-05-27T00:27:02.591 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5802": { "addr": "172.21.15.98:6837/2940211164", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 2940211164, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2940211164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5802, "incarnation": 48, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:27:00.260048+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5802 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:27:04.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:04.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f6c00640 1 Processor -- start 2024-05-27T00:27:04.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f6c00640 1 -- start start 2024-05-27T00:27:04.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f6c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb4f00cb7e0 0x7fb4f0158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:04.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f6c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 0x7fb4f0158c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:04.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f6c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4f015be40 con 0x7fb4f0152a50 2024-05-27T00:27:04.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f6c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb4f00c8a00 con 0x7fb4f00cb7e0 2024-05-27T00:27:04.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f6c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f00c50e0 con 0x7fb4f00b3ed0 2024-05-27T00:27:04.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4ef400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb4f00cb7e0 0x7fb4f0158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:04.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f4c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 0x7fb4f0158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:04.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4efe00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fb4f0152a50 0x7fb4f00709a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:35660/0 (socket says 172.21.15.98:35660) 2024-05-27T00:27:04.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4efe00640 1 -- 172.21.15.98:0/3554532451 learned_addr learned my addr 172.21.15.98:0/3554532451 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f4c00640 1 -- 172.21.15.98:0/3554532451 >> v2:172.21.15.204:3300/0 conn(0x7fb4f00cb7e0 msgr2=0x7fb4f0158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f4c00640 1 --2- 172.21.15.98:0/3554532451 >> v2:172.21.15.204:3300/0 conn(0x7fb4f00cb7e0 0x7fb4f0158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f4c00640 1 -- 172.21.15.98:0/3554532451 >> v1:172.21.15.204:6789/0 conn(0x7fb4f0152a50 legacy=0x7fb4f00709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4f4c00640 1 -- 172.21.15.98:0/3554532451 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f00c8c50 con 0x7fb4f00b3ed0 2024-05-27T00:27:04.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.731+0000 7fb4ef400640 1 --2- 172.21.15.98:0/3554532451 >> v2:172.21.15.204:3300/0 conn(0x7fb4f00cb7e0 0x7fb4f0158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:27:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4f4c00640 1 --2- 172.21.15.98:0/3554532451 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 0x7fb4f0158c90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009d90 tx=0x7fb4e801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e296d0e72c825255 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:04.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4eea00640 1 -- 172.21.15.98:0/3554532451 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e80b2070 con 0x7fb4f00b3ed0 2024-05-27T00:27:04.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4eea00640 1 -- 172.21.15.98:0/3554532451 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e80a5970 con 0x7fb4f00b3ed0 2024-05-27T00:27:04.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4eea00640 1 -- 172.21.15.98:0/3554532451 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e80a5d40 con 0x7fb4f00b3ed0 2024-05-27T00:27:04.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3554532451 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 msgr2=0x7fb4f0158c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:04.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4f6c00640 1 --2- 172.21.15.98:0/3554532451 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 0x7fb4f0158c90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009d90 tx=0x7fb4e801c8b0 comp rx=0 tx=0).stop 2024-05-27T00:27:04.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3554532451 shutdown_connections 2024-05-27T00:27:04.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4f6c00640 1 --2- 172.21.15.98:0/3554532451 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 0x7fb4f0158c90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:04.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4f6c00640 1 --2- 172.21.15.98:0/3554532451 >> v2:172.21.15.204:3300/0 conn(0x7fb4f00cb7e0 0x7fb4f0158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:04.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3554532451 >> 172.21.15.98:0/3554532451 conn(0x7fb4f0072f90 msgr2=0x7fb4f00733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:04.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3554532451 shutdown_connections 2024-05-27T00:27:04.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.735+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3554532451 wait complete. 2024-05-27T00:27:04.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f6c00640 1 Processor -- start 2024-05-27T00:27:04.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f6c00640 1 -- start start 2024-05-27T00:27:04.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f6c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 0x7fb4f01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:04.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f6c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f00cb7e0 0x7fb4f00c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:04.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f6c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0152a50 0x7fb4f00c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:04.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f6c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f0152e30 con 0x7fb4f0152a50 2024-05-27T00:27:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f6c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f00cb570 con 0x7fb4f00cb7e0 2024-05-27T00:27:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f6c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f00c9c10 con 0x7fb4f00b3ed0 2024-05-27T00:27:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f4c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0152a50 0x7fb4f00c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f4c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0152a50 0x7fb4f00c6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51982/0 (socket says 172.21.15.98:51982) 2024-05-27T00:27:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f4c00640 1 -- 172.21.15.98:0/3390656807 learned_addr learned my addr 172.21.15.98:0/3390656807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4efe00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 0x7fb4f01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:04.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4ef400640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f00cb7e0 0x7fb4f00c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f4c00640 1 -- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 msgr2=0x7fb4f01570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:04.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f4c00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 0x7fb4f01570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f4c00640 1 -- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f00cb7e0 msgr2=0x7fb4f00c6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f4c00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f00cb7e0 0x7fb4f00c6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f4c00640 1 -- 172.21.15.98:0/3390656807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f01f7580 con 0x7fb4f0152a50 2024-05-27T00:27:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f4c00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0152a50 0x7fb4f00c6940 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009d90 tx=0x7fb4e8019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4e7e00640 1 -- 172.21.15.98:0/3390656807 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e80b2050 con 0x7fb4f0152a50 2024-05-27T00:27:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4e7e00640 1 -- 172.21.15.98:0/3390656807 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e80be070 con 0x7fb4f0152a50 2024-05-27T00:27:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4e7e00640 1 -- 172.21.15.98:0/3390656807 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e80ada90 con 0x7fb4f0152a50 2024-05-27T00:27:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.739+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3390656807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f01f35a0 con 0x7fb4f0152a50 2024-05-27T00:27:04.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.743+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3390656807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f01f3b60 con 0x7fb4f0152a50 2024-05-27T00:27:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.743+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3390656807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f0075cb0 con 0x7fb4f0152a50 2024-05-27T00:27:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.743+0000 7fb4e7e00640 1 -- 172.21.15.98:0/3390656807 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb4e80adc30 con 0x7fb4f0152a50 2024-05-27T00:27:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.743+0000 7fb4e7e00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4d00678f0 0x7fb4d0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:04.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.743+0000 7fb4e7e00640 1 -- 172.21.15.98:0/3390656807 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7fb4e8121220 con 0x7fb4f0152a50 2024-05-27T00:27:04.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.743+0000 7fb4efe00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4d00678f0 0x7fb4d0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:04.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.743+0000 7fb4efe00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4d00678f0 0x7fb4d0069db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0000f70 tx=0x7fb4e0007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:04.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.747+0000 7fb4e7e00640 1 -- 172.21.15.98:0/3390656807 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb4e8016560 con 0x7fb4f0152a50 2024-05-27T00:27:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.943+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3390656807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4f01563a0 con 0x7fb4f0152a50 2024-05-27T00:27:04.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.943+0000 7fb4e7e00640 1 -- 172.21.15.98:0/3390656807 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6006 (secure 0 0 0) 0x7fb4e80f12b0 con 0x7fb4f0152a50 2024-05-27T00:27:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:04.949 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:00.260048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:04.949 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 48 2024-05-27T00:27:04.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4d00678f0 msgr2=0x7fb4d0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4d00678f0 0x7fb4d0069db0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0000f70 tx=0x7fb4e0007e90 comp rx=0 tx=0).stop 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3390656807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0152a50 msgr2=0x7fb4f00c6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0152a50 0x7fb4f00c6940 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009d90 tx=0x7fb4e8019650 comp rx=0 tx=0).stop 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3390656807 shutdown_connections 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4f0152a50 0x7fb4f00c6940 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4d00678f0 0x7fb4d0069db0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4f00cb7e0 0x7fb4f00c6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 --2- 172.21.15.98:0/3390656807 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4f00b3ed0 0x7fb4f01570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3390656807 >> 172.21.15.98:0/3390656807 conn(0x7fb4f0072f90 msgr2=0x7fb4f00ca280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3390656807 shutdown_connections 2024-05-27T00:27:04.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:04.947+0000 7fb4f6c00640 1 -- 172.21.15.98:0/3390656807 wait complete. 2024-05-27T00:27:04.971 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:27:06.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.167+0000 7f7f83600640 1 Processor -- start 2024-05-27T00:27:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.167+0000 7f7f83600640 1 -- start start 2024-05-27T00:27:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f83600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f7c0731f0 0x7f7f7c0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f83600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7f7c0c79f0 0x7f7f7c157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f83600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f7c0c17d0 con 0x7f7f7c0735d0 2024-05-27T00:27:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f83600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7f7c071990 con 0x7f7f7c0731f0 2024-05-27T00:27:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f83600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7f7c070e10 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f7be00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7f7c0735d0 0x7f7f7c151430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52456/0 (socket says 172.21.15.98:52456) 2024-05-27T00:27:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f7be00640 1 -- 172.21.15.98:0/1253774185 learned_addr learned my addr 172.21.15.98:0/1253774185 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f80c00640 1 --2- 172.21.15.98:0/1253774185 >> v2:172.21.15.98:3300/0 conn(0x7f7f7c0731f0 0x7f7f7c0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:07.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f81600640 1 --2- 172.21.15.98:0/1253774185 >> v2:172.21.15.204:3300/0 conn(0x7f7f7c0c79f0 0x7f7f7c157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:07.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f7b400640 1 -- 172.21.15.98:0/1253774185 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4281022136 0 0) 0x7f7f7c0c17d0 con 0x7f7f7c0735d0 2024-05-27T00:27:07.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f7b400640 1 -- 172.21.15.98:0/1253774185 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f64003530 con 0x7f7f7c0735d0 2024-05-27T00:27:07.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f80c00640 1 -- 172.21.15.98:0/1253774185 >> v2:172.21.15.204:3300/0 conn(0x7f7f7c0c79f0 msgr2=0x7f7f7c157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:07.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f80c00640 1 --2- 172.21.15.98:0/1253774185 >> v2:172.21.15.204:3300/0 conn(0x7f7f7c0c79f0 0x7f7f7c157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:07.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f80c00640 1 -- 172.21.15.98:0/1253774185 >> v1:172.21.15.98:6789/0 conn(0x7f7f7c0735d0 legacy=0x7f7f7c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:07.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.171+0000 7f7f80c00640 1 -- 172.21.15.98:0/1253774185 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c0710f0 con 0x7f7f7c0731f0 2024-05-27T00:27:07.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.175+0000 7f7f80c00640 1 --2- 172.21.15.98:0/1253774185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0731f0 0x7f7f7c0c0eb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c06c530 tx=0x7f7f7001e660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a5d6f052f0b11f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:07.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f7b400640 1 -- 172.21.15.98:0/1253774185 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700ab630 con 0x7f7f7c0731f0 2024-05-27T00:27:07.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f7b400640 1 -- 172.21.15.98:0/1253774185 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f700ab7d0 con 0x7f7f7c0731f0 2024-05-27T00:27:07.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f7b400640 1 -- 172.21.15.98:0/1253774185 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700abba0 con 0x7f7f7c0731f0 2024-05-27T00:27:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 -- 172.21.15.98:0/1253774185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0731f0 msgr2=0x7f7f7c0c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 --2- 172.21.15.98:0/1253774185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0731f0 0x7f7f7c0c0eb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c06c530 tx=0x7f7f7001e660 comp rx=0 tx=0).stop 2024-05-27T00:27:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 -- 172.21.15.98:0/1253774185 shutdown_connections 2024-05-27T00:27:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 --2- 172.21.15.98:0/1253774185 >> v2:172.21.15.204:3300/0 conn(0x7f7f7c0c79f0 0x7f7f7c157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 --2- 172.21.15.98:0/1253774185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0731f0 0x7f7f7c0c0eb0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 -- 172.21.15.98:0/1253774185 >> 172.21.15.98:0/1253774185 conn(0x7f7f7c067530 msgr2=0x7f7f7c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 -- 172.21.15.98:0/1253774185 shutdown_connections 2024-05-27T00:27:07.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 -- 172.21.15.98:0/1253774185 wait complete. 2024-05-27T00:27:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 Processor -- start 2024-05-27T00:27:07.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 -- start start 2024-05-27T00:27:07.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f7c0731f0 0x7f7f7c147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:07.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f7c0735d0 0x7f7f7c1f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:07.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0c79f0 0x7f7f7c1f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c0c13f0 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c0c8400 con 0x7f7f7c0731f0 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f83600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7f7c0c1570 con 0x7f7f7c0735d0 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.179+0000 7f7f81600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0c79f0 0x7f7f7c1f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f81600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0c79f0 0x7f7f7c1f7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52006/0 (socket says 172.21.15.98:52006) 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f81600640 1 -- 172.21.15.98:0/2563818467 learned_addr learned my addr 172.21.15.98:0/2563818467 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f80c00640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f7c0731f0 0x7f7f7c147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f7be00640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f7c0735d0 0x7f7f7c1f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f81600640 1 -- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f7c0735d0 msgr2=0x7f7f7c1f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f81600640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f7c0735d0 0x7f7f7c1f7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f81600640 1 -- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f7c0731f0 msgr2=0x7f7f7c147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f81600640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f7c0731f0 0x7f7f7c147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:07.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f81600640 1 -- 172.21.15.98:0/2563818467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c0c1080 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f81600640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0c79f0 0x7f7f7c1f7750 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400efc0 tx=0x7f7f7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f78c00640 1 -- 172.21.15.98:0/2563818467 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74018020 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f78c00640 1 -- 172.21.15.98:0/2563818467 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f74014480 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f83600640 1 -- 172.21.15.98:0/2563818467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7c1587e0 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f78c00640 1 -- 172.21.15.98:0/2563818467 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74014870 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f83600640 1 -- 172.21.15.98:0/2563818467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f7c147cc0 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f83600640 1 -- 172.21.15.98:0/2563818467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7c075cb0 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.183+0000 7f7f78c00640 1 -- 172.21.15.98:0/2563818467 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7f7401b020 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.187+0000 7f7f78c00640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 0x7f7f5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:07.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.187+0000 7f7f78c00640 1 -- 172.21.15.98:0/2563818467 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7f7f74086cd0 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.187+0000 7f7f80c00640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 0x7f7f5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:07.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.187+0000 7f7f80c00640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 0x7f7f5c069d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7f700b0000 tx=0x7f7f700a9000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:07.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.191+0000 7f7f78c00640 1 -- 172.21.15.98:0/2563818467 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7f74022030 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.399+0000 7f7f83600640 1 -- 172.21.15.98:0/2563818467 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f7c158510 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.399+0000 7f7f78c00640 1 -- 172.21.15.98:0/2563818467 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6006 (secure 0 0 0) 0x7f7f74010040 con 0x7f7f7c0c79f0 2024-05-27T00:27:07.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:07.404 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:00.260048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:07.404 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 48 2024-05-27T00:27:07.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 -- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 msgr2=0x7f7f5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:07.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 0x7f7f5c069d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7f700b0000 tx=0x7f7f700a9000 comp rx=0 tx=0).stop 2024-05-27T00:27:07.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 -- 172.21.15.98:0/2563818467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0c79f0 msgr2=0x7f7f7c1f7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:07.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0c79f0 0x7f7f7c1f7750 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400efc0 tx=0x7f7f7400c490 comp rx=0 tx=0).stop 2024-05-27T00:27:07.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 -- 172.21.15.98:0/2563818467 shutdown_connections 2024-05-27T00:27:07.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f7c0c79f0 0x7f7f7c1f7750 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:07.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f7c0735d0 0x7f7f7c1f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:07.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 0x7f7f5c069d80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:07.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 --2- 172.21.15.98:0/2563818467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f7c0731f0 0x7f7f7c147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:07.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 -- 172.21.15.98:0/2563818467 >> 172.21.15.98:0/2563818467 conn(0x7f7f7c067530 msgr2=0x7f7f7c151200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:07.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 -- 172.21.15.98:0/2563818467 shutdown_connections 2024-05-27T00:27:07.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:07.403+0000 7f7f83600640 1 -- 172.21.15.98:0/2563818467 wait complete. 2024-05-27T00:27:07.427 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:27:09.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:09.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.575+0000 7fc0d1a00640 1 Processor -- start 2024-05-27T00:27:09.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.575+0000 7fc0d1a00640 1 -- start start 2024-05-27T00:27:09.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0d1a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc158d90 0x7fc0cc0c26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:09.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0d1a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0cc0c8230 con 0x7fc0cc0c7dc0 2024-05-27T00:27:09.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0d1a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0cc159170 con 0x7fc0cc0c0eb0 2024-05-27T00:27:09.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0d1a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc0cc15f840 con 0x7fc0cc158d90 2024-05-27T00:27:09.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0cb400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc0cc0c7dc0 0x7fc0cc15ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52472/0 (socket says 172.21.15.98:52472) 2024-05-27T00:27:09.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0cb400640 1 -- 172.21.15.98:0/2576107412 learned_addr learned my addr 172.21.15.98:0/2576107412 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:09.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0caa00640 1 --2- 172.21.15.98:0/2576107412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc158d90 0x7fc0cc0c26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:09.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0c9600640 1 -- 172.21.15.98:0/2576107412 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 9961699 0 0) 0x7fc0cc0c8230 con 0x7fc0cc0c7dc0 2024-05-27T00:27:09.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0c9600640 1 -- 172.21.15.98:0/2576107412 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0b0003530 con 0x7fc0cc0c7dc0 2024-05-27T00:27:09.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0c9600640 1 -- 172.21.15.98:0/2576107412 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2617674727 0 0) 0x7fc0cc159170 con 0x7fc0cc0c0eb0 2024-05-27T00:27:09.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0c9600640 1 -- 172.21.15.98:0/2576107412 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0cc0c8230 con 0x7fc0cc0c0eb0 2024-05-27T00:27:09.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0caa00640 1 -- 172.21.15.98:0/2576107412 >> v1:172.21.15.204:6789/0 conn(0x7fc0cc0c0eb0 legacy=0x7fc0cc0c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:09.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0caa00640 1 -- 172.21.15.98:0/2576107412 >> v1:172.21.15.98:6789/0 conn(0x7fc0cc0c7dc0 legacy=0x7fc0cc15ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:09.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.579+0000 7fc0caa00640 1 -- 172.21.15.98:0/2576107412 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0cc15fb20 con 0x7fc0cc158d90 2024-05-27T00:27:09.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.583+0000 7fc0caa00640 1 --2- 172.21.15.98:0/2576107412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc158d90 0x7fc0cc0c26a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0097f0 tx=0x7fc0bc01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd89a559f96a4dbf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:09.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.583+0000 7fc0c9600640 1 -- 172.21.15.98:0/2576107412 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc0b5070 con 0x7fc0cc158d90 2024-05-27T00:27:09.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.583+0000 7fc0c9600640 1 -- 172.21.15.98:0/2576107412 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0bc0a8990 con 0x7fc0cc158d90 2024-05-27T00:27:09.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.583+0000 7fc0c9600640 1 -- 172.21.15.98:0/2576107412 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc0a8d40 con 0x7fc0cc158d90 2024-05-27T00:27:09.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.583+0000 7fc0d1a00640 1 -- 172.21.15.98:0/2576107412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc158d90 msgr2=0x7fc0cc0c26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:09.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.583+0000 7fc0d1a00640 1 --2- 172.21.15.98:0/2576107412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc158d90 0x7fc0cc0c26a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0097f0 tx=0x7fc0bc01ec90 comp rx=0 tx=0).stop 2024-05-27T00:27:09.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.583+0000 7fc0d1a00640 1 -- 172.21.15.98:0/2576107412 shutdown_connections 2024-05-27T00:27:09.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.583+0000 7fc0d1a00640 1 --2- 172.21.15.98:0/2576107412 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc158d90 0x7fc0cc0c26a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:09.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.583+0000 7fc0d1a00640 1 -- 172.21.15.98:0/2576107412 >> 172.21.15.98:0/2576107412 conn(0x7fc0cc067530 msgr2=0x7fc0cc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0d1a00640 1 -- 172.21.15.98:0/2576107412 shutdown_connections 2024-05-27T00:27:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0d1a00640 1 -- 172.21.15.98:0/2576107412 wait complete. 2024-05-27T00:27:09.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0d1a00640 1 Processor -- start 2024-05-27T00:27:09.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0d1a00640 1 -- start start 2024-05-27T00:27:09.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0d1a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc0c0eb0 0x7fc0cc162020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:09.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0d1a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0cc0c7dc0 0x7fc0cc160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:09.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0d1a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0cc158d90 0x7fc0cc1609b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:09.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0d1a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc0c1ea0 con 0x7fc0cc158d90 2024-05-27T00:27:09.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0d1a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc0c1bc0 con 0x7fc0cc0c7dc0 2024-05-27T00:27:09.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0d1a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc0cc0c2180 con 0x7fc0cc0c0eb0 2024-05-27T00:27:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0cb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0cc158d90 0x7fc0cc1609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0caa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc0c0eb0 0x7fc0cc162020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0cb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0cc158d90 0x7fc0cc1609b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52016/0 (socket says 172.21.15.98:52016) 2024-05-27T00:27:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0cb400640 1 -- 172.21.15.98:0/3311554141 learned_addr learned my addr 172.21.15.98:0/3311554141 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.587+0000 7fc0ca000640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0cc0c7dc0 0x7fc0cc160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0cb400640 1 -- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc0c0eb0 msgr2=0x7fc0cc162020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0cb400640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc0c0eb0 0x7fc0cc162020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0cb400640 1 -- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0cc0c7dc0 msgr2=0x7fc0cc160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0cb400640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0cc0c7dc0 0x7fc0cc160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0cb400640 1 -- 172.21.15.98:0/3311554141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0cc0c3370 con 0x7fc0cc158d90 2024-05-27T00:27:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0cb400640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0cc158d90 0x7fc0cc1609b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400cb70 tx=0x7fc0c400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0c2a00640 1 -- 172.21.15.98:0/3311554141 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4017070 con 0x7fc0cc158d90 2024-05-27T00:27:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0c2a00640 1 -- 172.21.15.98:0/3311554141 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0c400ad40 con 0x7fc0cc158d90 2024-05-27T00:27:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0c2a00640 1 -- 172.21.15.98:0/3311554141 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4012620 con 0x7fc0cc158d90 2024-05-27T00:27:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0d1a00640 1 -- 172.21.15.98:0/3311554141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0cc0c3090 con 0x7fc0cc158d90 2024-05-27T00:27:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0d1a00640 1 -- 172.21.15.98:0/3311554141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0cc162d40 con 0x7fc0cc158d90 2024-05-27T00:27:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0d1a00640 1 -- 172.21.15.98:0/3311554141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0cc075cb0 con 0x7fc0cc158d90 2024-05-27T00:27:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.591+0000 7fc0c2a00640 1 -- 172.21.15.98:0/3311554141 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc0c401a070 con 0x7fc0cc158d90 2024-05-27T00:27:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.595+0000 7fc0c2a00640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0ac0678f0 0x7fc0ac069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.595+0000 7fc0c2a00640 1 -- 172.21.15.98:0/3311554141 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7fc0c4085ca0 con 0x7fc0cc158d90 2024-05-27T00:27:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.595+0000 7fc0caa00640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0ac0678f0 0x7fc0ac069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.595+0000 7fc0caa00640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0ac0678f0 0x7fc0ac069db0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0b2000 tx=0x7fc0bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.599+0000 7fc0c2a00640 1 -- 172.21.15.98:0/3311554141 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc0c4055e60 con 0x7fc0cc158d90 2024-05-27T00:27:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 -- 172.21.15.98:0/3311554141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0cc0c7420 con 0x7fc0cc158d90 2024-05-27T00:27:09.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0c2a00640 1 -- 172.21.15.98:0/3311554141 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6006 (secure 0 0 0) 0x7fc0c405a160 con 0x7fc0cc158d90 2024-05-27T00:27:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:09.810 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:00.260048+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:09.810 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 48 2024-05-27T00:27:09.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 -- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0ac0678f0 msgr2=0x7fc0ac069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:09.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0ac0678f0 0x7fc0ac069db0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0b2000 tx=0x7fc0bc008040 comp rx=0 tx=0).stop 2024-05-27T00:27:09.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 -- 172.21.15.98:0/3311554141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0cc158d90 msgr2=0x7fc0cc1609b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:09.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0cc158d90 0x7fc0cc1609b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400cb70 tx=0x7fc0c400a430 comp rx=0 tx=0).stop 2024-05-27T00:27:09.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 -- 172.21.15.98:0/3311554141 shutdown_connections 2024-05-27T00:27:09.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0cc158d90 0x7fc0cc1609b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:09.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0ac0678f0 0x7fc0ac069db0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:09.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0cc0c7dc0 0x7fc0cc160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:09.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 --2- 172.21.15.98:0/3311554141 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0cc0c0eb0 0x7fc0cc162020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:09.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.807+0000 7fc0d1a00640 1 -- 172.21.15.98:0/3311554141 >> 172.21.15.98:0/3311554141 conn(0x7fc0cc067530 msgr2=0x7fc0cc0c8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:09.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.811+0000 7fc0d1a00640 1 -- 172.21.15.98:0/3311554141 shutdown_connections 2024-05-27T00:27:09.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:09.811+0000 7fc0d1a00640 1 -- 172.21.15.98:0/3311554141 wait complete. 2024-05-27T00:27:09.837 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:27:11.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:11.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9ee600640 1 Processor -- start 2024-05-27T00:27:11.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9ee600640 1 -- start start 2024-05-27T00:27:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9ee600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80731f0 0x7fa9e80c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9ee600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa9e80c79f0 0x7fa9e8157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9ee600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9e80c17d0 con 0x7fa9e80735d0 2024-05-27T00:27:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9ee600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa9e8071990 con 0x7fa9e80c79f0 2024-05-27T00:27:11.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9ee600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e8070e10 con 0x7fa9e80731f0 2024-05-27T00:27:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9e7e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa9e80c79f0 0x7fa9e8157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80731f0 0x7fa9e80c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9e7e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa9e80c79f0 0x7fa9e8157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:47724/0 (socket says 172.21.15.98:47724) 2024-05-27T00:27:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80731f0 0x7fa9e80c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:58548/0 (socket says 172.21.15.98:58548) 2024-05-27T00:27:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9e7e00640 1 -- 172.21.15.98:0/365594474 learned_addr learned my addr 172.21.15.98:0/365594474 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.971+0000 7fa9e6a00640 1 --1- 172.21.15.98:0/365594474 >> v1:172.21.15.204:6789/0 conn(0x7fa9e80735d0 0x7fa9e8151430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:35676/0 (socket says 172.21.15.98:35676) 2024-05-27T00:27:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9e7400640 1 -- 172.21.15.98:0/365594474 >> v2:172.21.15.204:3300/0 conn(0x7fa9e80c79f0 msgr2=0x7fa9e8157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:11.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9e7400640 1 --2- 172.21.15.98:0/365594474 >> v2:172.21.15.204:3300/0 conn(0x7fa9e80c79f0 0x7fa9e8157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9e7400640 1 -- 172.21.15.98:0/365594474 >> v1:172.21.15.204:6789/0 conn(0x7fa9e80735d0 legacy=0x7fa9e8151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9e7400640 1 -- 172.21.15.98:0/365594474 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e80710f0 con 0x7fa9e80731f0 2024-05-27T00:27:11.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9e6000640 1 -- 172.21.15.98:0/365594474 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104533006 0 0) 0x7fa9e80c17d0 con 0x7fa9e80735d0 2024-05-27T00:27:11.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9e7400640 1 --2- 172.21.15.98:0/365594474 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80731f0 0x7fa9e80c0eb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa9d80097c0 tx=0x7fa9d801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc06c9f4d3518de1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9e6000640 1 -- 172.21.15.98:0/365594474 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d80b5070 con 0x7fa9e80731f0 2024-05-27T00:27:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9e6000640 1 -- 172.21.15.98:0/365594474 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d80a8990 con 0x7fa9e80731f0 2024-05-27T00:27:11.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9e6000640 1 -- 172.21.15.98:0/365594474 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d80a8d60 con 0x7fa9e80731f0 2024-05-27T00:27:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9ee600640 1 -- 172.21.15.98:0/365594474 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80731f0 msgr2=0x7fa9e80c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.975+0000 7fa9ee600640 1 --2- 172.21.15.98:0/365594474 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80731f0 0x7fa9e80c0eb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa9d80097c0 tx=0x7fa9d801ec90 comp rx=0 tx=0).stop 2024-05-27T00:27:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 -- 172.21.15.98:0/365594474 shutdown_connections 2024-05-27T00:27:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 --2- 172.21.15.98:0/365594474 >> v2:172.21.15.204:3300/0 conn(0x7fa9e80c79f0 0x7fa9e8157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 --2- 172.21.15.98:0/365594474 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80731f0 0x7fa9e80c0eb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 -- 172.21.15.98:0/365594474 >> 172.21.15.98:0/365594474 conn(0x7fa9e8067530 msgr2=0x7fa9e806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 -- 172.21.15.98:0/365594474 shutdown_connections 2024-05-27T00:27:11.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 -- 172.21.15.98:0/365594474 wait complete. 2024-05-27T00:27:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 Processor -- start 2024-05-27T00:27:11.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 -- start start 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9e80731f0 0x7fa9e81f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80735d0 0x7fa9e8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e80c79f0 0x7fa9e814f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e80c8400 con 0x7fa9e80c79f0 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e80716b0 con 0x7fa9e80731f0 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9ee600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e80c13f0 con 0x7fa9e80735d0 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9e80731f0 0x7fa9e81f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e80c79f0 0x7fa9e814f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80735d0 0x7fa9e8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9e80731f0 0x7fa9e81f3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:47740/0 (socket says 172.21.15.98:47740) 2024-05-27T00:27:11.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e7400640 1 -- 172.21.15.98:0/1639475948 learned_addr learned my addr 172.21.15.98:0/1639475948 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e7e00640 1 -- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80735d0 msgr2=0x7fa9e8159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e7e00640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80735d0 0x7fa9e8159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:11.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e7e00640 1 -- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9e80731f0 msgr2=0x7fa9e81f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e7e00640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9e80731f0 0x7fa9e81f3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e7e00640 1 -- 172.21.15.98:0/1639475948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e815a170 con 0x7fa9e80c79f0 2024-05-27T00:27:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.979+0000 7fa9e7400640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9e80731f0 0x7fa9e81f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:27:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9e7e00640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e80c79f0 0x7fa9e814f4c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000efc0 tx=0x7fa9e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9df400640 1 -- 172.21.15.98:0/1639475948 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000cd20 con 0x7fa9e80c79f0 2024-05-27T00:27:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9df400640 1 -- 172.21.15.98:0/1639475948 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e00039a0 con 0x7fa9e80c79f0 2024-05-27T00:27:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9df400640 1 -- 172.21.15.98:0/1639475948 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0003d90 con 0x7fa9e80c79f0 2024-05-27T00:27:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9ee600640 1 -- 172.21.15.98:0/1639475948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e814d970 con 0x7fa9e80c79f0 2024-05-27T00:27:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9ee600640 1 -- 172.21.15.98:0/1639475948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e81f0190 con 0x7fa9e80c79f0 2024-05-27T00:27:11.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9ee600640 1 -- 172.21.15.98:0/1639475948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e8075cb0 con 0x7fa9e80c79f0 2024-05-27T00:27:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9df400640 1 -- 172.21.15.98:0/1639475948 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9e0012050 con 0x7fa9e80c79f0 2024-05-27T00:27:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9df400640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9c80678f0 0x7fa9c8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9df400640 1 -- 172.21.15.98:0/1639475948 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7fa9e0086370 con 0x7fa9e80c79f0 2024-05-27T00:27:11.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.983+0000 7fa9e7400640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9c80678f0 0x7fa9c8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:11.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.987+0000 7fa9e7400640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9c80678f0 0x7fa9c8069db0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8000c10 tx=0x7fa9d80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:11.991+0000 7fa9df400640 1 -- 172.21.15.98:0/1639475948 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa9e0022020 con 0x7fa9e80c79f0 2024-05-27T00:27:12.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.183+0000 7fa9ee600640 1 -- 172.21.15.98:0/1639475948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9e81f0780 con 0x7fa9e80c79f0 2024-05-27T00:27:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.183+0000 7fa9df400640 1 -- 172.21.15.98:0/1639475948 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6010 (secure 0 0 0) 0x7fa9e0010040 con 0x7fa9e80c79f0 2024-05-27T00:27:12.188 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:12.188 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:10.669108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:12.189 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 49 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 -- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9c80678f0 msgr2=0x7fa9c8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9c80678f0 0x7fa9c8069db0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8000c10 tx=0x7fa9d80aa070 comp rx=0 tx=0).stop 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 -- 172.21.15.98:0/1639475948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e80c79f0 msgr2=0x7fa9e814f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e80c79f0 0x7fa9e814f4c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000efc0 tx=0x7fa9e000c490 comp rx=0 tx=0).stop 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 -- 172.21.15.98:0/1639475948 shutdown_connections 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9e80c79f0 0x7fa9e814f4c0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9e80735d0 0x7fa9e8159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9c80678f0 0x7fa9c8069db0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 --2- 172.21.15.98:0/1639475948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9e80731f0 0x7fa9e81f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 -- 172.21.15.98:0/1639475948 >> 172.21.15.98:0/1639475948 conn(0x7fa9e8067530 msgr2=0x7fa9e80c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:12.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 -- 172.21.15.98:0/1639475948 shutdown_connections 2024-05-27T00:27:12.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:12.187+0000 7fa9ee600640 1 -- 172.21.15.98:0/1639475948 wait complete. 2024-05-27T00:27:12.209 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:27:14.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.343+0000 7ff946a00640 1 Processor -- start 2024-05-27T00:27:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.343+0000 7ff946a00640 1 -- start start 2024-05-27T00:27:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.343+0000 7ff946a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff94006c620 0x7ff9400c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:14.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.343+0000 7ff946a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9400c78a0 0x7ff940067130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:14.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.343+0000 7ff946a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff94006cd90 con 0x7ff940154e90 2024-05-27T00:27:14.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.343+0000 7ff946a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff94006c200 con 0x7ff9400c78a0 2024-05-27T00:27:14.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.343+0000 7ff946a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff9400c97e0 con 0x7ff94006c620 2024-05-27T00:27:14.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.347+0000 7ff944a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9400c78a0 0x7ff940067130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:14.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.347+0000 7ff944a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff9400c78a0 0x7ff940067130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37992/0 (socket says 172.21.15.98:37992) 2024-05-27T00:27:14.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.347+0000 7ff944a00640 1 -- 172.21.15.98:0/85003401 learned_addr learned my addr 172.21.15.98:0/85003401 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:14.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.347+0000 7ff93f400640 1 --2- 172.21.15.98:0/85003401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff94006c620 0x7ff9400c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:14.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.347+0000 7ff944a00640 1 -- 172.21.15.98:0/85003401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff94006c620 msgr2=0x7ff9400c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:14.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.347+0000 7ff944a00640 1 --2- 172.21.15.98:0/85003401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff94006c620 0x7ff9400c92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:14.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.347+0000 7ff944a00640 1 -- 172.21.15.98:0/85003401 >> v1:172.21.15.204:6789/0 conn(0x7ff940154e90 legacy=0x7ff9400732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:14.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.347+0000 7ff944a00640 1 -- 172.21.15.98:0/85003401 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94006cab0 con 0x7ff9400c78a0 2024-05-27T00:27:14.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff944a00640 1 --2- 172.21.15.98:0/85003401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 0x7ff940067130 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff938009e50 tx=0x7ff93801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9732d58cbbb80f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:14.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff93ea00640 1 -- 172.21.15.98:0/85003401 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9380b4070 con 0x7ff9400c78a0 2024-05-27T00:27:14.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff93ea00640 1 -- 172.21.15.98:0/85003401 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9380af400 con 0x7ff9400c78a0 2024-05-27T00:27:14.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff93ea00640 1 -- 172.21.15.98:0/85003401 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9380af7d0 con 0x7ff9400c78a0 2024-05-27T00:27:14.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff946a00640 1 -- 172.21.15.98:0/85003401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 msgr2=0x7ff940067130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:14.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff946a00640 1 --2- 172.21.15.98:0/85003401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 0x7ff940067130 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff938009e50 tx=0x7ff93801f210 comp rx=0 tx=0).stop 2024-05-27T00:27:14.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff946a00640 1 -- 172.21.15.98:0/85003401 shutdown_connections 2024-05-27T00:27:14.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff946a00640 1 --2- 172.21.15.98:0/85003401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 0x7ff940067130 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:14.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff946a00640 1 --2- 172.21.15.98:0/85003401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff94006c620 0x7ff9400c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:14.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff946a00640 1 -- 172.21.15.98:0/85003401 >> 172.21.15.98:0/85003401 conn(0x7ff9400c9b70 msgr2=0x7ff9400c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:14.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff946a00640 1 -- 172.21.15.98:0/85003401 shutdown_connections 2024-05-27T00:27:14.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff946a00640 1 -- 172.21.15.98:0/85003401 wait complete. 2024-05-27T00:27:14.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff946a00640 1 Processor -- start 2024-05-27T00:27:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.351+0000 7ff946a00640 1 -- start start 2024-05-27T00:27:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff946a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 0x7ff940149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff946a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff940154e90 0x7ff9401f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff946a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9400b1c20 0x7ff940067390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff946a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff940072db0 con 0x7ff9400c78a0 2024-05-27T00:27:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff946a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff94006bf20 con 0x7ff9400b1c20 2024-05-27T00:27:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff946a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff9400739e0 con 0x7ff940154e90 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff93fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 0x7ff940149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff93fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 0x7ff940149610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37998/0 (socket says 172.21.15.98:37998) 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff93fe00640 1 -- 172.21.15.98:0/3102412263 learned_addr learned my addr 172.21.15.98:0/3102412263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff93f400640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff940154e90 0x7ff9401f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff944a00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9400b1c20 0x7ff940067390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff93fe00640 1 -- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff940154e90 msgr2=0x7ff9401f48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff93fe00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff940154e90 0x7ff9401f48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff93fe00640 1 -- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9400b1c20 msgr2=0x7ff940067390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff93fe00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9400b1c20 0x7ff940067390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff93fe00640 1 -- 172.21.15.98:0/3102412263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9401f1420 con 0x7ff9400c78a0 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff93fe00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 0x7ff940149610 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff93000c3c0 tx=0x7ff93000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff937e00640 1 -- 172.21.15.98:0/3102412263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930017070 con 0x7ff9400c78a0 2024-05-27T00:27:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff937e00640 1 -- 172.21.15.98:0/3102412263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9300052d0 con 0x7ff9400c78a0 2024-05-27T00:27:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff946a00640 1 -- 172.21.15.98:0/3102412263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9401f16c0 con 0x7ff9400c78a0 2024-05-27T00:27:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff937e00640 1 -- 172.21.15.98:0/3102412263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930003840 con 0x7ff9400c78a0 2024-05-27T00:27:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff946a00640 1 -- 172.21.15.98:0/3102412263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9401f40d0 con 0x7ff9400c78a0 2024-05-27T00:27:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff946a00640 1 -- 172.21.15.98:0/3102412263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff940075cb0 con 0x7ff9400c78a0 2024-05-27T00:27:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.355+0000 7ff937e00640 1 -- 172.21.15.98:0/3102412263 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff93001a070 con 0x7ff9400c78a0 2024-05-27T00:27:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.359+0000 7ff937e00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9200678c0 0x7ff920069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.359+0000 7ff937e00640 1 -- 172.21.15.98:0/3102412263 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7ff930085300 con 0x7ff9400c78a0 2024-05-27T00:27:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.359+0000 7ff93f400640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9200678c0 0x7ff920069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.359+0000 7ff93f400640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9200678c0 0x7ff920069d80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff9401536c0 tx=0x7ff92c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.363+0000 7ff937e00640 1 -- 172.21.15.98:0/3102412263 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7ff930005480 con 0x7ff9400c78a0 2024-05-27T00:27:14.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 -- 172.21.15.98:0/3102412263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9400b24a0 con 0x7ff9400c78a0 2024-05-27T00:27:14.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff937e00640 1 -- 172.21.15.98:0/3102412263 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6010 (secure 0 0 0) 0x7ff930055b10 con 0x7ff9400c78a0 2024-05-27T00:27:14.559 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:14.559 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:10.669108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 49 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 -- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9200678c0 msgr2=0x7ff920069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9200678c0 0x7ff920069d80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff9401536c0 tx=0x7ff92c008040 comp rx=0 tx=0).stop 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 -- 172.21.15.98:0/3102412263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 msgr2=0x7ff940149610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 0x7ff940149610 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff93000c3c0 tx=0x7ff93000c880 comp rx=0 tx=0).stop 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 -- 172.21.15.98:0/3102412263 shutdown_connections 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ff9200678c0 0x7ff920069d80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9400b1c20 0x7ff940067390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff940154e90 0x7ff9401f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 --2- 172.21.15.98:0/3102412263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9400c78a0 0x7ff940149610 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 -- 172.21.15.98:0/3102412263 >> 172.21.15.98:0/3102412263 conn(0x7ff9400c9b70 msgr2=0x7ff9401587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 -- 172.21.15.98:0/3102412263 shutdown_connections 2024-05-27T00:27:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:14.555+0000 7ff946a00640 1 -- 172.21.15.98:0/3102412263 wait complete. 2024-05-27T00:27:14.581 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5802": { "addr": "172.21.15.98:6837/2940211164", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 2940211164, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2940211164, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5802, "incarnation": 48, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:27:10.669108+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5802 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:27:16.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:16.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc09b800640 1 Processor -- start 2024-05-27T00:27:16.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc09b800640 1 -- start start 2024-05-27T00:27:16.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc09b800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc0940c79f0 0x7fc094157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:16.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc09b800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0940c17d0 con 0x7fc0940735d0 2024-05-27T00:27:16.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc09b800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc094070e10 con 0x7fc0940731f0 2024-05-27T00:27:16.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc09b800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fc094071990 con 0x7fc0940c79f0 2024-05-27T00:27:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc093e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc0940735d0 0x7fc094151430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54906/0 (socket says 172.21.15.98:54906) 2024-05-27T00:27:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc093e00640 1 -- 172.21.15.98:0/4149258909 learned_addr learned my addr 172.21.15.98:0/4149258909 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc099800640 1 --2- 172.21.15.98:0/4149258909 >> v2:172.21.15.204:3300/0 conn(0x7fc0940c79f0 0x7fc094157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1851622421 0 0) 0x7fc0940c17d0 con 0x7fc0940735d0 2024-05-27T00:27:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.719+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc07c005740 con 0x7fc0940735d0 2024-05-27T00:27:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1274452519 0 0) 0x7fc07c005740 con 0x7fc0940735d0 2024-05-27T00:27:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 >> v2:172.21.15.204:3300/0 conn(0x7fc0940c79f0 msgr2=0x7fc094157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 --2- 172.21.15.98:0/4149258909 >> v2:172.21.15.204:3300/0 conn(0x7fc0940c79f0 0x7fc094157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 >> v1:172.21.15.204:6789/0 conn(0x7fc0940731f0 legacy=0x7fc0940c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0940710f0 con 0x7fc0940735d0 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fc080001680 con 0x7fc0940735d0 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 >> v1:172.21.15.98:6789/0 conn(0x7fc0940735d0 legacy=0x7fc094151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc07c005bd0 0x7fc07c005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc07c006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc07c0073e0 0x7fc07c00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc094071990 con 0x7fc07c006a20 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c00e260 con 0x7fc07c005bd0 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc07c00e460 con 0x7fc07c0073e0 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093e00640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc07c006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc098e00640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc07c005bd0 0x7fc07c005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc099800640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc07c0073e0 0x7fc07c00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093e00640 1 -- 172.21.15.98:0/4149258909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc07c0073e0 msgr2=0x7fc07c00dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093e00640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc07c0073e0 0x7fc07c00dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093e00640 1 -- 172.21.15.98:0/4149258909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc07c005bd0 msgr2=0x7fc07c005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:16.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093e00640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc07c005bd0 0x7fc07c005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093e00640 1 -- 172.21.15.98:0/4149258909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c005740 con 0x7fc07c006a20 2024-05-27T00:27:16.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc099800640 1 -- 172.21.15.98:0/4149258909 reap_dead start 2024-05-27T00:27:16.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc098e00640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc07c005bd0 0x7fc07c005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:27:16.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.723+0000 7fc093e00640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc07c006ea0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc08000c160 tx=0x7fc08001cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc6e0bf567fb77f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:16.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080007c70 con 0x7fc07c006a20 2024-05-27T00:27:16.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0800a9020 con 0x7fc07c006a20 2024-05-27T00:27:16.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc093400640 1 -- 172.21.15.98:0/4149258909 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080005b00 con 0x7fc07c006a20 2024-05-27T00:27:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 -- 172.21.15.98:0/4149258909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 msgr2=0x7fc07c006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc07c006ea0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc08000c160 tx=0x7fc08001cad0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 -- 172.21.15.98:0/4149258909 shutdown_connections 2024-05-27T00:27:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 --2- 172.21.15.98:0/4149258909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc07c006ea0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 -- 172.21.15.98:0/4149258909 >> 172.21.15.98:0/4149258909 conn(0x7fc094067530 msgr2=0x7fc09406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 -- 172.21.15.98:0/4149258909 shutdown_connections 2024-05-27T00:27:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 -- 172.21.15.98:0/4149258909 wait complete. 2024-05-27T00:27:16.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 Processor -- start 2024-05-27T00:27:16.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 -- start start 2024-05-27T00:27:16.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc0941564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:16.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc094155f80 0x7fc09414ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:16.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0941eea40 0x7fc09414aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:16.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0941515b0 con 0x7fc07c006a20 2024-05-27T00:27:16.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc094151430 con 0x7fc0941eea40 2024-05-27T00:27:16.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc09b800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc094151730 con 0x7fc094155f80 2024-05-27T00:27:16.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc098e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc0941564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:16.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc098e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc0941564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38028/0 (socket says 172.21.15.98:38028) 2024-05-27T00:27:16.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc098e00640 1 -- 172.21.15.98:0/3684806850 learned_addr learned my addr 172.21.15.98:0/3684806850 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:16.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc093e00640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc094155f80 0x7fc09414ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:16.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.727+0000 7fc099800640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0941eea40 0x7fc09414aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:16.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc098e00640 1 -- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc094155f80 msgr2=0x7fc09414ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:16.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc098e00640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc094155f80 0x7fc09414ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc098e00640 1 -- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0941eea40 msgr2=0x7fc09414aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:16.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc098e00640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0941eea40 0x7fc09414aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc098e00640 1 -- 172.21.15.98:0/3684806850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc094152210 con 0x7fc07c006a20 2024-05-27T00:27:16.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc099800640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0941eea40 0x7fc09414aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:27:16.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc098e00640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc0941564d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc08800e580 tx=0x7fc08800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:16.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc090c00640 1 -- 172.21.15.98:0/3684806850 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0880052e0 con 0x7fc07c006a20 2024-05-27T00:27:16.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc090c00640 1 -- 172.21.15.98:0/3684806850 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0880067e0 con 0x7fc07c006a20 2024-05-27T00:27:16.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc09b800640 1 -- 172.21.15.98:0/3684806850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0941f20d0 con 0x7fc07c006a20 2024-05-27T00:27:16.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc090c00640 1 -- 172.21.15.98:0/3684806850 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088005480 con 0x7fc07c006a20 2024-05-27T00:27:16.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc09b800640 1 -- 172.21.15.98:0/3684806850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0941f2690 con 0x7fc07c006a20 2024-05-27T00:27:16.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc09b800640 1 -- 172.21.15.98:0/3684806850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc094075cb0 con 0x7fc07c006a20 2024-05-27T00:27:16.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.731+0000 7fc090c00640 1 -- 172.21.15.98:0/3684806850 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc088006bd0 con 0x7fc07c006a20 2024-05-27T00:27:16.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.735+0000 7fc090c00640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0740678f0 0x7fc074069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:16.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.735+0000 7fc090c00640 1 -- 172.21.15.98:0/3684806850 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7fc088085ad0 con 0x7fc07c006a20 2024-05-27T00:27:16.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.735+0000 7fc093e00640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0740678f0 0x7fc074069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:16.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.735+0000 7fc093e00640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0740678f0 0x7fc074069db0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc0800aa000 tx=0x7fc0940c19a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:16.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.739+0000 7fc090c00640 1 -- 172.21.15.98:0/3684806850 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc088056250 con 0x7fc07c006a20 2024-05-27T00:27:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.927+0000 7fc09b800640 1 -- 172.21.15.98:0/3684806850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc094158510 con 0x7fc07c006a20 2024-05-27T00:27:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.927+0000 7fc090c00640 1 -- 172.21.15.98:0/3684806850 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6010 (secure 0 0 0) 0x7fc088055b50 con 0x7fc07c006a20 2024-05-27T00:27:16.933 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:16.933 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:10.669108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 49 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 -- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0740678f0 msgr2=0x7fc074069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0740678f0 0x7fc074069db0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc0800aa000 tx=0x7fc0940c19a0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 -- 172.21.15.98:0/3684806850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 msgr2=0x7fc0941564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc0941564d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc08800e580 tx=0x7fc08800ea40 comp rx=0 tx=0).stop 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 -- 172.21.15.98:0/3684806850 shutdown_connections 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc0740678f0 0x7fc074069db0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0941eea40 0x7fc09414aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc094155f80 0x7fc09414ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 --2- 172.21.15.98:0/3684806850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc07c006a20 0x7fc0941564d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 -- 172.21.15.98:0/3684806850 >> 172.21.15.98:0/3684806850 conn(0x7fc094067530 msgr2=0x7fc0940c8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:16.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 -- 172.21.15.98:0/3684806850 shutdown_connections 2024-05-27T00:27:16.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:16.931+0000 7fc09b800640 1 -- 172.21.15.98:0/3684806850 wait complete. 2024-05-27T00:27:16.953 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:27:18.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.087+0000 7f36a9200640 1 Processor -- start 2024-05-27T00:27:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.087+0000 7f36a9200640 1 -- start start 2024-05-27T00:27:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.087+0000 7f36a9200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f36a4070fe0 0x7f36a4154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.087+0000 7f36a9200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40cb570 0x7f36a4154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.087+0000 7f36a9200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36a415a950 con 0x7f36a4070c00 2024-05-27T00:27:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.087+0000 7f36a9200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f36a415e060 con 0x7f36a4070fe0 2024-05-27T00:27:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.087+0000 7f36a9200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f36a415e340 con 0x7f36a40cb570 2024-05-27T00:27:19.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.087+0000 7f36a1e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f36a4070c00 0x7f36a40c8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54920/0 (socket says 172.21.15.98:54920) 2024-05-27T00:27:19.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.087+0000 7f36a1e00640 1 -- 172.21.15.98:0/3607241683 learned_addr learned my addr 172.21.15.98:0/3607241683 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:19.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f369be00640 1 --2- 172.21.15.98:0/3607241683 >> v2:172.21.15.204:3300/0 conn(0x7f36a4070fe0 0x7f36a4154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a2800640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40cb570 0x7f36a4154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1900846505 0 0) 0x7f36a415a950 con 0x7f36a4070c00 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3688003530 con 0x7f36a4070c00 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1682069142 0 0) 0x7f3688003530 con 0x7f36a4070c00 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40cb570 msgr2=0x7f36a4154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40cb570 0x7f36a4154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 >> v2:172.21.15.204:3300/0 conn(0x7f36a4070fe0 msgr2=0x7f36a4154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 --2- 172.21.15.98:0/3607241683 >> v2:172.21.15.204:3300/0 conn(0x7f36a4070fe0 0x7f36a4154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a415e620 con 0x7f36a4070c00 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f3694001680 con 0x7f36a4070c00 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 >> v1:172.21.15.98:6789/0 conn(0x7f36a4070c00 legacy=0x7f36a40c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36880039c0 0x7f3688003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:19.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40cb570 0x7f3688004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3688004d50 0x7f36880051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36a415e060 con 0x7f36880039c0 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f36a415e340 con 0x7f3688004d50 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f368800b9d0 con 0x7f36a40cb570 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1e00640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36880039c0 0x7f3688003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a2800640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3688004d50 0x7f36880051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f369be00640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40cb570 0x7f3688004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1e00640 1 -- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40cb570 msgr2=0x7f3688004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1e00640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40cb570 0x7f3688004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1e00640 1 -- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3688004d50 msgr2=0x7f36880051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1e00640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3688004d50 0x7f36880051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.091+0000 7f36a1e00640 1 -- 172.21.15.98:0/3607241683 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3688003530 con 0x7f36880039c0 2024-05-27T00:27:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a1e00640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36880039c0 0x7f3688003de0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f369400c240 tx=0x7f369401f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361fb095dc86882d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3694007d60 con 0x7f36880039c0 2024-05-27T00:27:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3694005b00 con 0x7f36880039c0 2024-05-27T00:27:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a1400640 1 -- 172.21.15.98:0/3607241683 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3694007070 con 0x7f36880039c0 2024-05-27T00:27:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a9200640 1 -- 172.21.15.98:0/3607241683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36880039c0 msgr2=0x7f3688003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a9200640 1 --2- 172.21.15.98:0/3607241683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36880039c0 0x7f3688003de0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f369400c240 tx=0x7f369401f430 comp rx=0 tx=0).stop 2024-05-27T00:27:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a2800640 1 -- 172.21.15.98:0/3607241683 reap_dead start 2024-05-27T00:27:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a9200640 1 -- 172.21.15.98:0/3607241683 shutdown_connections 2024-05-27T00:27:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a9200640 1 -- 172.21.15.98:0/3607241683 >> 172.21.15.98:0/3607241683 conn(0x7f36a4072f90 msgr2=0x7f36a40733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a9200640 1 -- 172.21.15.98:0/3607241683 shutdown_connections 2024-05-27T00:27:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a9200640 1 -- 172.21.15.98:0/3607241683 wait complete. 2024-05-27T00:27:19.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a9200640 1 Processor -- start 2024-05-27T00:27:19.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a9200640 1 -- start start 2024-05-27T00:27:19.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.095+0000 7f36a9200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36880039c0 0x7f36a414a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:19.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a9200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40b3b20 0x7f36a414ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:19.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a9200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a414f480 0x7f36a415c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:19.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a9200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36a40c9110 con 0x7f36a414f480 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a9200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f36a415ebe0 con 0x7f36880039c0 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a9200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f36a4154cc0 con 0x7f36a40b3b20 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a2800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a414f480 0x7f36a415c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a2800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a414f480 0x7f36a415c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38050/0 (socket says 172.21.15.98:38050) 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a2800640 1 -- 172.21.15.98:0/3289155007 learned_addr learned my addr 172.21.15.98:0/3289155007 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f369be00640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40b3b20 0x7f36a414ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a1e00640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36880039c0 0x7f36a414a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a2800640 1 -- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40b3b20 msgr2=0x7f36a414ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a2800640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40b3b20 0x7f36a414ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a2800640 1 -- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36880039c0 msgr2=0x7f36a414a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a2800640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36880039c0 0x7f36a414a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a2800640 1 -- 172.21.15.98:0/3289155007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a40c5e50 con 0x7f36a414f480 2024-05-27T00:27:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a2800640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a414f480 0x7f36a415c170 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f36a40c86c0 tx=0x7f369c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f369a000640 1 -- 172.21.15.98:0/3289155007 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c011e10 con 0x7f36a414f480 2024-05-27T00:27:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f369a000640 1 -- 172.21.15.98:0/3289155007 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369c01b070 con 0x7f36a414f480 2024-05-27T00:27:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a9200640 1 -- 172.21.15.98:0/3289155007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a40c7850 con 0x7f36a414f480 2024-05-27T00:27:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f369a000640 1 -- 172.21.15.98:0/3289155007 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c0039a0 con 0x7f36a414f480 2024-05-27T00:27:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a9200640 1 -- 172.21.15.98:0/3289155007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a40c7290 con 0x7f36a414f480 2024-05-27T00:27:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f36a9200640 1 -- 172.21.15.98:0/3289155007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a4075cb0 con 0x7f36a414f480 2024-05-27T00:27:19.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.099+0000 7f369a000640 1 -- 172.21.15.98:0/3289155007 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f369c003b60 con 0x7f36a414f480 2024-05-27T00:27:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.103+0000 7f369a000640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36800678c0 0x7f3680069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.103+0000 7f369a000640 1 -- 172.21.15.98:0/3289155007 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7f369c0867d0 con 0x7f36a414f480 2024-05-27T00:27:19.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.103+0000 7f36a1e00640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36800678c0 0x7f3680069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.103+0000 7f36a1e00640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36800678c0 0x7f3680069d80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f36940ae000 tx=0x7f3694001ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:19.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.103+0000 7f369a000640 1 -- 172.21.15.98:0/3289155007 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f36a40c7850 con 0x7f36a414f480 2024-05-27T00:27:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.295+0000 7f36a9200640 1 -- 172.21.15.98:0/3289155007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36a414d9a0 con 0x7f36a414f480 2024-05-27T00:27:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.295+0000 7f369a000640 1 -- 172.21.15.98:0/3289155007 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6007 (secure 0 0 0) 0x7f369c056f40 con 0x7f36a414f480 2024-05-27T00:27:19.299 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:19.299 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:16.516746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 50 2024-05-27T00:27:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 -- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36800678c0 msgr2=0x7f3680069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:19.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36800678c0 0x7f3680069d80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f36940ae000 tx=0x7f3694001ac0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 -- 172.21.15.98:0/3289155007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a414f480 msgr2=0x7f36a415c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a414f480 0x7f36a415c170 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f36a40c86c0 tx=0x7f369c0115a0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 -- 172.21.15.98:0/3289155007 shutdown_connections 2024-05-27T00:27:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36a414f480 0x7f36a415c170 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36a40b3b20 0x7f36a414ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f36800678c0 0x7f3680069d80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 --2- 172.21.15.98:0/3289155007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36880039c0 0x7f36a414a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 -- 172.21.15.98:0/3289155007 >> 172.21.15.98:0/3289155007 conn(0x7f36a4072f90 msgr2=0x7f36a4073370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 -- 172.21.15.98:0/3289155007 shutdown_connections 2024-05-27T00:27:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:19.299+0000 7f36a9200640 1 -- 172.21.15.98:0/3289155007 wait complete. 2024-05-27T00:27:19.320 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:27:21.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:21.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f000aa00640 1 Processor -- start 2024-05-27T00:27:21.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f000aa00640 1 -- start start 2024-05-27T00:27:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f000aa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00040c7f00 0x7f00040c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f000aa00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f00040c9440 0x7f000415e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f000aa00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0004071810 con 0x7f0004154aa0 2024-05-27T00:27:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f000aa00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0004154e80 con 0x7f00040c7f00 2024-05-27T00:27:21.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f000aa00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f00040c9870 con 0x7f00040c9440 2024-05-27T00:27:21.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f0008c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00040c7f00 0x7f00040c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:21.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f0009600640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0004154aa0 0x7f00040c6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54930/0 (socket says 172.21.15.98:54930) 2024-05-27T00:27:21.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f0009600640 1 -- 172.21.15.98:0/2955590824 learned_addr learned my addr 172.21.15.98:0/2955590824 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:21.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f000a000640 1 --2- 172.21.15.98:0/2955590824 >> v2:172.21.15.204:3300/0 conn(0x7f00040c9440 0x7f000415e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:21.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f0008c00640 1 -- 172.21.15.98:0/2955590824 >> v2:172.21.15.204:3300/0 conn(0x7f00040c9440 msgr2=0x7f000415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:21.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f0008c00640 1 --2- 172.21.15.98:0/2955590824 >> v2:172.21.15.204:3300/0 conn(0x7f00040c9440 0x7f000415e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:21.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f0008c00640 1 -- 172.21.15.98:0/2955590824 >> v1:172.21.15.98:6789/0 conn(0x7f0004154aa0 legacy=0x7f00040c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:21.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f0008c00640 1 -- 172.21.15.98:0/2955590824 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0004072090 con 0x7f00040c7f00 2024-05-27T00:27:21.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.455+0000 7f0003e00640 1 -- 172.21.15.98:0/2955590824 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2975474054 0 0) 0x7f0004071810 con 0x7f0004154aa0 2024-05-27T00:27:21.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f0008c00640 1 --2- 172.21.15.98:0/2955590824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 0x7f00040c8f00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0004071c50 tx=0x7efff001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ca7994d6bf6b7b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:21.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f0003e00640 1 -- 172.21.15.98:0/2955590824 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff00b5070 con 0x7f00040c7f00 2024-05-27T00:27:21.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f0003e00640 1 -- 172.21.15.98:0/2955590824 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efff00a8970 con 0x7f00040c7f00 2024-05-27T00:27:21.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f0003e00640 1 -- 172.21.15.98:0/2955590824 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff00a8d40 con 0x7f00040c7f00 2024-05-27T00:27:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f000aa00640 1 -- 172.21.15.98:0/2955590824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 msgr2=0x7f00040c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f000aa00640 1 --2- 172.21.15.98:0/2955590824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 0x7f00040c8f00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0004071c50 tx=0x7efff001ec70 comp rx=0 tx=0).stop 2024-05-27T00:27:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f000aa00640 1 -- 172.21.15.98:0/2955590824 shutdown_connections 2024-05-27T00:27:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f000aa00640 1 --2- 172.21.15.98:0/2955590824 >> v2:172.21.15.204:3300/0 conn(0x7f00040c9440 0x7f000415e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f000aa00640 1 --2- 172.21.15.98:0/2955590824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 0x7f00040c8f00 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f000aa00640 1 -- 172.21.15.98:0/2955590824 >> 172.21.15.98:0/2955590824 conn(0x7f000406b470 msgr2=0x7f000406b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f000aa00640 1 -- 172.21.15.98:0/2955590824 shutdown_connections 2024-05-27T00:27:21.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.459+0000 7f000aa00640 1 -- 172.21.15.98:0/2955590824 wait complete. 2024-05-27T00:27:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000aa00640 1 Processor -- start 2024-05-27T00:27:21.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000aa00640 1 -- start start 2024-05-27T00:27:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 0x7f00040cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f00040c9440 0x7f00041e3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0004154aa0 0x7f00041e3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000aa00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0004072910 con 0x7f00040c7f00 2024-05-27T00:27:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0004072630 con 0x7f0004154aa0 2024-05-27T00:27:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0004072bf0 con 0x7f00040c9440 2024-05-27T00:27:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0009600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 0x7f00040cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0009600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 0x7f00040cb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38068/0 (socket says 172.21.15.98:38068) 2024-05-27T00:27:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0009600640 1 -- 172.21.15.98:0/3866246172 learned_addr learned my addr 172.21.15.98:0/3866246172 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:21.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000a000640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0004154aa0 0x7f00041e3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:21.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0008c00640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f00040c9440 0x7f00041e3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:21.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0009600640 1 -- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f00040c9440 msgr2=0x7f00041e3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0009600640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f00040c9440 0x7f00041e3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0009600640 1 -- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0004154aa0 msgr2=0x7f00041e3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0009600640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0004154aa0 0x7f00041e3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0009600640 1 -- 172.21.15.98:0/3866246172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0004155b40 con 0x7f00040c7f00 2024-05-27T00:27:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0009600640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 0x7f00040cb6d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efff800c4b0 tx=0x7efff800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0001600640 1 -- 172.21.15.98:0/3866246172 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff8018070 con 0x7f00040c7f00 2024-05-27T00:27:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0001600640 1 -- 172.21.15.98:0/3866246172 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efff8007500 con 0x7f00040c7f00 2024-05-27T00:27:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000aa00640 1 -- 172.21.15.98:0/3866246172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00041f0180 con 0x7f00040c7f00 2024-05-27T00:27:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f0001600640 1 -- 172.21.15.98:0/3866246172 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff8004a30 con 0x7f00040c7f00 2024-05-27T00:27:21.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.463+0000 7f000aa00640 1 -- 172.21.15.98:0/3866246172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00041f0740 con 0x7f00040c7f00 2024-05-27T00:27:21.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.467+0000 7f000aa00640 1 -- 172.21.15.98:0/3866246172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0004075c40 con 0x7f00040c7f00 2024-05-27T00:27:21.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.467+0000 7f0001600640 1 -- 172.21.15.98:0/3866246172 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efff801b070 con 0x7f00040c7f00 2024-05-27T00:27:21.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.467+0000 7f0001600640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7effe40678f0 0x7effe4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:21.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.467+0000 7f0001600640 1 -- 172.21.15.98:0/3866246172 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7efff8085350 con 0x7f00040c7f00 2024-05-27T00:27:21.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.467+0000 7f0008c00640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7effe40678f0 0x7effe4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:21.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.467+0000 7f0008c00640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7effe40678f0 0x7effe4069db0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efff00b2000 tx=0x7efff00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:21.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.471+0000 7f0001600640 1 -- 172.21.15.98:0/3866246172 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7efff80553e0 con 0x7f00040c7f00 2024-05-27T00:27:21.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 -- 172.21.15.98:0/3866246172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00040c7890 con 0x7f00040c7f00 2024-05-27T00:27:21.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f0001600640 1 -- 172.21.15.98:0/3866246172 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6007 (secure 0 0 0) 0x7efff80596e0 con 0x7f00040c7f00 2024-05-27T00:27:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:21.667 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:16.516746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:21.667 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 50 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 -- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7effe40678f0 msgr2=0x7effe4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7effe40678f0 0x7effe4069db0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efff00b2000 tx=0x7efff00aa070 comp rx=0 tx=0).stop 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 -- 172.21.15.98:0/3866246172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 msgr2=0x7f00040cb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 0x7f00040cb6d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efff800c4b0 tx=0x7efff800c970 comp rx=0 tx=0).stop 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 -- 172.21.15.98:0/3866246172 shutdown_connections 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7effe40678f0 0x7effe4069db0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0004154aa0 0x7f00041e3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f00040c9440 0x7f00041e3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 --2- 172.21.15.98:0/3866246172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00040c7f00 0x7f00040cb6d0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 -- 172.21.15.98:0/3866246172 >> 172.21.15.98:0/3866246172 conn(0x7f000406b470 msgr2=0x7f00041522d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.663+0000 7f000aa00640 1 -- 172.21.15.98:0/3866246172 shutdown_connections 2024-05-27T00:27:21.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:21.667+0000 7f000aa00640 1 -- 172.21.15.98:0/3866246172 wait complete. 2024-05-27T00:27:21.688 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:27:23.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:23.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2eb400640 1 Processor -- start 2024-05-27T00:27:23.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2eb400640 1 -- start start 2024-05-27T00:27:23.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2eb400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd2e4158010 0x7fd2e40ca480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:23.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2eb400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2e41583f0 con 0x7fd2e415def0 2024-05-27T00:27:23.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2eb400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2e40c94b0 con 0x7fd2e40c32d0 2024-05-27T00:27:23.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2eb400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd2e406f660 con 0x7fd2e4158010 2024-05-27T00:27:23.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2e8a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd2e4158010 0x7fd2e40ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:23.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2e9400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd2e415def0 0x7fd2e415e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49046/0 (socket says 172.21.15.98:49046) 2024-05-27T00:27:23.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2e9400640 1 -- 172.21.15.98:0/2615285873 learned_addr learned my addr 172.21.15.98:0/2615285873 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:23.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2e3400640 1 -- 172.21.15.98:0/2615285873 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 290903762 0 0) 0x7fd2e41583f0 con 0x7fd2e415def0 2024-05-27T00:27:23.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2e8a00640 1 -- 172.21.15.98:0/2615285873 >> v1:172.21.15.204:6789/0 conn(0x7fd2e40c32d0 legacy=0x7fd2e406faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:23.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2e8a00640 1 -- 172.21.15.98:0/2615285873 >> v1:172.21.15.98:6789/0 conn(0x7fd2e415def0 legacy=0x7fd2e415e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:23.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.823+0000 7fd2e8a00640 1 -- 172.21.15.98:0/2615285873 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e40c9790 con 0x7fd2e4158010 2024-05-27T00:27:23.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2e8a00640 1 --2- 172.21.15.98:0/2615285873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 0x7fd2e40ca480 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80097c0 tx=0x7fd2d801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f62dcc4298cc026 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:23.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2e3400640 1 -- 172.21.15.98:0/2615285873 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80b1070 con 0x7fd2e4158010 2024-05-27T00:27:23.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2e3400640 1 -- 172.21.15.98:0/2615285873 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2d80ac410 con 0x7fd2e4158010 2024-05-27T00:27:23.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2e3400640 1 -- 172.21.15.98:0/2615285873 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80ac7e0 con 0x7fd2e4158010 2024-05-27T00:27:23.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2eb400640 1 -- 172.21.15.98:0/2615285873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 msgr2=0x7fd2e40ca480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:23.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2eb400640 1 --2- 172.21.15.98:0/2615285873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 0x7fd2e40ca480 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80097c0 tx=0x7fd2d801c4a0 comp rx=0 tx=0).stop 2024-05-27T00:27:23.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2eb400640 1 -- 172.21.15.98:0/2615285873 shutdown_connections 2024-05-27T00:27:23.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2eb400640 1 --2- 172.21.15.98:0/2615285873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 0x7fd2e40ca480 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:23.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2eb400640 1 -- 172.21.15.98:0/2615285873 >> 172.21.15.98:0/2615285873 conn(0x7fd2e40ce000 msgr2=0x7fd2e40ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:23.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2eb400640 1 -- 172.21.15.98:0/2615285873 shutdown_connections 2024-05-27T00:27:23.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.827+0000 7fd2eb400640 1 -- 172.21.15.98:0/2615285873 wait complete. 2024-05-27T00:27:23.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2eb400640 1 Processor -- start 2024-05-27T00:27:23.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2eb400640 1 -- start start 2024-05-27T00:27:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2eb400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2e40c32d0 0x7fd2e40b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2eb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 0x7fd2e40c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2eb400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2e415def0 0x7fd2e414dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2eb400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e4070380 con 0x7fd2e4158010 2024-05-27T00:27:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2eb400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e4070200 con 0x7fd2e415def0 2024-05-27T00:27:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2eb400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd2e40c5ce0 con 0x7fd2e40c32d0 2024-05-27T00:27:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 0x7fd2e40c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 0x7fd2e40c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53782/0 (socket says 172.21.15.98:53782) 2024-05-27T00:27:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e3e00640 1 -- 172.21.15.98:0/2553515061 learned_addr learned my addr 172.21.15.98:0/2553515061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e9400640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2e415def0 0x7fd2e414dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e8a00640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2e40c32d0 0x7fd2e40b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:23.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e3e00640 1 -- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2e40c32d0 msgr2=0x7fd2e40b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:23.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e3e00640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2e40c32d0 0x7fd2e40b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:23.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e3e00640 1 -- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2e415def0 msgr2=0x7fd2e414dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:23.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e3e00640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2e415def0 0x7fd2e414dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:23.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e3e00640 1 -- 172.21.15.98:0/2553515061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e40b3130 con 0x7fd2e4158010 2024-05-27T00:27:23.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e3e00640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 0x7fd2e40c7980 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000c3f0 tx=0x7fd2d000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:23.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e0c00640 1 -- 172.21.15.98:0/2553515061 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0017070 con 0x7fd2e4158010 2024-05-27T00:27:23.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e0c00640 1 -- 172.21.15.98:0/2553515061 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2d00052d0 con 0x7fd2e4158010 2024-05-27T00:27:23.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2e0c00640 1 -- 172.21.15.98:0/2553515061 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0003840 con 0x7fd2e4158010 2024-05-27T00:27:23.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2eb400640 1 -- 172.21.15.98:0/2553515061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e41f6a00 con 0x7fd2e4158010 2024-05-27T00:27:23.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.831+0000 7fd2eb400640 1 -- 172.21.15.98:0/2553515061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e41f6fc0 con 0x7fd2e4158010 2024-05-27T00:27:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.835+0000 7fd2eb400640 1 -- 172.21.15.98:0/2553515061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e4075cb0 con 0x7fd2e4158010 2024-05-27T00:27:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.835+0000 7fd2e0c00640 1 -- 172.21.15.98:0/2553515061 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd2d001a070 con 0x7fd2e4158010 2024-05-27T00:27:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.835+0000 7fd2e0c00640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2c40678c0 0x7fd2c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.835+0000 7fd2e0c00640 1 -- 172.21.15.98:0/2553515061 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7fd2d0085300 con 0x7fd2e4158010 2024-05-27T00:27:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.835+0000 7fd2e8a00640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2c40678c0 0x7fd2c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.839+0000 7fd2e8a00640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2c40678c0 0x7fd2c4069d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80ae000 tx=0x7fd2d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:23.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:23.839+0000 7fd2e0c00640 1 -- 172.21.15.98:0/2553515061 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd2d0005480 con 0x7fd2e4158010 2024-05-27T00:27:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.027+0000 7fd2eb400640 1 -- 172.21.15.98:0/2553515061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2e40c4e50 con 0x7fd2e4158010 2024-05-27T00:27:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2e0c00640 1 -- 172.21.15.98:0/2553515061 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6014 (secure 0 0 0) 0x7fd2d0055430 con 0x7fd2e4158010 2024-05-27T00:27:24.034 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:24.034 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:23.529003+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5802,"qdb_cluster":[5802]},"id":1}]} 2024-05-27T00:27:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 51 2024-05-27T00:27:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 -- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2c40678c0 msgr2=0x7fd2c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2c40678c0 0x7fd2c4069d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80ae000 tx=0x7fd2d8008040 comp rx=0 tx=0).stop 2024-05-27T00:27:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 -- 172.21.15.98:0/2553515061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 msgr2=0x7fd2e40c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 0x7fd2e40c7980 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000c3f0 tx=0x7fd2d000c8b0 comp rx=0 tx=0).stop 2024-05-27T00:27:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 -- 172.21.15.98:0/2553515061 shutdown_connections 2024-05-27T00:27:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd2c40678c0 0x7fd2c4069d80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2e415def0 0x7fd2e414dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e4158010 0x7fd2e40c7980 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 --2- 172.21.15.98:0/2553515061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd2e40c32d0 0x7fd2e40b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 -- 172.21.15.98:0/2553515061 >> 172.21.15.98:0/2553515061 conn(0x7fd2e40ce000 msgr2=0x7fd2e41556b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:24.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 -- 172.21.15.98:0/2553515061 shutdown_connections 2024-05-27T00:27:24.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:24.031+0000 7fd2eb400640 1 -- 172.21.15.98:0/2553515061 wait complete. 2024-05-27T00:27:24.054 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=5667 2024-05-27T00:27:24.055 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2024-05-27T00:27:30.660 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 66% (207/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2024-05-27T00:27:30.795 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at cb7f1c9c752 v0.46 2024-05-27T00:27:30.798 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.46 2024-05-27T00:27:30.813 INFO:tasks.workunit.client.0.smithi098.stdout:v0.47 2024-05-27T00:27:30.814 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 47 -eq 48 ] 2024-05-27T00:27:30.814 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.47 2024-05-27T00:27:30.814 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.47 2024-05-27T00:27:30.814 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.47 2024-05-27T00:27:31.055 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-05-27T00:27:31.056 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-27T00:27:31.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-27T00:27:31.059 INFO:tasks.ceph.mds.a:Started 2024-05-27T00:27:31.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:31.193 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2024-05-27T00:27:31.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.255+0000 7f6f57e00640 1 Processor -- start 2024-05-27T00:27:31.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.255+0000 7f6f57e00640 1 -- start start 2024-05-27T00:27:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f57e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f6f580cc4f0 0x7f6f580c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f57e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6f580c67c0 0x7f6f580cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f57e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f580629d0 0x7f6f5815bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f57e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6f580c6330 con 0x7f6f580c67c0 2024-05-27T00:27:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f57e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f6f580c6e60 con 0x7f6f580cc4f0 2024-05-27T00:27:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f57e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6f5815c0a0 con 0x7f6f580629d0 2024-05-27T00:27:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f4fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6f580c67c0 0x7f6f580cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f4fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6f580c67c0 0x7f6f580cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53794/0 (socket says 172.21.15.98:53794) 2024-05-27T00:27:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f4fe00640 1 -- 172.21.15.98:0/252749753 learned_addr learned my addr 172.21.15.98:0/252749753 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f56a00640 1 --2- 172.21.15.98:0/252749753 >> v2:172.21.15.204:3300/0 conn(0x7f6f580cc4f0 0x7f6f580c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f57400640 1 --2- 172.21.15.98:0/252749753 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f580629d0 0x7f6f5815bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f56a00640 1 -- 172.21.15.98:0/252749753 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f580629d0 msgr2=0x7f6f5815bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f56a00640 1 --2- 172.21.15.98:0/252749753 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f580629d0 0x7f6f5815bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f56a00640 1 -- 172.21.15.98:0/252749753 >> v2:172.21.15.98:3300/0 conn(0x7f6f580c67c0 msgr2=0x7f6f580cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f56a00640 1 --2- 172.21.15.98:0/252749753 >> v2:172.21.15.98:3300/0 conn(0x7f6f580c67c0 0x7f6f580cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f56a00640 1 -- 172.21.15.98:0/252749753 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f580c70b0 con 0x7f6f580cc4f0 2024-05-27T00:27:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.259+0000 7f6f4fe00640 1 --2- 172.21.15.98:0/252749753 >> v2:172.21.15.98:3300/0 conn(0x7f6f580c67c0 0x7f6f580cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:27:31.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f56a00640 1 --2- 172.21.15.98:0/252749753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f580cc4f0 0x7f6f580c7720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f580c3930 tx=0x7f6f4801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=789d494359f67bfa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:31.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f56000640 1 -- 172.21.15.98:0/252749753 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f480b0070 con 0x7f6f580cc4f0 2024-05-27T00:27:31.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f56000640 1 -- 172.21.15.98:0/252749753 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f480ab600 con 0x7f6f580cc4f0 2024-05-27T00:27:31.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f56000640 1 -- 172.21.15.98:0/252749753 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f480ab9d0 con 0x7f6f580cc4f0 2024-05-27T00:27:31.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f57e00640 1 -- 172.21.15.98:0/252749753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f580cc4f0 msgr2=0x7f6f580c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:31.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f57e00640 1 --2- 172.21.15.98:0/252749753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f580cc4f0 0x7f6f580c7720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f580c3930 tx=0x7f6f4801c480 comp rx=0 tx=0).stop 2024-05-27T00:27:31.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f57e00640 1 -- 172.21.15.98:0/252749753 shutdown_connections 2024-05-27T00:27:31.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f57e00640 1 --2- 172.21.15.98:0/252749753 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f580629d0 0x7f6f5815bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f57e00640 1 --2- 172.21.15.98:0/252749753 >> v2:172.21.15.98:3300/0 conn(0x7f6f580c67c0 0x7f6f580cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f57e00640 1 --2- 172.21.15.98:0/252749753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f580cc4f0 0x7f6f580c7720 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f57e00640 1 -- 172.21.15.98:0/252749753 >> 172.21.15.98:0/252749753 conn(0x7f6f5806dd00 msgr2=0x7f6f5806b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:31.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f57e00640 1 -- 172.21.15.98:0/252749753 shutdown_connections 2024-05-27T00:27:31.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.263+0000 7f6f57e00640 1 -- 172.21.15.98:0/252749753 wait complete. 2024-05-27T00:27:31.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57e00640 1 Processor -- start 2024-05-27T00:27:31.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57e00640 1 -- start start 2024-05-27T00:27:31.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f580629d0 0x7f6f581f4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:31.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f580c67c0 0x7f6f5815b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:31.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f580cc4f0 0x7f6f58145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f580c9980 con 0x7f6f580c67c0 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6f580c9800 con 0x7f6f580cc4f0 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6f580c9b00 con 0x7f6f580629d0 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f580c67c0 0x7f6f5815b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f580c67c0 0x7f6f5815b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53806/0 (socket says 172.21.15.98:53806) 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4fe00640 1 -- 172.21.15.98:0/35751603 learned_addr learned my addr 172.21.15.98:0/35751603 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f56a00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f580629d0 0x7f6f581f4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57400640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f580cc4f0 0x7f6f58145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4fe00640 1 -- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f580629d0 msgr2=0x7f6f581f4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4fe00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f580629d0 0x7f6f581f4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4fe00640 1 -- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f580cc4f0 msgr2=0x7f6f58145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:31.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4fe00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f580cc4f0 0x7f6f58145ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4fe00640 1 -- 172.21.15.98:0/35751603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f581f7d30 con 0x7f6f580c67c0 2024-05-27T00:27:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4fe00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f580c67c0 0x7f6f5815b370 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000ea00 tx=0x7f6f4000eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4f400640 1 -- 172.21.15.98:0/35751603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f40017070 con 0x7f6f580c67c0 2024-05-27T00:27:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4f400640 1 -- 172.21.15.98:0/35751603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f4000ca60 con 0x7f6f580c67c0 2024-05-27T00:27:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f4f400640 1 -- 172.21.15.98:0/35751603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4000ce10 con 0x7f6f580c67c0 2024-05-27T00:27:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57e00640 1 -- 172.21.15.98:0/35751603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f58162b10 con 0x7f6f580c67c0 2024-05-27T00:27:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.267+0000 7f6f57e00640 1 -- 172.21.15.98:0/35751603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f581f3a60 con 0x7f6f580c67c0 2024-05-27T00:27:31.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.271+0000 7f6f57e00640 1 -- 172.21.15.98:0/35751603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f58075c40 con 0x7f6f580c67c0 2024-05-27T00:27:31.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.271+0000 7f6f4f400640 1 -- 172.21.15.98:0/35751603 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6f4001a070 con 0x7f6f580c67c0 2024-05-27T00:27:31.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.271+0000 7f6f4f400640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6f340678c0 0x7f6f34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:31.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.271+0000 7f6f4f400640 1 -- 172.21.15.98:0/35751603 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7f6f40086300 con 0x7f6f580c67c0 2024-05-27T00:27:31.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.275+0000 7f6f56a00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6f340678c0 0x7f6f34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:31.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.275+0000 7f6f56a00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6f340678c0 0x7f6f34069d80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6f480ad000 tx=0x7f6f48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:31.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.279+0000 7f6f4f400640 1 -- 172.21.15.98:0/35751603 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f6f40056420 con 0x7f6f580c67c0 2024-05-27T00:27:31.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.571+0000 7f6f57e00640 1 -- 172.21.15.98:0/35751603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f580c7500 con 0x7f6f580c67c0 2024-05-27T00:27:31.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f4f400640 1 -- 172.21.15.98:0/35751603 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6839 (secure 0 0 0) 0x7f6f4005a720 con 0x7f6f580c67c0 2024-05-27T00:27:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:31.577 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5943,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:23.529003+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5802,"qdb_cluster":[5802]},"id":1}]} 2024-05-27T00:27:31.577 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 52 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 -- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6f340678c0 msgr2=0x7f6f34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6f340678c0 0x7f6f34069d80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6f480ad000 tx=0x7f6f48008040 comp rx=0 tx=0).stop 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 -- 172.21.15.98:0/35751603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f580c67c0 msgr2=0x7f6f5815b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f580c67c0 0x7f6f5815b370 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000ea00 tx=0x7f6f4000eec0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 -- 172.21.15.98:0/35751603 shutdown_connections 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6f340678c0 0x7f6f34069d80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f580cc4f0 0x7f6f58145ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f580c67c0 0x7f6f5815b370 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 --2- 172.21.15.98:0/35751603 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f580629d0 0x7f6f581f4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 -- 172.21.15.98:0/35751603 >> 172.21.15.98:0/35751603 conn(0x7f6f5806dd00 msgr2=0x7f6f5806b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 -- 172.21.15.98:0/35751603 shutdown_connections 2024-05-27T00:27:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:31.575+0000 7f6f57e00640 1 -- 172.21.15.98:0/35751603 wait complete. 2024-05-27T00:27:31.607 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-05-27T00:27:31.608 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before thrashing 2024-05-27T00:27:32.629 INFO:tasks.ceph.mds.c.smithi098.stderr:2024-05-27T00:27:32.627+0000 7f03c5c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:27:32.892 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 74% (164/219) Updating files: 75% (165/219) Updating files: 76% (167/219) Updating files: 77% (169/219) Updating files: 78% (171/219) Updating files: 79% (174/219) Updating files: 80% (176/219) Updating files: 81% (178/219) Updating files: 82% (180/219) Updating files: 83% (182/219) Updating files: 84% (184/219) Updating files: 85% (187/219) Updating files: 86% (189/219) Updating files: 87% (191/219) Updating files: 88% (193/219) Updating files: 89% (195/219) Updating files: 90% (198/219) Updating files: 91% (200/219) Updating files: 92% (202/219) Updating files: 93% (204/219) Updating files: 94% (206/219) Updating files: 95% (209/219) Updating files: 96% (211/219) Updating files: 97% (213/219) Updating files: 98% (215/219) Updating files: 99% (217/219) Updating files: 100% (219/219) Updating files: 100% (219/219), done. 2024-05-27T00:27:32.934 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at ad663d58c89 v0.47 2024-05-27T00:27:32.937 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.47 2024-05-27T00:27:33.033 INFO:tasks.workunit.client.0.smithi098.stdout:v0.49 2024-05-27T00:27:33.033 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 48 -eq 48 ] 2024-05-27T00:27:33.033 INFO:tasks.workunit.client.0.smithi098.stderr:+ continue 2024-05-27T00:27:33.033 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 49 -eq 48 ] 2024-05-27T00:27:33.033 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.49 2024-05-27T00:27:33.033 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.49 2024-05-27T00:27:33.033 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.49 2024-05-27T00:27:36.093 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 54% (263/481) Updating files: 55% (265/481) Updating files: 56% (270/481) Updating files: 57% (275/481) Updating files: 58% (279/481) Updating files: 59% (284/481) Updating files: 60% (289/481) Updating files: 61% (294/481) Updating files: 62% (299/481) Updating files: 63% (304/481) Updating files: 64% (308/481) Updating files: 65% (313/481) Updating files: 66% (318/481) Updating files: 67% (323/481) Updating files: 68% (328/481) Updating files: 69% (332/481) Updating files: 70% (337/481) Updating files: 71% (342/481) Updating files: 72% (347/481) Updating files: 73% (352/481) Updating files: 74% (356/481) Updating files: 75% (361/481) Updating files: 76% (366/481) Updating files: 77% (371/481) Updating files: 78% (376/481) Updating files: 79% (380/481) Updating files: 80% (385/481) Updating files: 81% (390/481) Updating files: 82% (395/481) Updating files: 83% (400/481) Updating files: 84% (405/481) Updating files: 85% (409/481) Updating files: 86% (414/481) Updating files: 87% (419/481) Updating files: 88% (424/481) Updating files: 89% (429/481) Updating files: 90% (433/481) Updating files: 91% (438/481) Updating files: 92% (443/481) Updating files: 93% (448/481) Updating files: 93% (450/481) Updating files: 94% (453/481) Updating files: 95% (457/481) Updating files: 96% (462/481) Updating files: 97% (467/481) Updating files: 98% (472/481) Updating files: 99% (477/481) Updating files: 100% (481/481) Updating files: 100% (481/481), done. 2024-05-27T00:27:36.133 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at ca6265d0f4d v0.49 2024-05-27T00:27:36.139 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.49 2024-05-27T00:27:36.321 INFO:tasks.workunit.client.0.smithi098.stdout:v0.50 2024-05-27T00:27:36.321 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 50 -eq 48 ] 2024-05-27T00:27:36.321 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.50 2024-05-27T00:27:36.321 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.50 2024-05-27T00:27:36.321 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.50 2024-05-27T00:27:36.964 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at a4428bd0e19 v0.50 2024-05-27T00:27:36.967 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.50 2024-05-27T00:27:37.051 INFO:tasks.workunit.client.0.smithi098.stdout:v0.51 2024-05-27T00:27:37.051 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 51 -eq 48 ] 2024-05-27T00:27:37.051 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.51 2024-05-27T00:27:37.051 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.51 2024-05-27T00:27:37.052 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.51 2024-05-27T00:27:37.844 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at c03ca95d235 v0.51 2024-05-27T00:27:37.847 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.51 2024-05-27T00:27:37.882 INFO:tasks.workunit.client.0.smithi098.stdout:v0.52 2024-05-27T00:27:37.882 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 52 -eq 48 ] 2024-05-27T00:27:37.882 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.52 2024-05-27T00:27:37.882 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.52 2024-05-27T00:27:37.883 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.52 2024-05-27T00:27:39.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:39.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.743+0000 7f5d89800640 1 Processor -- start 2024-05-27T00:27:39.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.743+0000 7f5d89800640 1 -- start start 2024-05-27T00:27:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d89800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d84154690 0x7f5d8415b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d89800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5d840cc290 0x7f5d840c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d89800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d840ccda0 0x7f5d840cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d89800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5d8415b650 con 0x7f5d840ccda0 2024-05-27T00:27:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d89800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f5d840caf40 con 0x7f5d840cc290 2024-05-27T00:27:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d89800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5d840cb190 con 0x7f5d84154690 2024-05-27T00:27:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d88e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d840ccda0 0x7f5d840cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:39.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d88e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d840ccda0 0x7f5d840cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53394/0 (socket says 172.21.15.98:53394) 2024-05-27T00:27:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d88e00640 1 -- 172.21.15.98:0/150159462 learned_addr learned my addr 172.21.15.98:0/150159462 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d83e00640 1 --2- 172.21.15.98:0/150159462 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d84154690 0x7f5d8415b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d83400640 1 --2- 172.21.15.98:0/150159462 >> v2:172.21.15.204:3300/0 conn(0x7f5d840cc290 0x7f5d840c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d88e00640 1 -- 172.21.15.98:0/150159462 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d84154690 msgr2=0x7f5d8415b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:39.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d88e00640 1 --2- 172.21.15.98:0/150159462 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d84154690 0x7f5d8415b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d88e00640 1 -- 172.21.15.98:0/150159462 >> v2:172.21.15.204:3300/0 conn(0x7f5d840cc290 msgr2=0x7f5d840c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d88e00640 1 --2- 172.21.15.98:0/150159462 >> v2:172.21.15.204:3300/0 conn(0x7f5d840cc290 0x7f5d840c4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.747+0000 7f5d88e00640 1 -- 172.21.15.98:0/150159462 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d8415dd20 con 0x7f5d840ccda0 2024-05-27T00:27:39.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d88e00640 1 --2- 172.21.15.98:0/150159462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 0x7f5d840cd1f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5d8408d820 tx=0x7f5d7c01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fdcb437a6052286 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d82a00640 1 -- 172.21.15.98:0/150159462 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c0b4070 con 0x7f5d840ccda0 2024-05-27T00:27:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d82a00640 1 -- 172.21.15.98:0/150159462 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d7c0af400 con 0x7f5d840ccda0 2024-05-27T00:27:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d82a00640 1 -- 172.21.15.98:0/150159462 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c0af7d0 con 0x7f5d840ccda0 2024-05-27T00:27:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d89800640 1 -- 172.21.15.98:0/150159462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 msgr2=0x7f5d840cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d89800640 1 --2- 172.21.15.98:0/150159462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 0x7f5d840cd1f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5d8408d820 tx=0x7f5d7c01f1e0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d89800640 1 -- 172.21.15.98:0/150159462 shutdown_connections 2024-05-27T00:27:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d89800640 1 --2- 172.21.15.98:0/150159462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 0x7f5d840cd1f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d89800640 1 --2- 172.21.15.98:0/150159462 >> v2:172.21.15.204:3300/0 conn(0x7f5d840cc290 0x7f5d840c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d89800640 1 --2- 172.21.15.98:0/150159462 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d84154690 0x7f5d8415b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d89800640 1 -- 172.21.15.98:0/150159462 >> 172.21.15.98:0/150159462 conn(0x7f5d8406c740 msgr2=0x7f5d8406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:39.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d89800640 1 -- 172.21.15.98:0/150159462 shutdown_connections 2024-05-27T00:27:39.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.751+0000 7f5d89800640 1 -- 172.21.15.98:0/150159462 wait complete. 2024-05-27T00:27:39.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d89800640 1 Processor -- start 2024-05-27T00:27:39.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d89800640 1 -- start start 2024-05-27T00:27:39.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d89800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d840cc290 0x7f5d84156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:39.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d89800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 0x7f5d84154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:39.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d89800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d84154690 0x7f5d840c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:39.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d89800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8415e5c0 con 0x7f5d840ccda0 2024-05-27T00:27:39.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d89800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8415e2e0 con 0x7f5d840cc290 2024-05-27T00:27:39.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d89800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5d840cb7e0 con 0x7f5d84154690 2024-05-27T00:27:39.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d83400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 0x7f5d84154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:39.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d83400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 0x7f5d84154f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53402/0 (socket says 172.21.15.98:53402) 2024-05-27T00:27:39.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d83400640 1 -- 172.21.15.98:0/1856918265 learned_addr learned my addr 172.21.15.98:0/1856918265 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:39.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d88e00640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d84154690 0x7f5d840c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d83e00640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d840cc290 0x7f5d84156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d83400640 1 -- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d84154690 msgr2=0x7f5d840c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d83400640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d84154690 0x7f5d840c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d83400640 1 -- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d840cc290 msgr2=0x7f5d84156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d83400640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d840cc290 0x7f5d84156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d83400640 1 -- 172.21.15.98:0/1856918265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d840c89c0 con 0x7f5d840ccda0 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d83400640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 0x7f5d84154f80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000d830 tx=0x7f5d7000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d7be00640 1 -- 172.21.15.98:0/1856918265 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d70018030 con 0x7f5d840ccda0 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d7be00640 1 -- 172.21.15.98:0/1856918265 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d7000be10 con 0x7f5d840ccda0 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d7be00640 1 -- 172.21.15.98:0/1856918265 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d700098b0 con 0x7f5d840ccda0 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d89800640 1 -- 172.21.15.98:0/1856918265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d840c8ca0 con 0x7f5d840ccda0 2024-05-27T00:27:39.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.755+0000 7f5d89800640 1 -- 172.21.15.98:0/1856918265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d840c9260 con 0x7f5d840ccda0 2024-05-27T00:27:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.759+0000 7f5d89800640 1 -- 172.21.15.98:0/1856918265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d84075c40 con 0x7f5d840ccda0 2024-05-27T00:27:39.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.759+0000 7f5d7be00640 1 -- 172.21.15.98:0/1856918265 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5d7001b070 con 0x7f5d840ccda0 2024-05-27T00:27:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.759+0000 7f5d7be00640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5d640678c0 0x7f5d64069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.759+0000 7f5d7be00640 1 -- 172.21.15.98:0/1856918265 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7f5d70086160 con 0x7f5d840ccda0 2024-05-27T00:27:39.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.759+0000 7f5d83e00640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5d640678c0 0x7f5d64069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:39.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.759+0000 7f5d83e00640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5d640678c0 0x7f5d64069d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5d74007a60 tx=0x7f5d74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:39.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.763+0000 7f5d7be00640 1 -- 172.21.15.98:0/1856918265 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f5d700561f0 con 0x7f5d840ccda0 2024-05-27T00:27:39.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 -- 172.21.15.98:0/1856918265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d840cba80 con 0x7f5d840ccda0 2024-05-27T00:27:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d7be00640 1 -- 172.21.15.98:0/1856918265 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6839 (secure 0 0 0) 0x7f5d7005a4f0 con 0x7f5d840ccda0 2024-05-27T00:27:39.959 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:39.959 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5943,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:23.529003+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5802,"qdb_cluster":[5802]},"id":1}]} 2024-05-27T00:27:39.959 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 52 2024-05-27T00:27:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 -- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5d640678c0 msgr2=0x7f5d64069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5d640678c0 0x7f5d64069d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5d74007a60 tx=0x7f5d74008040 comp rx=0 tx=0).stop 2024-05-27T00:27:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 -- 172.21.15.98:0/1856918265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 msgr2=0x7f5d84154f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 0x7f5d84154f80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000d830 tx=0x7f5d7000dcf0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 -- 172.21.15.98:0/1856918265 shutdown_connections 2024-05-27T00:27:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d84154690 0x7f5d840c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d840ccda0 0x7f5d84154f80 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5d640678c0 0x7f5d64069d80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 --2- 172.21.15.98:0/1856918265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d840cc290 0x7f5d84156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.955+0000 7f5d89800640 1 -- 172.21.15.98:0/1856918265 >> 172.21.15.98:0/1856918265 conn(0x7f5d8406c740 msgr2=0x7f5d840ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.959+0000 7f5d89800640 1 -- 172.21.15.98:0/1856918265 shutdown_connections 2024-05-27T00:27:39.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:39.959+0000 7f5d89800640 1 -- 172.21.15.98:0/1856918265 wait complete. 2024-05-27T00:27:39.979 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-05-27T00:27:39.979 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-27T00:27:39.979 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:27:40.012 INFO:tasks.ceph.mds.c:Stopped 2024-05-27T00:27:40.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816f200640 1 Processor -- start 2024-05-27T00:27:40.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816f200640 1 -- start start 2024-05-27T00:27:40.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816f200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f81700c6a60 0x7f8170073340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 0x7f8170158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816f200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8170151f00 0x7f8170152350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816f200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8170152890 con 0x7f8170151f00 2024-05-27T00:27:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816f200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f8170073880 con 0x7f81700c6a60 2024-05-27T00:27:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8170159300 con 0x7f81700c6e40 2024-05-27T00:27:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816e800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8170151f00 0x7f8170152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816e800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8170151f00 0x7f8170152350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53408/0 (socket says 172.21.15.98:53408) 2024-05-27T00:27:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816e800640 1 -- 172.21.15.98:0/4133150938 learned_addr learned my addr 172.21.15.98:0/4133150938 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f8167e00640 1 --2- 172.21.15.98:0/4133150938 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 0x7f8170158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816de00640 1 --2- 172.21.15.98:0/4133150938 >> v2:172.21.15.204:3300/0 conn(0x7f81700c6a60 0x7f8170073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816e800640 1 -- 172.21.15.98:0/4133150938 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 msgr2=0x7f8170158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816e800640 1 --2- 172.21.15.98:0/4133150938 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 0x7f8170158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816e800640 1 -- 172.21.15.98:0/4133150938 >> v2:172.21.15.204:3300/0 conn(0x7f81700c6a60 msgr2=0x7f8170073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816e800640 1 --2- 172.21.15.98:0/4133150938 >> v2:172.21.15.204:3300/0 conn(0x7f81700c6a60 0x7f8170073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816e800640 1 -- 172.21.15.98:0/4133150938 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817015bb10 con 0x7f8170151f00 2024-05-27T00:27:40.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f8167e00640 1 --2- 172.21.15.98:0/4133150938 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 0x7f8170158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:27:40.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.187+0000 7f816de00640 1 --2- 172.21.15.98:0/4133150938 >> v2:172.21.15.204:3300/0 conn(0x7f81700c6a60 0x7f8170073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:27:40.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816e800640 1 --2- 172.21.15.98:0/4133150938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8170151f00 0x7f8170152350 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f817006f8e0 tx=0x7f816801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0b6701f104382f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816d400640 1 -- 172.21.15.98:0/4133150938 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81680a5670 con 0x7f8170151f00 2024-05-27T00:27:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816d400640 1 -- 172.21.15.98:0/4133150938 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81680a5810 con 0x7f8170151f00 2024-05-27T00:27:40.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816d400640 1 -- 172.21.15.98:0/4133150938 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81680a5be0 con 0x7f8170151f00 2024-05-27T00:27:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816f200640 1 -- 172.21.15.98:0/4133150938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8170151f00 msgr2=0x7f8170152350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816f200640 1 --2- 172.21.15.98:0/4133150938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8170151f00 0x7f8170152350 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f817006f8e0 tx=0x7f816801c9f0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816f200640 1 -- 172.21.15.98:0/4133150938 shutdown_connections 2024-05-27T00:27:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816f200640 1 --2- 172.21.15.98:0/4133150938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8170151f00 0x7f8170152350 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816f200640 1 --2- 172.21.15.98:0/4133150938 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 0x7f8170158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816f200640 1 --2- 172.21.15.98:0/4133150938 >> v2:172.21.15.204:3300/0 conn(0x7f81700c6a60 0x7f8170073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816f200640 1 -- 172.21.15.98:0/4133150938 >> 172.21.15.98:0/4133150938 conn(0x7f81700cbe00 msgr2=0x7f81700cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816f200640 1 -- 172.21.15.98:0/4133150938 shutdown_connections 2024-05-27T00:27:40.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.191+0000 7f816f200640 1 -- 172.21.15.98:0/4133150938 wait complete. 2024-05-27T00:27:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816f200640 1 Processor -- start 2024-05-27T00:27:40.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816f200640 1 -- start start 2024-05-27T00:27:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816f200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81700c6a60 0x7f81700c3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 0x7f817014faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170151f00 0x7f81701f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816f200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f81700c76f0 con 0x7f81700c6a60 2024-05-27T00:27:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f81700c7570 con 0x7f8170151f00 2024-05-27T00:27:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f81700c7870 con 0x7f81700c6e40 2024-05-27T00:27:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81700c6a60 0x7f81700c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:40.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81700c6a60 0x7f81700c3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53420/0 (socket says 172.21.15.98:53420) 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816de00640 1 -- 172.21.15.98:0/1834195788 learned_addr learned my addr 172.21.15.98:0/1834195788 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816e800640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170151f00 0x7f81701f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f8167e00640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 0x7f817014faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816de00640 1 -- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 msgr2=0x7f817014faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816de00640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 0x7f817014faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816de00640 1 -- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170151f00 msgr2=0x7f81701f30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816de00640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170151f00 0x7f81701f30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816de00640 1 -- 172.21.15.98:0/1834195788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8170153a90 con 0x7f81700c6a60 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816e800640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170151f00 0x7f81701f30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816de00640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81700c6a60 0x7f81700c3140 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f816000e810 tx=0x7f816000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f8166000640 1 -- 172.21.15.98:0/1834195788 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816000f040 con 0x7f81700c6a60 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f8166000640 1 -- 172.21.15.98:0/1834195788 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8160004030 con 0x7f81700c6a60 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816f200640 1 -- 172.21.15.98:0/1834195788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8170152d10 con 0x7f81700c6a60 2024-05-27T00:27:40.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f8166000640 1 -- 172.21.15.98:0/1834195788 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160004400 con 0x7f81700c6a60 2024-05-27T00:27:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.195+0000 7f816f200640 1 -- 172.21.15.98:0/1834195788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8170152fb0 con 0x7f81700c6a60 2024-05-27T00:27:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.199+0000 7f816f200640 1 -- 172.21.15.98:0/1834195788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8170075c40 con 0x7f81700c6a60 2024-05-27T00:27:40.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.199+0000 7f8166000640 1 -- 172.21.15.98:0/1834195788 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f816000c6d0 con 0x7f81700c6a60 2024-05-27T00:27:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.199+0000 7f8166000640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f814c0678c0 0x7f814c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.199+0000 7f8166000640 1 -- 172.21.15.98:0/1834195788 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7f8160085d30 con 0x7f81700c6a60 2024-05-27T00:27:40.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.199+0000 7f8167e00640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f814c0678c0 0x7f814c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:40.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.199+0000 7f8167e00640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f814c0678c0 0x7f814c069d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f81701f0d20 tx=0x7f8158008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:40.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.203+0000 7f8166000640 1 -- 172.21.15.98:0/1834195788 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8160017020 con 0x7f81700c6a60 2024-05-27T00:27:40.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.395+0000 7f816f200640 1 -- 172.21.15.98:0/1834195788 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8170157c00 con 0x7f81700c6a60 2024-05-27T00:27:40.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.395+0000 7f8166000640 1 -- 172.21.15.98:0/1834195788 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6839 (secure 0 0 0) 0x7f8160056410 con 0x7f81700c6a60 2024-05-27T00:27:40.400 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:40.400 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5943,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:23.529003+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5802,"qdb_cluster":[5802]},"id":1}]} 2024-05-27T00:27:40.400 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 52 2024-05-27T00:27:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 -- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f814c0678c0 msgr2=0x7f814c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f814c0678c0 0x7f814c069d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f81701f0d20 tx=0x7f8158008040 comp rx=0 tx=0).stop 2024-05-27T00:27:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 -- 172.21.15.98:0/1834195788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81700c6a60 msgr2=0x7f81700c3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81700c6a60 0x7f81700c3140 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f816000e810 tx=0x7f816000ecd0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 -- 172.21.15.98:0/1834195788 shutdown_connections 2024-05-27T00:27:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f814c0678c0 0x7f814c069d80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8170151f00 0x7f81701f30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f81700c6e40 0x7f817014faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 --2- 172.21.15.98:0/1834195788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f81700c6a60 0x7f81700c3140 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:40.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 -- 172.21.15.98:0/1834195788 >> 172.21.15.98:0/1834195788 conn(0x7f81700cbe00 msgr2=0x7f81701569f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:40.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 -- 172.21.15.98:0/1834195788 shutdown_connections 2024-05-27T00:27:40.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:40.399+0000 7f816f200640 1 -- 172.21.15.98:0/1834195788 wait complete. 2024-05-27T00:27:40.420 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:27:42.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.555+0000 7fa687c00640 1 Processor -- start 2024-05-27T00:27:42.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.555+0000 7fa687c00640 1 -- start start 2024-05-27T00:27:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.555+0000 7fa687c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa680152700 0x7fa6800c3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.555+0000 7fa687c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa6800ce800 0x7fa680158590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.555+0000 7fa687c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa6800c8250 0x7fa68006f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.555+0000 7fa687c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa6800ce370 con 0x7fa680152700 2024-05-27T00:27:42.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.555+0000 7fa687c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa6800c4be0 con 0x7fa6800c8250 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.555+0000 7fa687c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa6800c35d0 con 0x7fa6800ce800 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa685200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa680152700 0x7fa6800c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa685200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa680152700 0x7fa6800c3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53430/0 (socket says 172.21.15.98:53430) 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa685200640 1 -- 172.21.15.98:0/3351161133 learned_addr learned my addr 172.21.15.98:0/3351161133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa677e00640 1 --2- 172.21.15.98:0/3351161133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa6800ce800 0x7fa680158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa685c00640 1 --2- 172.21.15.98:0/3351161133 >> v2:172.21.15.204:3300/0 conn(0x7fa6800c8250 0x7fa68006f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa685200640 1 -- 172.21.15.98:0/3351161133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa6800ce800 msgr2=0x7fa680158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa685200640 1 --2- 172.21.15.98:0/3351161133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa6800ce800 0x7fa680158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa685200640 1 -- 172.21.15.98:0/3351161133 >> v2:172.21.15.204:3300/0 conn(0x7fa6800c8250 msgr2=0x7fa68006f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa685200640 1 --2- 172.21.15.98:0/3351161133 >> v2:172.21.15.204:3300/0 conn(0x7fa6800c8250 0x7fa68006f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa685200640 1 -- 172.21.15.98:0/3351161133 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6800c8eb0 con 0x7fa680152700 2024-05-27T00:27:42.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa677e00640 1 --2- 172.21.15.98:0/3351161133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa6800ce800 0x7fa680158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:27:42.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.559+0000 7fa685200640 1 --2- 172.21.15.98:0/3351161133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa680152700 0x7fa6800c3090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa680072fb0 tx=0x7fa67001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb99c0cbe85971a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa67fe00640 1 -- 172.21.15.98:0/3351161133 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700b0070 con 0x7fa680152700 2024-05-27T00:27:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa67fe00640 1 -- 172.21.15.98:0/3351161133 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6700ab600 con 0x7fa680152700 2024-05-27T00:27:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa67fe00640 1 -- 172.21.15.98:0/3351161133 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700ab9d0 con 0x7fa680152700 2024-05-27T00:27:42.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 -- 172.21.15.98:0/3351161133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa680152700 msgr2=0x7fa6800c3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 --2- 172.21.15.98:0/3351161133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa680152700 0x7fa6800c3090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa680072fb0 tx=0x7fa67001c490 comp rx=0 tx=0).stop 2024-05-27T00:27:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 -- 172.21.15.98:0/3351161133 shutdown_connections 2024-05-27T00:27:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 --2- 172.21.15.98:0/3351161133 >> v2:172.21.15.204:3300/0 conn(0x7fa6800c8250 0x7fa68006f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 --2- 172.21.15.98:0/3351161133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa6800ce800 0x7fa680158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 --2- 172.21.15.98:0/3351161133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa680152700 0x7fa6800c3090 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 -- 172.21.15.98:0/3351161133 >> 172.21.15.98:0/3351161133 conn(0x7fa68006d680 msgr2=0x7fa68006da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 -- 172.21.15.98:0/3351161133 shutdown_connections 2024-05-27T00:27:42.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 -- 172.21.15.98:0/3351161133 wait complete. 2024-05-27T00:27:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 Processor -- start 2024-05-27T00:27:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 -- start start 2024-05-27T00:27:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6800c8250 0x7fa6801f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6800ce800 0x7fa68014c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa680152700 0x7fa6801f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa680152ae0 con 0x7fa6800c8250 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa680158ad0 con 0x7fa6800ce800 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa687c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa680152c60 con 0x7fa680152700 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa685200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6800c8250 0x7fa6801f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa685200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6800c8250 0x7fa6801f1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53438/0 (socket says 172.21.15.98:53438) 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa685200640 1 -- 172.21.15.98:0/2301854177 learned_addr learned my addr 172.21.15.98:0/2301854177 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa677e00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6800ce800 0x7fa68014c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa685200640 1 -- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa680152700 msgr2=0x7fa6801f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa685c00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa680152700 0x7fa6801f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa685200640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa680152700 0x7fa6801f4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa685200640 1 -- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6800ce800 msgr2=0x7fa68014c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa685200640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6800ce800 0x7fa68014c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa685200640 1 -- 172.21.15.98:0/2301854177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6801f2540 con 0x7fa6800c8250 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa677e00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6800ce800 0x7fa68014c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.563+0000 7fa685c00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa680152700 0x7fa6801f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.567+0000 7fa685200640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6800c8250 0x7fa6801f1af0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa6700ae040 tx=0x7fa670019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.567+0000 7fa67d600640 1 -- 172.21.15.98:0/2301854177 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700b0040 con 0x7fa6800c8250 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.567+0000 7fa67d600640 1 -- 172.21.15.98:0/2301854177 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6700b4620 con 0x7fa6800c8250 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.567+0000 7fa67d600640 1 -- 172.21.15.98:0/2301854177 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6700b4a10 con 0x7fa6800c8250 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.567+0000 7fa687c00640 1 -- 172.21.15.98:0/2301854177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa680151240 con 0x7fa6800c8250 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.567+0000 7fa687c00640 1 -- 172.21.15.98:0/2301854177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa680151800 con 0x7fa6800c8250 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.567+0000 7fa687c00640 1 -- 172.21.15.98:0/2301854177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa680075cb0 con 0x7fa6800c8250 2024-05-27T00:27:42.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.567+0000 7fa67d600640 1 -- 172.21.15.98:0/2301854177 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa6700ab4c0 con 0x7fa6800c8250 2024-05-27T00:27:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.571+0000 7fa67d600640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa65c0678f0 0x7fa65c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.571+0000 7fa67d600640 1 -- 172.21.15.98:0/2301854177 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7fa67011fb20 con 0x7fa6800c8250 2024-05-27T00:27:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.571+0000 7fa677e00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa65c0678f0 0x7fa65c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:42.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.571+0000 7fa677e00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa65c0678f0 0x7fa65c069db0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa680150a90 tx=0x7fa668008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:42.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.575+0000 7fa67d600640 1 -- 172.21.15.98:0/2301854177 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa670016110 con 0x7fa6800c8250 2024-05-27T00:27:42.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.767+0000 7fa687c00640 1 -- 172.21.15.98:0/2301854177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa68006fea0 con 0x7fa6800c8250 2024-05-27T00:27:42.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.767+0000 7fa67d600640 1 -- 172.21.15.98:0/2301854177 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6839 (secure 0 0 0) 0x7fa6700f0450 con 0x7fa6800c8250 2024-05-27T00:27:42.775 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:42.776 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5943,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:23.529003+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5802,"qdb_cluster":[5802]},"id":1}]} 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 52 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 -- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa65c0678f0 msgr2=0x7fa65c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa65c0678f0 0x7fa65c069db0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa680150a90 tx=0x7fa668008040 comp rx=0 tx=0).stop 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 -- 172.21.15.98:0/2301854177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6800c8250 msgr2=0x7fa6801f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6800c8250 0x7fa6801f1af0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa6700ae040 tx=0x7fa670019230 comp rx=0 tx=0).stop 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 -- 172.21.15.98:0/2301854177 shutdown_connections 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa680152700 0x7fa6801f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa65c0678f0 0x7fa65c069db0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa6800ce800 0x7fa68014c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 --2- 172.21.15.98:0/2301854177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6800c8250 0x7fa6801f1af0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 -- 172.21.15.98:0/2301854177 >> 172.21.15.98:0/2301854177 conn(0x7fa68006d680 msgr2=0x7fa6800cbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:42.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 -- 172.21.15.98:0/2301854177 shutdown_connections 2024-05-27T00:27:42.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:42.771+0000 7fa687c00640 1 -- 172.21.15.98:0/2301854177 wait complete. 2024-05-27T00:27:42.793 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:27:44.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f963c600640 1 Processor -- start 2024-05-27T00:27:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f963c600640 1 -- start start 2024-05-27T00:27:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f963c600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 0x7f9634154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f963c600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f96340cb570 0x7f9634154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f963c600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f963415a950 con 0x7f9634070c00 2024-05-27T00:27:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f963c600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f963415e060 con 0x7f96340cb570 2024-05-27T00:27:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f963c600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f963415e340 con 0x7f9634070fe0 2024-05-27T00:27:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f9639c00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f9634070c00 0x7f96340c8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:50426/0 (socket says 172.21.15.98:50426) 2024-05-27T00:27:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f9639c00640 1 -- 172.21.15.98:0/166029300 learned_addr learned my addr 172.21.15.98:0/166029300 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f9639200640 1 --2- 172.21.15.98:0/166029300 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 0x7f9634154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f963a600640 1 --2- 172.21.15.98:0/166029300 >> v2:172.21.15.204:3300/0 conn(0x7f96340cb570 0x7f9634154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f9633e00640 1 -- 172.21.15.98:0/166029300 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 695711508 0 0) 0x7f963415a950 con 0x7f9634070c00 2024-05-27T00:27:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f9633e00640 1 -- 172.21.15.98:0/166029300 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f961c003530 con 0x7f9634070c00 2024-05-27T00:27:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f9639200640 1 -- 172.21.15.98:0/166029300 >> v2:172.21.15.204:3300/0 conn(0x7f96340cb570 msgr2=0x7f9634154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f9639200640 1 --2- 172.21.15.98:0/166029300 >> v2:172.21.15.204:3300/0 conn(0x7f96340cb570 0x7f9634154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f9639200640 1 -- 172.21.15.98:0/166029300 >> v1:172.21.15.204:6789/0 conn(0x7f9634070c00 legacy=0x7f96340c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:44.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.931+0000 7f9639200640 1 -- 172.21.15.98:0/166029300 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963415e620 con 0x7f9634070fe0 2024-05-27T00:27:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f9639200640 1 --2- 172.21.15.98:0/166029300 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 0x7f9634154780 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f96200097c0 tx=0x7f962001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a210c9a37017316 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f9633e00640 1 -- 172.21.15.98:0/166029300 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96200b5070 con 0x7f9634070fe0 2024-05-27T00:27:44.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f9633e00640 1 -- 172.21.15.98:0/166029300 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96200a8990 con 0x7f9634070fe0 2024-05-27T00:27:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f9633e00640 1 -- 172.21.15.98:0/166029300 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96200a8d40 con 0x7f9634070fe0 2024-05-27T00:27:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f963c600640 1 -- 172.21.15.98:0/166029300 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 msgr2=0x7f9634154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f963c600640 1 --2- 172.21.15.98:0/166029300 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 0x7f9634154780 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f96200097c0 tx=0x7f962001ec90 comp rx=0 tx=0).stop 2024-05-27T00:27:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f963c600640 1 -- 172.21.15.98:0/166029300 shutdown_connections 2024-05-27T00:27:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f963c600640 1 --2- 172.21.15.98:0/166029300 >> v2:172.21.15.204:3300/0 conn(0x7f96340cb570 0x7f9634154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f963c600640 1 --2- 172.21.15.98:0/166029300 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 0x7f9634154780 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f963c600640 1 -- 172.21.15.98:0/166029300 >> 172.21.15.98:0/166029300 conn(0x7f9634072f90 msgr2=0x7f96340733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f963c600640 1 -- 172.21.15.98:0/166029300 shutdown_connections 2024-05-27T00:27:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.935+0000 7f963c600640 1 -- 172.21.15.98:0/166029300 wait complete. 2024-05-27T00:27:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963c600640 1 Processor -- start 2024-05-27T00:27:44.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963c600640 1 -- start start 2024-05-27T00:27:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963c600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9634070c00 0x7f96341f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963c600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 0x7f963415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963c600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340cb570 0x7f96341570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963c600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f96340c9110 con 0x7f96340cb570 2024-05-27T00:27:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963c600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f963415ebe0 con 0x7f9634070c00 2024-05-27T00:27:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963c600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f963415ae50 con 0x7f9634070fe0 2024-05-27T00:27:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963a600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340cb570 0x7f96341570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:44.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963a600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340cb570 0x7f96341570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37710/0 (socket says 172.21.15.98:37710) 2024-05-27T00:27:44.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963a600640 1 -- 172.21.15.98:0/2023550699 learned_addr learned my addr 172.21.15.98:0/2023550699 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:44.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f9639c00640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9634070c00 0x7f96341f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:44.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f9639200640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 0x7f963415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:44.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963a600640 1 -- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 msgr2=0x7f963415d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:44.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963a600640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 0x7f963415d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:44.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963a600640 1 -- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9634070c00 msgr2=0x7f96341f0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:44.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963a600640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9634070c00 0x7f96341f0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:44.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963a600640 1 -- 172.21.15.98:0/2023550699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9634156310 con 0x7f96340cb570 2024-05-27T00:27:44.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f9639200640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 0x7f963415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:27:44.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963a600640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340cb570 0x7f96341570b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f962c009d30 tx=0x7f962c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:44.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f9631600640 1 -- 172.21.15.98:0/2023550699 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c018020 con 0x7f96340cb570 2024-05-27T00:27:44.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f9631600640 1 -- 172.21.15.98:0/2023550699 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f962c014480 con 0x7f96340cb570 2024-05-27T00:27:44.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963c600640 1 -- 172.21.15.98:0/2023550699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96340c3e40 con 0x7f96340cb570 2024-05-27T00:27:44.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f9631600640 1 -- 172.21.15.98:0/2023550699 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962c014870 con 0x7f96340cb570 2024-05-27T00:27:44.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.939+0000 7f963c600640 1 -- 172.21.15.98:0/2023550699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96341565d0 con 0x7f96340cb570 2024-05-27T00:27:44.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.943+0000 7f963c600640 1 -- 172.21.15.98:0/2023550699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9634075cb0 con 0x7f96340cb570 2024-05-27T00:27:44.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.943+0000 7f9631600640 1 -- 172.21.15.98:0/2023550699 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f962c01b020 con 0x7f96340cb570 2024-05-27T00:27:44.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.943+0000 7f9631600640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96140678f0 0x7f9614069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:44.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.943+0000 7f9631600640 1 -- 172.21.15.98:0/2023550699 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7f962c086ac0 con 0x7f96340cb570 2024-05-27T00:27:44.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.943+0000 7f9639c00640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96140678f0 0x7f9614069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:44.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.943+0000 7f9639c00640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96140678f0 0x7f9614069db0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9628007fd0 tx=0x7f9628009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:44.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:44.947+0000 7f9631600640 1 -- 172.21.15.98:0/2023550699 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f962c022070 con 0x7f96340cb570 2024-05-27T00:27:45.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.139+0000 7f963c600640 1 -- 172.21.15.98:0/2023550699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96341580d0 con 0x7f96340cb570 2024-05-27T00:27:45.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.139+0000 7f9631600640 1 -- 172.21.15.98:0/2023550699 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6839 (secure 0 0 0) 0x7f962c010040 con 0x7f96340cb570 2024-05-27T00:27:45.143 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:45.143 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5943,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:23.529003+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5802,"qdb_cluster":[5802]},"id":1}]} 2024-05-27T00:27:45.143 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 52 2024-05-27T00:27:45.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.139+0000 7f963c600640 1 -- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96140678f0 msgr2=0x7f9614069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:45.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.139+0000 7f963c600640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96140678f0 0x7f9614069db0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9628007fd0 tx=0x7f9628009040 comp rx=0 tx=0).stop 2024-05-27T00:27:45.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.139+0000 7f963c600640 1 -- 172.21.15.98:0/2023550699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340cb570 msgr2=0x7f96341570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:45.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.139+0000 7f963c600640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340cb570 0x7f96341570b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f962c009d30 tx=0x7f962c00c490 comp rx=0 tx=0).stop 2024-05-27T00:27:45.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.143+0000 7f963c600640 1 -- 172.21.15.98:0/2023550699 shutdown_connections 2024-05-27T00:27:45.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.143+0000 7f963c600640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96340cb570 0x7f96341570b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:45.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.143+0000 7f963c600640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9634070fe0 0x7f963415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:45.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.143+0000 7f963c600640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f96140678f0 0x7f9614069db0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:45.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.143+0000 7f963c600640 1 --2- 172.21.15.98:0/2023550699 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9634070c00 0x7f96341f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:45.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.143+0000 7f963c600640 1 -- 172.21.15.98:0/2023550699 >> 172.21.15.98:0/2023550699 conn(0x7f9634072f90 msgr2=0x7f963406d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:45.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.143+0000 7f963c600640 1 -- 172.21.15.98:0/2023550699 shutdown_connections 2024-05-27T00:27:45.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:45.143+0000 7f963c600640 1 -- 172.21.15.98:0/2023550699 wait complete. 2024-05-27T00:27:45.164 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:27:47.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.299+0000 7f5d12400640 1 Processor -- start 2024-05-27T00:27:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.299+0000 7f5d12400640 1 -- start start 2024-05-27T00:27:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d12400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d0c0722e0 0x7f5d0c0c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d12400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d0c073110 con 0x7f5d0c0c83b0 2024-05-27T00:27:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d12400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d0c073380 con 0x7f5d0c1517d0 2024-05-27T00:27:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d12400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5d0c157720 con 0x7f5d0c0722e0 2024-05-27T00:27:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d03e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d0c0722e0 0x7f5d0c0c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d11a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5d0c0c83b0 0x7f5d0c0c3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42646/0 (socket says 172.21.15.98:42646) 2024-05-27T00:27:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d03e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d0c0722e0 0x7f5d0c0c33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37716/0 (socket says 172.21.15.98:37716) 2024-05-27T00:27:47.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d11a00640 1 -- 172.21.15.98:0/1204228574 learned_addr learned my addr 172.21.15.98:0/1204228574 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:47.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d03e00640 1 -- 172.21.15.98:0/1204228574 >> v1:172.21.15.204:6789/0 conn(0x7f5d0c1517d0 legacy=0x7f5d0c157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:47.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d03e00640 1 -- 172.21.15.98:0/1204228574 >> v1:172.21.15.98:6789/0 conn(0x7f5d0c0c83b0 legacy=0x7f5d0c0c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:47.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.303+0000 7f5d03e00640 1 -- 172.21.15.98:0/1204228574 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0c073660 con 0x7f5d0c0722e0 2024-05-27T00:27:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d03e00640 1 --2- 172.21.15.98:0/1204228574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c0722e0 0x7f5d0c0c33c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5cf40097c0 tx=0x7f5cf401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e36159ac8b9d9c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d0be00640 1 -- 172.21.15.98:0/1204228574 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf40b5070 con 0x7f5d0c0722e0 2024-05-27T00:27:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d0be00640 1 -- 172.21.15.98:0/1204228574 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf40a8990 con 0x7f5d0c0722e0 2024-05-27T00:27:47.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d0be00640 1 -- 172.21.15.98:0/1204228574 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf40a8d60 con 0x7f5d0c0722e0 2024-05-27T00:27:47.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d12400640 1 -- 172.21.15.98:0/1204228574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c0722e0 msgr2=0x7f5d0c0c33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:47.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d12400640 1 --2- 172.21.15.98:0/1204228574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c0722e0 0x7f5d0c0c33c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5cf40097c0 tx=0x7f5cf401ec90 comp rx=0 tx=0).stop 2024-05-27T00:27:47.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d12400640 1 -- 172.21.15.98:0/1204228574 shutdown_connections 2024-05-27T00:27:47.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d12400640 1 --2- 172.21.15.98:0/1204228574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c0722e0 0x7f5d0c0c33c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:47.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d12400640 1 -- 172.21.15.98:0/1204228574 >> 172.21.15.98:0/1204228574 conn(0x7f5d0c06f660 msgr2=0x7f5d0c06fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:47.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d12400640 1 -- 172.21.15.98:0/1204228574 shutdown_connections 2024-05-27T00:27:47.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.307+0000 7f5d12400640 1 -- 172.21.15.98:0/1204228574 wait complete. 2024-05-27T00:27:47.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d12400640 1 Processor -- start 2024-05-27T00:27:47.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d12400640 1 -- start start 2024-05-27T00:27:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d12400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d0c0722e0 0x7f5d0c14ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d12400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d0c0c83b0 0x7f5d0c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d12400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c1517d0 0x7f5d0c1e4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d12400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0c0c4010 con 0x7f5d0c1517d0 2024-05-27T00:27:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d12400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0c0726e0 con 0x7f5d0c0722e0 2024-05-27T00:27:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d12400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5d0c072940 con 0x7f5d0c0c83b0 2024-05-27T00:27:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d11a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c1517d0 0x7f5d0c1e4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d11a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c1517d0 0x7f5d0c1e4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37726/0 (socket says 172.21.15.98:37726) 2024-05-27T00:27:47.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d11a00640 1 -- 172.21.15.98:0/686455144 learned_addr learned my addr 172.21.15.98:0/686455144 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:47.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d03e00640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d0c0c83b0 0x7f5d0c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:47.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d11000640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d0c0722e0 0x7f5d0c14ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d11a00640 1 -- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d0c0c83b0 msgr2=0x7f5d0c158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d11a00640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d0c0c83b0 0x7f5d0c158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d11a00640 1 -- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d0c0722e0 msgr2=0x7f5d0c14ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d11a00640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d0c0722e0 0x7f5d0c14ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d11a00640 1 -- 172.21.15.98:0/686455144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0c0ccb60 con 0x7f5d0c1517d0 2024-05-27T00:27:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d03e00640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d0c0c83b0 0x7f5d0c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:27:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d11a00640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c1517d0 0x7f5d0c1e4dc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400cab0 tx=0x7f5d0400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d09600640 1 -- 172.21.15.98:0/686455144 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d04017070 con 0x7f5d0c1517d0 2024-05-27T00:27:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d09600640 1 -- 172.21.15.98:0/686455144 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d0400ac30 con 0x7f5d0c1517d0 2024-05-27T00:27:47.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d09600640 1 -- 172.21.15.98:0/686455144 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d04012400 con 0x7f5d0c1517d0 2024-05-27T00:27:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d12400640 1 -- 172.21.15.98:0/686455144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d0c143500 con 0x7f5d0c1517d0 2024-05-27T00:27:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.311+0000 7f5d12400640 1 -- 172.21.15.98:0/686455144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d0c0cbf50 con 0x7f5d0c1517d0 2024-05-27T00:27:47.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.315+0000 7f5d12400640 1 -- 172.21.15.98:0/686455144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d0c075c40 con 0x7f5d0c1517d0 2024-05-27T00:27:47.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.315+0000 7f5d09600640 1 -- 172.21.15.98:0/686455144 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5d0401a070 con 0x7f5d0c1517d0 2024-05-27T00:27:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.315+0000 7f5d09600640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5ce80678c0 0x7f5ce8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:47.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.315+0000 7f5d09600640 1 -- 172.21.15.98:0/686455144 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7f5d04085c90 con 0x7f5d0c1517d0 2024-05-27T00:27:47.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.315+0000 7f5d11000640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5ce80678c0 0x7f5ce8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:47.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.319+0000 7f5d11000640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5ce80678c0 0x7f5ce8069d80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc007fd0 tx=0x7f5cfc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:47.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.323+0000 7f5d09600640 1 -- 172.21.15.98:0/686455144 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f5d04055db0 con 0x7f5d0c1517d0 2024-05-27T00:27:47.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.511+0000 7f5d12400640 1 -- 172.21.15.98:0/686455144 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d0c0c7370 con 0x7f5d0c1517d0 2024-05-27T00:27:47.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.511+0000 7f5d09600640 1 -- 172.21.15.98:0/686455144 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6839 (secure 0 0 0) 0x7f5d0405a0b0 con 0x7f5d0c1517d0 2024-05-27T00:27:47.516 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:47.516 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5943,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:23.529003+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5802,"qdb_cluster":[5802]},"id":1}]} 2024-05-27T00:27:47.517 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 52 2024-05-27T00:27:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 -- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5ce80678c0 msgr2=0x7f5ce8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5ce80678c0 0x7f5ce8069d80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc007fd0 tx=0x7f5cfc009040 comp rx=0 tx=0).stop 2024-05-27T00:27:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 -- 172.21.15.98:0/686455144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c1517d0 msgr2=0x7f5d0c1e4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c1517d0 0x7f5d0c1e4dc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400cab0 tx=0x7f5d0400cf70 comp rx=0 tx=0).stop 2024-05-27T00:27:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 -- 172.21.15.98:0/686455144 shutdown_connections 2024-05-27T00:27:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d0c1517d0 0x7f5d0c1e4dc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d0c0c83b0 0x7f5d0c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5ce80678c0 0x7f5ce8069d80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 --2- 172.21.15.98:0/686455144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d0c0722e0 0x7f5d0c14ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 -- 172.21.15.98:0/686455144 >> 172.21.15.98:0/686455144 conn(0x7f5d0c06f660 msgr2=0x7f5d0c0c3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 -- 172.21.15.98:0/686455144 shutdown_connections 2024-05-27T00:27:47.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:47.515+0000 7f5d12400640 1 -- 172.21.15.98:0/686455144 wait complete. 2024-05-27T00:27:47.537 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:27:49.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.675+0000 7fc4fde00640 1 Processor -- start 2024-05-27T00:27:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.675+0000 7fc4fde00640 1 -- start start 2024-05-27T00:27:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4fde00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f80cb7e0 0x7fc4f8158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4fde00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f815be40 con 0x7fc4f8152a50 2024-05-27T00:27:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4fde00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f80c50e0 con 0x7fc4f80b3ed0 2024-05-27T00:27:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4fde00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f80c8a00 con 0x7fc4f80cb7e0 2024-05-27T00:27:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f6a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc4f8152a50 0x7fc4f80709a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42648/0 (socket says 172.21.15.98:42648) 2024-05-27T00:27:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f6a00640 1 -- 172.21.15.98:0/3959855396 learned_addr learned my addr 172.21.15.98:0/3959855396 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f6000640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f80cb7e0 0x7fc4f8158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 584532641 0 0) 0x7fc4f815be40 con 0x7fc4f8152a50 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4dc003530 con 0x7fc4f8152a50 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 240206790 0 0) 0x7fc4dc003530 con 0x7fc4f8152a50 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f80cb7e0 msgr2=0x7fc4f8158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f80cb7e0 0x7fc4f8158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 >> v1:172.21.15.204:6789/0 conn(0x7fc4f80b3ed0 legacy=0x7fc4f8162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f80c8c50 con 0x7fc4f8152a50 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fc4e80016a0 con 0x7fc4f8152a50 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 >> v1:172.21.15.98:6789/0 conn(0x7fc4f8152a50 legacy=0x7fc4f80709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4dc0039f0 0x7fc4dc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f80cb7e0 0x7fc4dc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc004d50 0x7fc4dc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f80c8a00 con 0x7fc4dc004d50 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc4dc00bb00 con 0x7fc4dc0039f0 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc4dc00bd00 con 0x7fc4f80cb7e0 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f6a00640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4dc0039f0 0x7fc4dc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f7400640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc004d50 0x7fc4dc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f6000640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f80cb7e0 0x7fc4dc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f7400640 1 -- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f80cb7e0 msgr2=0x7fc4dc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f7400640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f80cb7e0 0x7fc4dc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f7400640 1 -- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4dc0039f0 msgr2=0x7fc4dc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f7400640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4dc0039f0 0x7fc4dc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.679+0000 7fc4f7400640 1 -- 172.21.15.98:0/3959855396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4dc003530 con 0x7fc4dc004d50 2024-05-27T00:27:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4f7400640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc004d50 0x7fc4dc0051d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0001e70 tx=0x7fc4f001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9550679d4e45f0bb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0001a90 con 0x7fc4dc004d50 2024-05-27T00:27:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f00a4b90 con 0x7fc4dc004d50 2024-05-27T00:27:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4f5600640 1 -- 172.21.15.98:0/3959855396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0003980 con 0x7fc4dc004d50 2024-05-27T00:27:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4fde00640 1 -- 172.21.15.98:0/3959855396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc004d50 msgr2=0x7fc4dc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4fde00640 1 --2- 172.21.15.98:0/3959855396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4dc004d50 0x7fc4dc0051d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0001e70 tx=0x7fc4f001c430 comp rx=0 tx=0).stop 2024-05-27T00:27:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4f7400640 1 -- 172.21.15.98:0/3959855396 reap_dead start 2024-05-27T00:27:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4fde00640 1 -- 172.21.15.98:0/3959855396 shutdown_connections 2024-05-27T00:27:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4fde00640 1 -- 172.21.15.98:0/3959855396 >> 172.21.15.98:0/3959855396 conn(0x7fc4f8072f90 msgr2=0x7fc4f80733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4fde00640 1 -- 172.21.15.98:0/3959855396 shutdown_connections 2024-05-27T00:27:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.683+0000 7fc4fde00640 1 -- 172.21.15.98:0/3959855396 wait complete. 2024-05-27T00:27:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 Processor -- start 2024-05-27T00:27:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 -- start start 2024-05-27T00:27:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f81e1e40 0x7fc4f81f1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f81e6a00 0x7fc4f80ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f81e5fb0 0x7fc4f81e2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f8152e30 con 0x7fc4f81e1e40 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f80cb570 con 0x7fc4f81e6a00 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f8158f70 con 0x7fc4f81e5fb0 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f81e1e40 0x7fc4f81f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f81e1e40 0x7fc4f81f1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37730/0 (socket says 172.21.15.98:37730) 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f6a00640 1 -- 172.21.15.98:0/3831685640 learned_addr learned my addr 172.21.15.98:0/3831685640 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f6000640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f81e6a00 0x7fc4f80ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f7400640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f81e5fb0 0x7fc4f81e2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f6a00640 1 -- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f81e5fb0 msgr2=0x7fc4f81e2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f6a00640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f81e5fb0 0x7fc4f81e2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f6a00640 1 -- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f81e6a00 msgr2=0x7fc4f80ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:49.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f6a00640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f81e6a00 0x7fc4f80ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f6a00640 1 -- 172.21.15.98:0/3831685640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f80cd120 con 0x7fc4f81e1e40 2024-05-27T00:27:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4f6a00640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f81e1e40 0x7fc4f81f1040 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8012570 tx=0x7fc4e8012a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4ef400640 1 -- 172.21.15.98:0/3831685640 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e801c070 con 0x7fc4f81e1e40 2024-05-27T00:27:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4ef400640 1 -- 172.21.15.98:0/3831685640 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4e8014810 con 0x7fc4f81e1e40 2024-05-27T00:27:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 -- 172.21.15.98:0/3831685640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f80cd400 con 0x7fc4f81e1e40 2024-05-27T00:27:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4ef400640 1 -- 172.21.15.98:0/3831685640 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8014be0 con 0x7fc4f81e1e40 2024-05-27T00:27:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 -- 172.21.15.98:0/3831685640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f80cd9c0 con 0x7fc4f81e1e40 2024-05-27T00:27:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.687+0000 7fc4fde00640 1 -- 172.21.15.98:0/3831685640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f8075cb0 con 0x7fc4f81e1e40 2024-05-27T00:27:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.691+0000 7fc4ef400640 1 -- 172.21.15.98:0/3831685640 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc4e801f070 con 0x7fc4f81e1e40 2024-05-27T00:27:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.691+0000 7fc4ef400640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d80678f0 0x7fc4d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.691+0000 7fc4ef400640 1 -- 172.21.15.98:0/3831685640 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7fc4e8088b30 con 0x7fc4f81e1e40 2024-05-27T00:27:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.691+0000 7fc4f6000640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d80678f0 0x7fc4d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.691+0000 7fc4f6000640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d80678f0 0x7fc4d8069db0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc4f80c6ab0 tx=0x7fc4e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:49.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.695+0000 7fc4ef400640 1 -- 172.21.15.98:0/3831685640 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc4e8059210 con 0x7fc4f81e1e40 2024-05-27T00:27:49.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 -- 172.21.15.98:0/3831685640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4f80b3cb0 con 0x7fc4f81e1e40 2024-05-27T00:27:49.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4ef400640 1 -- 172.21.15.98:0/3831685640 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6839 (secure 0 0 0) 0x7fc4e8058b10 con 0x7fc4f81e1e40 2024-05-27T00:27:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:49.891 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5943,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:23.529003+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5802,"qdb_cluster":[5802]},"id":1}]} 2024-05-27T00:27:49.891 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 52 2024-05-27T00:27:49.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 -- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d80678f0 msgr2=0x7fc4d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:49.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d80678f0 0x7fc4d8069db0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc4f80c6ab0 tx=0x7fc4e4008040 comp rx=0 tx=0).stop 2024-05-27T00:27:49.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 -- 172.21.15.98:0/3831685640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f81e1e40 msgr2=0x7fc4f81f1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:49.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f81e1e40 0x7fc4f81f1040 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8012570 tx=0x7fc4e8012a30 comp rx=0 tx=0).stop 2024-05-27T00:27:49.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 -- 172.21.15.98:0/3831685640 shutdown_connections 2024-05-27T00:27:49.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f81e5fb0 0x7fc4f81e2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:49.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d80678f0 0x7fc4d8069db0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:49.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f81e6a00 0x7fc4f80ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:49.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 --2- 172.21.15.98:0/3831685640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f81e1e40 0x7fc4f81f1040 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:49.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.887+0000 7fc4fde00640 1 -- 172.21.15.98:0/3831685640 >> 172.21.15.98:0/3831685640 conn(0x7fc4f8072f90 msgr2=0x7fc4f80c75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:49.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.891+0000 7fc4fde00640 1 -- 172.21.15.98:0/3831685640 shutdown_connections 2024-05-27T00:27:49.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:49.891+0000 7fc4fde00640 1 -- 172.21.15.98:0/3831685640 wait complete. 2024-05-27T00:27:49.912 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:27:51.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:52.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e9b400640 1 Processor -- start 2024-05-27T00:27:52.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e9b400640 1 -- start start 2024-05-27T00:27:52.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e9b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940cc8f0 0x7f3e94070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:52.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e9b400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e941515f0 con 0x7f3e940c1310 2024-05-27T00:27:52.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e9b400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e940c1000 con 0x7f3e941511d0 2024-05-27T00:27:52.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e9b400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3e940cc370 con 0x7f3e940cc8f0 2024-05-27T00:27:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e99400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3e940c1310 0x7f3e940c8670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42650/0 (socket says 172.21.15.98:42650) 2024-05-27T00:27:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e99400640 1 -- 172.21.15.98:0/4110727452 learned_addr learned my addr 172.21.15.98:0/4110727452 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e98a00640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940cc8f0 0x7f3e94070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1987749715 0 0) 0x7f3e941515f0 con 0x7f3e940c1310 2024-05-27T00:27:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e78003530 con 0x7f3e940c1310 2024-05-27T00:27:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3777817555 0 0) 0x7f3e940c1000 con 0x7f3e941511d0 2024-05-27T00:27:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e941515f0 con 0x7f3e941511d0 2024-05-27T00:27:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2159357233 0 0) 0x7f3e78003530 con 0x7f3e940c1310 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940cc8f0 msgr2=0x7f3e94070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e93e00640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940cc8f0 0x7f3e94070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 >> v1:172.21.15.204:6789/0 conn(0x7f3e941511d0 legacy=0x7f3e940cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e940c83c0 con 0x7f3e940c1310 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.046+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f3e8c001a00 con 0x7f3e940c1310 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 >> v1:172.21.15.98:6789/0 conn(0x7f3e940c1310 legacy=0x7f3e940c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e93e00640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e78003b80 0x7f3e78003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e93e00640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940cc8f0 0x7f3e78004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e93e00640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e78004d50 0x7f3e780051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e940cc370 con 0x7f3e78003b80 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7800dd70 con 0x7f3e78004d50 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7800df40 con 0x7f3e940cc8f0 2024-05-27T00:27:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e98a00640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e78003b80 0x7f3e78003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e99400640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e78004d50 0x7f3e780051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e98a00640 1 -- 172.21.15.98:0/4110727452 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940cc8f0 msgr2=0x7f3e78004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e98a00640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940cc8f0 0x7f3e78004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e98a00640 1 -- 172.21.15.98:0/4110727452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e78004d50 msgr2=0x7f3e780051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e98a00640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e78004d50 0x7f3e780051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e98a00640 1 -- 172.21.15.98:0/4110727452 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e78003530 con 0x7f3e78003b80 2024-05-27T00:27:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e98a00640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e78003b80 0x7f3e78003f80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009b10 tx=0x7f3e8401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99beaaf6829bf800 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e840096b0 con 0x7f3e78003b80 2024-05-27T00:27:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e840a6490 con 0x7f3e78003b80 2024-05-27T00:27:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.050+0000 7f3e93e00640 1 -- 172.21.15.98:0/4110727452 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e840a6880 con 0x7f3e78003b80 2024-05-27T00:27:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 -- 172.21.15.98:0/4110727452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e78003b80 msgr2=0x7f3e78003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 --2- 172.21.15.98:0/4110727452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e78003b80 0x7f3e78003f80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009b10 tx=0x7f3e8401cab0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e99400640 1 -- 172.21.15.98:0/4110727452 reap_dead start 2024-05-27T00:27:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 -- 172.21.15.98:0/4110727452 shutdown_connections 2024-05-27T00:27:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 -- 172.21.15.98:0/4110727452 >> 172.21.15.98:0/4110727452 conn(0x7f3e94067530 msgr2=0x7f3e9406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 -- 172.21.15.98:0/4110727452 shutdown_connections 2024-05-27T00:27:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 -- 172.21.15.98:0/4110727452 wait complete. 2024-05-27T00:27:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 Processor -- start 2024-05-27T00:27:52.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 -- start start 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e78003b80 0x7f3e941469c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e940c48b0 0x7f3e940c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e94150050 0x7f3e941504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e940b1dd0 con 0x7f3e940c48b0 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3e94071220 con 0x7f3e94150050 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e9b400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3e940c2150 con 0x7f3e78003b80 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e8be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e940c48b0 0x7f3e940c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e8be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e940c48b0 0x7f3e940c4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37736/0 (socket says 172.21.15.98:37736) 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e8be00640 1 -- 172.21.15.98:0/2068453267 learned_addr learned my addr 172.21.15.98:0/2068453267 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e98a00640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e78003b80 0x7f3e941469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.054+0000 7f3e99400640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e94150050 0x7f3e941504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:52.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e8be00640 1 -- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e78003b80 msgr2=0x7f3e941469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e8be00640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e78003b80 0x7f3e941469c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e8be00640 1 -- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e94150050 msgr2=0x7f3e941504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e8be00640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e94150050 0x7f3e941504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e8be00640 1 -- 172.21.15.98:0/2068453267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9414f9b0 con 0x7f3e940c48b0 2024-05-27T00:27:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e8be00640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e940c48b0 0x7f3e940c4d10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00c500 tx=0x7f3e7c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e91600640 1 -- 172.21.15.98:0/2068453267 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c0052d0 con 0x7f3e940c48b0 2024-05-27T00:27:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e91600640 1 -- 172.21.15.98:0/2068453267 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e7c005470 con 0x7f3e940c48b0 2024-05-27T00:27:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e9b400640 1 -- 172.21.15.98:0/2068453267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e941f3260 con 0x7f3e940c48b0 2024-05-27T00:27:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e91600640 1 -- 172.21.15.98:0/2068453267 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c012830 con 0x7f3e940c48b0 2024-05-27T00:27:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e9b400640 1 -- 172.21.15.98:0/2068453267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e941f3fe0 con 0x7f3e940c48b0 2024-05-27T00:27:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e9b400640 1 -- 172.21.15.98:0/2068453267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e94075cb0 con 0x7f3e940c48b0 2024-05-27T00:27:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.058+0000 7f3e91600640 1 -- 172.21.15.98:0/2068453267 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3e7c0129d0 con 0x7f3e940c48b0 2024-05-27T00:27:52.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.062+0000 7f3e91600640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3e700678c0 0x7f3e70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.062+0000 7f3e91600640 1 -- 172.21.15.98:0/2068453267 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7f3e7c085d10 con 0x7f3e940c48b0 2024-05-27T00:27:52.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.062+0000 7f3e98a00640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3e700678c0 0x7f3e70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:52.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.062+0000 7f3e98a00640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3e700678c0 0x7f3e70069d80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3e840ad000 tx=0x7f3e840b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:52.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.066+0000 7f3e91600640 1 -- 172.21.15.98:0/2068453267 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f3e7c08b020 con 0x7f3e940c48b0 2024-05-27T00:27:52.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.258+0000 7f3e9b400640 1 -- 172.21.15.98:0/2068453267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e94154880 con 0x7f3e940c48b0 2024-05-27T00:27:52.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.258+0000 7f3e91600640 1 -- 172.21.15.98:0/2068453267 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6839 (secure 0 0 0) 0x7f3e7c056400 con 0x7f3e940c48b0 2024-05-27T00:27:52.263 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:52.263 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5943,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:23.529003+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5802,"qdb_cluster":[5802]},"id":1}]} 2024-05-27T00:27:52.264 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 52 2024-05-27T00:27:52.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 -- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3e700678c0 msgr2=0x7f3e70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:52.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3e700678c0 0x7f3e70069d80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3e840ad000 tx=0x7f3e840b2040 comp rx=0 tx=0).stop 2024-05-27T00:27:52.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 -- 172.21.15.98:0/2068453267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e940c48b0 msgr2=0x7f3e940c4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:52.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e940c48b0 0x7f3e940c4d10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00c500 tx=0x7f3e7c00c9c0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 -- 172.21.15.98:0/2068453267 shutdown_connections 2024-05-27T00:27:52.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3e700678c0 0x7f3e70069d80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e94150050 0x7f3e941504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e940c48b0 0x7f3e940c4d10 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 --2- 172.21.15.98:0/2068453267 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e78003b80 0x7f3e941469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:52.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 -- 172.21.15.98:0/2068453267 >> 172.21.15.98:0/2068453267 conn(0x7f3e94067530 msgr2=0x7f3e940c7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:52.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 -- 172.21.15.98:0/2068453267 shutdown_connections 2024-05-27T00:27:52.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:52.262+0000 7f3e9b400640 1 -- 172.21.15.98:0/2068453267 wait complete. 2024-05-27T00:27:52.285 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:27:54.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe34c00640 1 Processor -- start 2024-05-27T00:27:54.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe34c00640 1 -- start start 2024-05-27T00:27:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe34c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe3006f260 0x7fbe300c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe34c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe30159c70 0x7fbe30157140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe34c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe300c6bf0 con 0x7fbe30070d30 2024-05-27T00:27:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe34c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbe30157680 con 0x7fbe30159c70 2024-05-27T00:27:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe34c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbe30156680 con 0x7fbe3006f260 2024-05-27T00:27:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe2fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe30159c70 0x7fbe30157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe2fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbe30159c70 0x7fbe30157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37544/0 (socket says 172.21.15.98:37544) 2024-05-27T00:27:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe2fe00640 1 -- 172.21.15.98:0/2356749966 learned_addr learned my addr 172.21.15.98:0/2356749966 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe2f400640 1 --2- 172.21.15.98:0/2356749966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe3006f260 0x7fbe300c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe2fe00640 1 -- 172.21.15.98:0/2356749966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe3006f260 msgr2=0x7fbe300c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe2fe00640 1 --2- 172.21.15.98:0/2356749966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe3006f260 0x7fbe300c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe2fe00640 1 -- 172.21.15.98:0/2356749966 >> v1:172.21.15.204:6789/0 conn(0x7fbe30070d30 legacy=0x7fbe3006e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.418+0000 7fbe2fe00640 1 -- 172.21.15.98:0/2356749966 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe300c6eb0 con 0x7fbe30159c70 2024-05-27T00:27:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.422+0000 7fbe2fe00640 1 --2- 172.21.15.98:0/2356749966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 0x7fbe30157140 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009e50 tx=0x7fbe2801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d1e172430c2949 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.422+0000 7fbe2ea00640 1 -- 172.21.15.98:0/2356749966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe280a57f0 con 0x7fbe30159c70 2024-05-27T00:27:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe2ea00640 1 -- 172.21.15.98:0/2356749966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe280a5990 con 0x7fbe30159c70 2024-05-27T00:27:54.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe2ea00640 1 -- 172.21.15.98:0/2356749966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe280a5d60 con 0x7fbe30159c70 2024-05-27T00:27:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 -- 172.21.15.98:0/2356749966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 msgr2=0x7fbe30157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 --2- 172.21.15.98:0/2356749966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 0x7fbe30157140 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009e50 tx=0x7fbe2801ca30 comp rx=0 tx=0).stop 2024-05-27T00:27:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 -- 172.21.15.98:0/2356749966 shutdown_connections 2024-05-27T00:27:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 --2- 172.21.15.98:0/2356749966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 0x7fbe30157140 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 --2- 172.21.15.98:0/2356749966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe3006f260 0x7fbe300c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:54.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 -- 172.21.15.98:0/2356749966 >> 172.21.15.98:0/2356749966 conn(0x7fbe300cbe00 msgr2=0x7fbe300cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 -- 172.21.15.98:0/2356749966 shutdown_connections 2024-05-27T00:27:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 -- 172.21.15.98:0/2356749966 wait complete. 2024-05-27T00:27:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 Processor -- start 2024-05-27T00:27:54.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 -- start start 2024-05-27T00:27:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe3006f260 0x7fbe3014d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe30070d30 0x7fbe300ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 0x7fbe301f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbe300c6980 con 0x7fbe30159c70 2024-05-27T00:27:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbe300c6800 con 0x7fbe3006f260 2024-05-27T00:27:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe34c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbe3006ec50 con 0x7fbe30070d30 2024-05-27T00:27:54.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe2fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 0x7fbe301f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe2fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 0x7fbe301f6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37550/0 (socket says 172.21.15.98:37550) 2024-05-27T00:27:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe2fe00640 1 -- 172.21.15.98:0/3417740133 learned_addr learned my addr 172.21.15.98:0/3417740133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe27e00640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe30070d30 0x7fbe300ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.426+0000 7fbe2f400640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe3006f260 0x7fbe3014d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe2fe00640 1 -- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe30070d30 msgr2=0x7fbe300ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe2fe00640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe30070d30 0x7fbe300ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe2fe00640 1 -- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe3006f260 msgr2=0x7fbe3014d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe2fe00640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe3006f260 0x7fbe3014d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe2fe00640 1 -- 172.21.15.98:0/3417740133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3014d950 con 0x7fbe30159c70 2024-05-27T00:27:54.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe2fe00640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 0x7fbe301f6970 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbe280af040 tx=0x7fbe280197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe27400640 1 -- 172.21.15.98:0/3417740133 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe280bc070 con 0x7fbe30159c70 2024-05-27T00:27:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe27400640 1 -- 172.21.15.98:0/3417740133 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe280b8640 con 0x7fbe30159c70 2024-05-27T00:27:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe27400640 1 -- 172.21.15.98:0/3417740133 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe280b8a30 con 0x7fbe30159c70 2024-05-27T00:27:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe34c00640 1 -- 172.21.15.98:0/3417740133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe3014f300 con 0x7fbe30159c70 2024-05-27T00:27:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe34c00640 1 -- 172.21.15.98:0/3417740133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe300cb1b0 con 0x7fbe30159c70 2024-05-27T00:27:54.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe34c00640 1 -- 172.21.15.98:0/3417740133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe30075c40 con 0x7fbe30159c70 2024-05-27T00:27:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.430+0000 7fbe27400640 1 -- 172.21.15.98:0/3417740133 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbe280b8bd0 con 0x7fbe30159c70 2024-05-27T00:27:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.434+0000 7fbe27400640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbe0c0678f0 0x7fbe0c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.434+0000 7fbe27400640 1 -- 172.21.15.98:0/3417740133 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6513+0+0 (secure 0 0 0) 0x7fbe281208d0 con 0x7fbe30159c70 2024-05-27T00:27:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.434+0000 7fbe2f400640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbe0c0678f0 0x7fbe0c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.434+0000 7fbe2f400640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbe0c0678f0 0x7fbe0c069db0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009ac0 tx=0x7fbe20009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:54.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.438+0000 7fbe27400640 1 -- 172.21.15.98:0/3417740133 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fbe28016620 con 0x7fbe30159c70 2024-05-27T00:27:54.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.630+0000 7fbe34c00640 1 -- 172.21.15.98:0/3417740133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe3006e530 con 0x7fbe30159c70 2024-05-27T00:27:54.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.630+0000 7fbe27400640 1 -- 172.21.15.98:0/3417740133 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6839 (secure 0 0 0) 0x7fbe280f0c40 con 0x7fbe30159c70 2024-05-27T00:27:54.637 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:54.637 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5943,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:23.529003+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5802":{"gid":5802,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":33,"addr":"172.21.15.98:6837/2940211164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2940211164},{"type":"v1","addr":"172.21.15.98:6837","nonce":2940211164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5802,"qdb_cluster":[5802]},"id":1}]} 2024-05-27T00:27:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 52 2024-05-27T00:27:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.630+0000 7fbe34c00640 1 -- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbe0c0678f0 msgr2=0x7fbe0c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.630+0000 7fbe34c00640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbe0c0678f0 0x7fbe0c069db0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009ac0 tx=0x7fbe20009190 comp rx=0 tx=0).stop 2024-05-27T00:27:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.630+0000 7fbe34c00640 1 -- 172.21.15.98:0/3417740133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 msgr2=0x7fbe301f6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.630+0000 7fbe34c00640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 0x7fbe301f6970 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbe280af040 tx=0x7fbe280197d0 comp rx=0 tx=0).stop 2024-05-27T00:27:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.630+0000 7fbe34c00640 1 -- 172.21.15.98:0/3417740133 shutdown_connections 2024-05-27T00:27:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.634+0000 7fbe34c00640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe30159c70 0x7fbe301f6970 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.634+0000 7fbe34c00640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe30070d30 0x7fbe300ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.634+0000 7fbe34c00640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbe0c0678f0 0x7fbe0c069db0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.634+0000 7fbe34c00640 1 --2- 172.21.15.98:0/3417740133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe3006f260 0x7fbe3014d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.634+0000 7fbe34c00640 1 -- 172.21.15.98:0/3417740133 >> 172.21.15.98:0/3417740133 conn(0x7fbe300cbe00 msgr2=0x7fbe300c7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.634+0000 7fbe34c00640 1 -- 172.21.15.98:0/3417740133 shutdown_connections 2024-05-27T00:27:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:54.634+0000 7fbe34c00640 1 -- 172.21.15.98:0/3417740133 wait complete. 2024-05-27T00:27:54.659 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:27:56.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f0853400640 1 Processor -- start 2024-05-27T00:27:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f0853400640 1 -- start start 2024-05-27T00:27:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f0853400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 0x7f084c0c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:56.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f0853400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f084c154e90 0x7f084c15ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:56.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f0853400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f084c15a950 con 0x7f084c070fe0 2024-05-27T00:27:56.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f0853400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f084c15e060 con 0x7f084c154e90 2024-05-27T00:27:56.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f0853400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f084c15e340 con 0x7f084c070c00 2024-05-27T00:27:56.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f0851400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f084c154e90 0x7f084c15ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:56.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f0850a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 0x7f084c0c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:56.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f084be00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f084c070fe0 0x7f084c154780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:37972/0 (socket says 172.21.15.98:37972) 2024-05-27T00:27:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.794+0000 7f084be00640 1 -- 172.21.15.98:0/3675853352 learned_addr learned my addr 172.21.15.98:0/3675853352 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:27:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.798+0000 7f0850a00640 1 -- 172.21.15.98:0/3675853352 >> v2:172.21.15.204:3300/0 conn(0x7f084c154e90 msgr2=0x7f084c15ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.798+0000 7f0850a00640 1 --2- 172.21.15.98:0/3675853352 >> v2:172.21.15.204:3300/0 conn(0x7f084c154e90 0x7f084c15ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.798+0000 7f0850a00640 1 -- 172.21.15.98:0/3675853352 >> v1:172.21.15.204:6789/0 conn(0x7f084c070fe0 legacy=0x7f084c154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.798+0000 7f0850a00640 1 -- 172.21.15.98:0/3675853352 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c15e620 con 0x7f084c070c00 2024-05-27T00:27:56.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.798+0000 7f0851400640 1 --2- 172.21.15.98:0/3675853352 >> v2:172.21.15.204:3300/0 conn(0x7f084c154e90 0x7f084c15ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:27:56.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.798+0000 7f0850a00640 1 --2- 172.21.15.98:0/3675853352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 0x7f084c0c8a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f08400097c0 tx=0x7f084001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6997dc353bc2d81 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:56.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.798+0000 7f084b400640 1 -- 172.21.15.98:0/3675853352 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08400b5070 con 0x7f084c070c00 2024-05-27T00:27:56.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.798+0000 7f084b400640 1 -- 172.21.15.98:0/3675853352 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08400a8990 con 0x7f084c070c00 2024-05-27T00:27:56.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.798+0000 7f084b400640 1 -- 172.21.15.98:0/3675853352 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08400a8d60 con 0x7f084c070c00 2024-05-27T00:27:56.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 -- 172.21.15.98:0/3675853352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 msgr2=0x7f084c0c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:56.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 --2- 172.21.15.98:0/3675853352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 0x7f084c0c8a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f08400097c0 tx=0x7f084001ec90 comp rx=0 tx=0).stop 2024-05-27T00:27:56.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 -- 172.21.15.98:0/3675853352 shutdown_connections 2024-05-27T00:27:56.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 --2- 172.21.15.98:0/3675853352 >> v2:172.21.15.204:3300/0 conn(0x7f084c154e90 0x7f084c15ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:56.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 --2- 172.21.15.98:0/3675853352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 0x7f084c0c8a00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:56.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 -- 172.21.15.98:0/3675853352 >> 172.21.15.98:0/3675853352 conn(0x7f084c072f90 msgr2=0x7f084c0733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:56.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 -- 172.21.15.98:0/3675853352 shutdown_connections 2024-05-27T00:27:56.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 -- 172.21.15.98:0/3675853352 wait complete. 2024-05-27T00:27:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 Processor -- start 2024-05-27T00:27:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 -- start start 2024-05-27T00:27:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 0x7f084c14e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:56.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f084c070fe0 0x7f084c159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f084c154e90 0x7f084c14f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f084c0c8f40 con 0x7f084c070fe0 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f084c15ebe0 con 0x7f084c154e90 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0853400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f084c0c90c0 con 0x7f084c070c00 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f084be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f084c070fe0 0x7f084c159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f084be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f084c070fe0 0x7f084c159f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37560/0 (socket says 172.21.15.98:37560) 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f084be00640 1 -- 172.21.15.98:0/1606170403 learned_addr learned my addr 172.21.15.98:0/1606170403 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0850a00640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 0x7f084c14e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0851400640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f084c154e90 0x7f084c14f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f084be00640 1 -- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 msgr2=0x7f084c14e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f084be00640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 0x7f084c14e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f084be00640 1 -- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f084c154e90 msgr2=0x7f084c14f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:56.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f084be00640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f084c154e90 0x7f084c14f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f084be00640 1 -- 172.21.15.98:0/1606170403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c14ead0 con 0x7f084c070fe0 2024-05-27T00:27:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.802+0000 7f0851400640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f084c154e90 0x7f084c14f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:27:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f084be00640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f084c070fe0 0x7f084c159f50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f083800c460 tx=0x7f083800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f0848c00640 1 -- 172.21.15.98:0/1606170403 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838017070 con 0x7f084c070fe0 2024-05-27T00:27:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f0848c00640 1 -- 172.21.15.98:0/1606170403 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0838005cf0 con 0x7f084c070fe0 2024-05-27T00:27:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f0848c00640 1 -- 172.21.15.98:0/1606170403 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838007290 con 0x7f084c070fe0 2024-05-27T00:27:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f0853400640 1 -- 172.21.15.98:0/1606170403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084c157900 con 0x7f084c070fe0 2024-05-27T00:27:56.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f0853400640 1 -- 172.21.15.98:0/1606170403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f084c157ec0 con 0x7f084c070fe0 2024-05-27T00:27:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f0853400640 1 -- 172.21.15.98:0/1606170403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084c075cb0 con 0x7f084c070fe0 2024-05-27T00:27:56.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f0848c00640 1 -- 172.21.15.98:0/1606170403 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f083801a070 con 0x7f084c070fe0 2024-05-27T00:27:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f0848c00640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f082c0678c0 0x7f082c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f0848c00640 1 -- 172.21.15.98:0/1606170403 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f0838085350 con 0x7f084c070fe0 2024-05-27T00:27:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.806+0000 7f0850a00640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f082c0678c0 0x7f082c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:56.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.810+0000 7f0850a00640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f082c0678c0 0x7f082c069d80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f08400b2000 tx=0x7f0840008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:56.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:56.810+0000 7f0848c00640 1 -- 172.21.15.98:0/1606170403 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f0838055390 con 0x7f084c070fe0 2024-05-27T00:27:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.002+0000 7f0853400640 1 -- 172.21.15.98:0/1606170403 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f084c0c9720 con 0x7f084c070fe0 2024-05-27T00:27:57.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.002+0000 7f0848c00640 1 -- 172.21.15.98:0/1606170403 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6006 (secure 0 0 0) 0x7f0838059690 con 0x7f084c070fe0 2024-05-27T00:27:57.007 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:57.007 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:55.271286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:57.008 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 53 2024-05-27T00:27:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 -- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f082c0678c0 msgr2=0x7f082c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f082c0678c0 0x7f082c069d80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f08400b2000 tx=0x7f0840008040 comp rx=0 tx=0).stop 2024-05-27T00:27:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 -- 172.21.15.98:0/1606170403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f084c070fe0 msgr2=0x7f084c159f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f084c070fe0 0x7f084c159f50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f083800c460 tx=0x7f083800c920 comp rx=0 tx=0).stop 2024-05-27T00:27:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 -- 172.21.15.98:0/1606170403 shutdown_connections 2024-05-27T00:27:57.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f082c0678c0 0x7f082c069d80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f084c154e90 0x7f084c14f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f084c070fe0 0x7f084c159f50 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 --2- 172.21.15.98:0/1606170403 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f084c070c00 0x7f084c14e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 -- 172.21.15.98:0/1606170403 >> 172.21.15.98:0/1606170403 conn(0x7f084c072f90 msgr2=0x7f084c15d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 -- 172.21.15.98:0/1606170403 shutdown_connections 2024-05-27T00:27:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.006+0000 7f0853400640 1 -- 172.21.15.98:0/1606170403 wait complete. 2024-05-27T00:27:57.029 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-05-27T00:27:57.029 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:27:57.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa81c00640 1 Processor -- start 2024-05-27T00:27:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa81c00640 1 -- start start 2024-05-27T00:27:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa81c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7faa7c152a50 0x7faa7c0709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa81c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa7c0cb7e0 0x7faa7c070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa81c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa7c15be40 con 0x7faa7c0b3ed0 2024-05-27T00:27:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa81c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7faa7c0c8a00 con 0x7faa7c0cb7e0 2024-05-27T00:27:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa81c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7faa7c0c50e0 con 0x7faa7c152a50 2024-05-27T00:27:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa7a000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa7c0cb7e0 0x7faa7c070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa7aa00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7faa7c152a50 0x7faa7c0709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa7a000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa7c0cb7e0 0x7faa7c070ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37566/0 (socket says 172.21.15.98:37566) 2024-05-27T00:27:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa7a000640 1 -- 172.21.15.98:0/2952378397 learned_addr learned my addr 172.21.15.98:0/2952378397 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa7b400640 1 --1- 172.21.15.98:0/2952378397 >> v1:172.21.15.98:6789/0 conn(0x7faa7c0b3ed0 0x7faa7c158750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36530/0 (socket says 172.21.15.98:36530) 2024-05-27T00:27:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa7a000640 1 -- 172.21.15.98:0/2952378397 >> v2:172.21.15.204:3300/0 conn(0x7faa7c152a50 msgr2=0x7faa7c0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa7a000640 1 --2- 172.21.15.98:0/2952378397 >> v2:172.21.15.204:3300/0 conn(0x7faa7c152a50 0x7faa7c0709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa7a000640 1 -- 172.21.15.98:0/2952378397 >> v1:172.21.15.98:6789/0 conn(0x7faa7c0b3ed0 legacy=0x7faa7c158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.162+0000 7faa7a000640 1 -- 172.21.15.98:0/2952378397 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7c0c8c50 con 0x7faa7c0cb7e0 2024-05-27T00:27:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa7a000640 1 --2- 172.21.15.98:0/2952378397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c0cb7e0 0x7faa7c070ee0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0c8b80 tx=0x7faa6801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c81eb1258da2265 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa79600640 1 -- 172.21.15.98:0/2952378397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa680ab660 con 0x7faa7c0cb7e0 2024-05-27T00:27:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa79600640 1 -- 172.21.15.98:0/2952378397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa680b0070 con 0x7faa7c0cb7e0 2024-05-27T00:27:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa79600640 1 -- 172.21.15.98:0/2952378397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa680aba20 con 0x7faa7c0cb7e0 2024-05-27T00:27:57.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa81c00640 1 -- 172.21.15.98:0/2952378397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c0cb7e0 msgr2=0x7faa7c070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa81c00640 1 --2- 172.21.15.98:0/2952378397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c0cb7e0 0x7faa7c070ee0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0c8b80 tx=0x7faa6801c460 comp rx=0 tx=0).stop 2024-05-27T00:27:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa81c00640 1 -- 172.21.15.98:0/2952378397 shutdown_connections 2024-05-27T00:27:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa81c00640 1 --2- 172.21.15.98:0/2952378397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c0cb7e0 0x7faa7c070ee0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa81c00640 1 --2- 172.21.15.98:0/2952378397 >> v2:172.21.15.204:3300/0 conn(0x7faa7c152a50 0x7faa7c0709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa81c00640 1 -- 172.21.15.98:0/2952378397 >> 172.21.15.98:0/2952378397 conn(0x7faa7c072f90 msgr2=0x7faa7c0733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa81c00640 1 -- 172.21.15.98:0/2952378397 shutdown_connections 2024-05-27T00:27:57.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.166+0000 7faa81c00640 1 -- 172.21.15.98:0/2952378397 wait complete. 2024-05-27T00:27:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 Processor -- start 2024-05-27T00:27:57.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 -- start start 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa7c0b3ed0 0x7faa7c0cc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa7c0cb7e0 0x7faa7c1f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c152a50 0x7faa7c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c152e30 con 0x7faa7c152a50 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c0cb570 con 0x7faa7c0cb7e0 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faa7c0c9c10 con 0x7faa7c0b3ed0 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c152a50 0x7faa7c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c152a50 0x7faa7c15a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37572/0 (socket says 172.21.15.98:37572) 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7b400640 1 -- 172.21.15.98:0/801459770 learned_addr learned my addr 172.21.15.98:0/801459770 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7a000640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa7c0cb7e0 0x7faa7c1f5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7aa00640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa7c0b3ed0 0x7faa7c0cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:57.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7b400640 1 -- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa7c0b3ed0 msgr2=0x7faa7c0cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7b400640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa7c0b3ed0 0x7faa7c0cc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7b400640 1 -- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa7c0cb7e0 msgr2=0x7faa7c1f5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7b400640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa7c0cb7e0 0x7faa7c1f5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7b400640 1 -- 172.21.15.98:0/801459770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7c06b610 con 0x7faa7c152a50 2024-05-27T00:27:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa7b400640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c152a50 0x7faa7c15a4e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7faa7400c7d0 tx=0x7faa7400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa72a00640 1 -- 172.21.15.98:0/801459770 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74017070 con 0x7faa7c152a50 2024-05-27T00:27:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa72a00640 1 -- 172.21.15.98:0/801459770 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa7400a640 con 0x7faa7c152a50 2024-05-27T00:27:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 -- 172.21.15.98:0/801459770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa7c06d220 con 0x7faa7c152a50 2024-05-27T00:27:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa72a00640 1 -- 172.21.15.98:0/801459770 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7400a9f0 con 0x7faa7c152a50 2024-05-27T00:27:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 -- 172.21.15.98:0/801459770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa7c1477f0 con 0x7faa7c152a50 2024-05-27T00:27:57.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.170+0000 7faa81c00640 1 -- 172.21.15.98:0/801459770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa7c075cb0 con 0x7faa7c152a50 2024-05-27T00:27:57.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.174+0000 7faa72a00640 1 -- 172.21.15.98:0/801459770 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faa7401a070 con 0x7faa7c152a50 2024-05-27T00:27:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.174+0000 7faa72a00640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa5c0678f0 0x7faa5c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.174+0000 7faa72a00640 1 -- 172.21.15.98:0/801459770 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7faa74085630 con 0x7faa7c152a50 2024-05-27T00:27:57.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.174+0000 7faa7aa00640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa5c0678f0 0x7faa5c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:57.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.174+0000 7faa7aa00640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa5c0678f0 0x7faa5c069db0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009ac0 tx=0x7faa6c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:57.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.182+0000 7faa72a00640 1 -- 172.21.15.98:0/801459770 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7faa74055700 con 0x7faa7c152a50 2024-05-27T00:27:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 -- 172.21.15.98:0/801459770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa7c152280 con 0x7faa7c152a50 2024-05-27T00:27:57.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa72a00640 1 -- 172.21.15.98:0/801459770 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6006 (secure 0 0 0) 0x7faa74059a00 con 0x7faa7c152a50 2024-05-27T00:27:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:57.378 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:55.271286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 53 2024-05-27T00:27:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 -- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa5c0678f0 msgr2=0x7faa5c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa5c0678f0 0x7faa5c069db0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009ac0 tx=0x7faa6c009190 comp rx=0 tx=0).stop 2024-05-27T00:27:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 -- 172.21.15.98:0/801459770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c152a50 msgr2=0x7faa7c15a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c152a50 0x7faa7c15a4e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7faa7400c7d0 tx=0x7faa7400cc90 comp rx=0 tx=0).stop 2024-05-27T00:27:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 -- 172.21.15.98:0/801459770 shutdown_connections 2024-05-27T00:27:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c152a50 0x7faa7c15a4e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa5c0678f0 0x7faa5c069db0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa7c0cb7e0 0x7faa7c1f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 --2- 172.21.15.98:0/801459770 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa7c0b3ed0 0x7faa7c0cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:57.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 -- 172.21.15.98:0/801459770 >> 172.21.15.98:0/801459770 conn(0x7faa7c072f90 msgr2=0x7faa7c0c6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:57.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 -- 172.21.15.98:0/801459770 shutdown_connections 2024-05-27T00:27:57.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:57.374+0000 7faa81c00640 1 -- 172.21.15.98:0/801459770 wait complete. 2024-05-27T00:27:57.400 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 53, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 53, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5943": { "addr": "172.21.15.98:6834/2148124202", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 2148124202, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 2148124202, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5943, "incarnation": 53, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:27:55.271286+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5943 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:27:59.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:27:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefdb400640 1 Processor -- start 2024-05-27T00:27:59.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefdb400640 1 -- start start 2024-05-27T00:27:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefdb400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fefd40c32d0 0x7fefd40ca480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefdb400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefd41583f0 con 0x7fefd4158010 2024-05-27T00:27:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefdb400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefd40c94b0 con 0x7fefd415def0 2024-05-27T00:27:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefdb400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fefd406f660 con 0x7fefd40c32d0 2024-05-27T00:27:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefcbe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fefd40c32d0 0x7fefd40ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefcbe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fefd40c32d0 0x7fefd40ca480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37584/0 (socket says 172.21.15.98:37584) 2024-05-27T00:27:59.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefcbe00640 1 -- 172.21.15.98:0/721482960 learned_addr learned my addr 172.21.15.98:0/721482960 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefcbe00640 1 -- 172.21.15.98:0/721482960 >> v1:172.21.15.204:6789/0 conn(0x7fefd415def0 legacy=0x7fefd415e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:27:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefcbe00640 1 -- 172.21.15.98:0/721482960 >> v1:172.21.15.98:6789/0 conn(0x7fefd4158010 legacy=0x7fefd406faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:59.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.534+0000 7fefcbe00640 1 -- 172.21.15.98:0/721482960 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd40c9790 con 0x7fefd40c32d0 2024-05-27T00:27:59.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefcbe00640 1 --2- 172.21.15.98:0/721482960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd40c32d0 0x7fefd40ca480 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009870 tx=0x7fefbc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792f2846a93fbe13 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefd3e00640 1 -- 172.21.15.98:0/721482960 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc0ab430 con 0x7fefd40c32d0 2024-05-27T00:27:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefd3e00640 1 -- 172.21.15.98:0/721482960 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefbc0b0070 con 0x7fefd40c32d0 2024-05-27T00:27:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefd3e00640 1 -- 172.21.15.98:0/721482960 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc0ab7f0 con 0x7fefd40c32d0 2024-05-27T00:27:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefdb400640 1 -- 172.21.15.98:0/721482960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd40c32d0 msgr2=0x7fefd40ca480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:59.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefdb400640 1 --2- 172.21.15.98:0/721482960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd40c32d0 0x7fefd40ca480 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009870 tx=0x7fefbc01c520 comp rx=0 tx=0).stop 2024-05-27T00:27:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefdb400640 1 -- 172.21.15.98:0/721482960 shutdown_connections 2024-05-27T00:27:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefdb400640 1 --2- 172.21.15.98:0/721482960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd40c32d0 0x7fefd40ca480 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefdb400640 1 -- 172.21.15.98:0/721482960 >> 172.21.15.98:0/721482960 conn(0x7fefd40ce000 msgr2=0x7fefd40ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefdb400640 1 -- 172.21.15.98:0/721482960 shutdown_connections 2024-05-27T00:27:59.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.538+0000 7fefdb400640 1 -- 172.21.15.98:0/721482960 wait complete. 2024-05-27T00:27:59.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefdb400640 1 Processor -- start 2024-05-27T00:27:59.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefdb400640 1 -- start start 2024-05-27T00:27:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefdb400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fefd40c32d0 0x7fefd415bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefdb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd4158010 0x7fefd415aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefdb400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd415def0 0x7fefd415b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefdb400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fefd4070380 con 0x7fefd4158010 2024-05-27T00:27:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefdb400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fefd4070200 con 0x7fefd415def0 2024-05-27T00:27:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefdb400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fefd40c5ce0 con 0x7fefd40c32d0 2024-05-27T00:27:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefcbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd4158010 0x7fefd415aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefcbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd4158010 0x7fefd415aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37600/0 (socket says 172.21.15.98:37600) 2024-05-27T00:27:59.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefcbe00640 1 -- 172.21.15.98:0/3492145748 learned_addr learned my addr 172.21.15.98:0/3492145748 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefd9400640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd415def0 0x7fefd415b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefcbe00640 1 -- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fefd40c32d0 msgr2=0x7fefd415bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefcbe00640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fefd40c32d0 0x7fefd415bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefcbe00640 1 -- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd415def0 msgr2=0x7fefd415b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefcbe00640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd415def0 0x7fefd415b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefcbe00640 1 -- 172.21.15.98:0/3492145748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd41567d0 con 0x7fefd4158010 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefd9400640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd415def0 0x7fefd415b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefcbe00640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd4158010 0x7fefd415aee0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0ad040 tx=0x7fefbc0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefd1600640 1 -- 172.21.15.98:0/3492145748 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc0b4420 con 0x7fefd4158010 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefd1600640 1 -- 172.21.15.98:0/3492145748 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefbc0bb070 con 0x7fefd4158010 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefd1600640 1 -- 172.21.15.98:0/3492145748 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc0b47e0 con 0x7fefd4158010 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefdb400640 1 -- 172.21.15.98:0/3492145748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd41f3940 con 0x7fefd4158010 2024-05-27T00:27:59.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.542+0000 7fefdb400640 1 -- 172.21.15.98:0/3492145748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd4158a60 con 0x7fefd4158010 2024-05-27T00:27:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.546+0000 7fefdb400640 1 -- 172.21.15.98:0/3492145748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefd4075cb0 con 0x7fefd4158010 2024-05-27T00:27:59.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.546+0000 7fefd1600640 1 -- 172.21.15.98:0/3492145748 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fefbc0b4980 con 0x7fefd4158010 2024-05-27T00:27:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.546+0000 7fefd1600640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fefb00678c0 0x7fefb0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:27:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.546+0000 7fefd1600640 1 -- 172.21.15.98:0/3492145748 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7fefbc11fff0 con 0x7fefd4158010 2024-05-27T00:27:59.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.546+0000 7fefd8a00640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fefb00678c0 0x7fefb0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:27:59.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.546+0000 7fefd8a00640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fefb00678c0 0x7fefb0069d80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fefc4000f70 tx=0x7fefc40079e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:27:59.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.554+0000 7fefd1600640 1 -- 172.21.15.98:0/3492145748 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fefbc016110 con 0x7fefd4158010 2024-05-27T00:27:59.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 -- 172.21.15.98:0/3492145748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefd40c4e50 con 0x7fefd4158010 2024-05-27T00:27:59.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefd1600640 1 -- 172.21.15.98:0/3492145748 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6006 (secure 0 0 0) 0x7fefbc0f0710 con 0x7fefd4158010 2024-05-27T00:27:59.751 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:27:59.751 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:55.271286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:27:59.751 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 53 2024-05-27T00:27:59.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 -- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fefb00678c0 msgr2=0x7fefb0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:59.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fefb00678c0 0x7fefb0069d80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fefc4000f70 tx=0x7fefc40079e0 comp rx=0 tx=0).stop 2024-05-27T00:27:59.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 -- 172.21.15.98:0/3492145748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd4158010 msgr2=0x7fefd415aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:27:59.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd4158010 0x7fefd415aee0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0ad040 tx=0x7fefbc0192c0 comp rx=0 tx=0).stop 2024-05-27T00:27:59.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 -- 172.21.15.98:0/3492145748 shutdown_connections 2024-05-27T00:27:59.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fefb00678c0 0x7fefb0069d80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:59.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefd415def0 0x7fefd415b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefd4158010 0x7fefd415aee0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 --2- 172.21.15.98:0/3492145748 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fefd40c32d0 0x7fefd415bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:27:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 -- 172.21.15.98:0/3492145748 >> 172.21.15.98:0/3492145748 conn(0x7fefd40ce000 msgr2=0x7fefd414c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:27:59.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 -- 172.21.15.98:0/3492145748 shutdown_connections 2024-05-27T00:27:59.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:27:59.746+0000 7fefdb400640 1 -- 172.21.15.98:0/3492145748 wait complete. 2024-05-27T00:27:59.773 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:28:01.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.926+0000 7f7d49e00640 1 Processor -- start 2024-05-27T00:28:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.926+0000 7f7d49e00640 1 -- start start 2024-05-27T00:28:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d49e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d440ce800 0x7f7d44158590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d49e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7d440c8250 0x7f7d4406f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d49e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d440c4be0 con 0x7f7d44152700 2024-05-27T00:28:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d49e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7d440ce370 con 0x7f7d440ce800 2024-05-27T00:28:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d49e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7d440c8eb0 con 0x7f7d440c8250 2024-05-27T00:28:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d42000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d440ce800 0x7f7d44158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d42000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d440ce800 0x7f7d44158590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37604/0 (socket says 172.21.15.98:37604) 2024-05-27T00:28:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d42000640 1 -- 172.21.15.98:0/871259572 learned_addr learned my addr 172.21.15.98:0/871259572 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d43400640 1 --2- 172.21.15.98:0/871259572 >> v2:172.21.15.204:3300/0 conn(0x7f7d440c8250 0x7f7d4406f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d42000640 1 -- 172.21.15.98:0/871259572 >> v2:172.21.15.204:3300/0 conn(0x7f7d440c8250 msgr2=0x7f7d4406f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d42000640 1 --2- 172.21.15.98:0/871259572 >> v2:172.21.15.204:3300/0 conn(0x7f7d440c8250 0x7f7d4406f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d42000640 1 -- 172.21.15.98:0/871259572 >> v1:172.21.15.98:6789/0 conn(0x7f7d44152700 legacy=0x7f7d440c3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d42000640 1 -- 172.21.15.98:0/871259572 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d440c4e50 con 0x7f7d440ce800 2024-05-27T00:28:01.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.930+0000 7f7d41600640 1 -- 172.21.15.98:0/871259572 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388521002 0 0) 0x7f7d440c4be0 con 0x7f7d44152700 2024-05-27T00:28:01.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d42000640 1 --2- 172.21.15.98:0/871259572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d440ce800 0x7f7d44158590 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7d440c5440 tx=0x7f7d3001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c1901d8ab473b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d41600640 1 -- 172.21.15.98:0/871259572 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d300b0070 con 0x7f7d440ce800 2024-05-27T00:28:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d41600640 1 -- 172.21.15.98:0/871259572 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d300ab600 con 0x7f7d440ce800 2024-05-27T00:28:01.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d41600640 1 -- 172.21.15.98:0/871259572 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d300ab9d0 con 0x7f7d440ce800 2024-05-27T00:28:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d49e00640 1 -- 172.21.15.98:0/871259572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d440ce800 msgr2=0x7f7d44158590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d49e00640 1 --2- 172.21.15.98:0/871259572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d440ce800 0x7f7d44158590 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7d440c5440 tx=0x7f7d3001c490 comp rx=0 tx=0).stop 2024-05-27T00:28:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d49e00640 1 -- 172.21.15.98:0/871259572 shutdown_connections 2024-05-27T00:28:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d49e00640 1 --2- 172.21.15.98:0/871259572 >> v2:172.21.15.204:3300/0 conn(0x7f7d440c8250 0x7f7d4406f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d49e00640 1 --2- 172.21.15.98:0/871259572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d440ce800 0x7f7d44158590 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d49e00640 1 -- 172.21.15.98:0/871259572 >> 172.21.15.98:0/871259572 conn(0x7f7d4406d680 msgr2=0x7f7d4406da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d49e00640 1 -- 172.21.15.98:0/871259572 shutdown_connections 2024-05-27T00:28:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.934+0000 7f7d49e00640 1 -- 172.21.15.98:0/871259572 wait complete. 2024-05-27T00:28:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d49e00640 1 Processor -- start 2024-05-27T00:28:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d49e00640 1 -- start start 2024-05-27T00:28:01.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d49e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d440c8250 0x7f7d44161c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:01.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d49e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d440ce800 0x7f7d441577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:01.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d49e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d44152700 0x7f7d441f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:01.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d49e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d44152c60 con 0x7f7d44152700 2024-05-27T00:28:01.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d49e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7d44152ae0 con 0x7f7d440ce800 2024-05-27T00:28:01.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d49e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7d440c8a70 con 0x7f7d440c8250 2024-05-27T00:28:01.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d43400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d44152700 0x7f7d441f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:01.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d43400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d44152700 0x7f7d441f7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37608/0 (socket says 172.21.15.98:37608) 2024-05-27T00:28:01.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d43400640 1 -- 172.21.15.98:0/3532712034 learned_addr learned my addr 172.21.15.98:0/3532712034 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:01.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d42000640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d440ce800 0x7f7d441577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:01.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.938+0000 7f7d42a00640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d440c8250 0x7f7d44161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:01.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d43400640 1 -- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d440c8250 msgr2=0x7f7d44161c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:01.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d43400640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d440c8250 0x7f7d44161c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:01.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d43400640 1 -- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d440ce800 msgr2=0x7f7d441577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:01.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d43400640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d440ce800 0x7f7d441577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:01.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d43400640 1 -- 172.21.15.98:0/3532712034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d441f6b90 con 0x7f7d44152700 2024-05-27T00:28:01.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d42000640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d440ce800 0x7f7d441577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d43400640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d44152700 0x7f7d441f7070 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c007e70 tx=0x7f7d3c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d3b400640 1 -- 172.21.15.98:0/3532712034 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c014620 con 0x7f7d44152700 2024-05-27T00:28:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d3b400640 1 -- 172.21.15.98:0/3532712034 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d3c019070 con 0x7f7d44152700 2024-05-27T00:28:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d49e00640 1 -- 172.21.15.98:0/3532712034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d440cc010 con 0x7f7d44152700 2024-05-27T00:28:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d3b400640 1 -- 172.21.15.98:0/3532712034 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c014a20 con 0x7f7d44152700 2024-05-27T00:28:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d49e00640 1 -- 172.21.15.98:0/3532712034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d44161960 con 0x7f7d44152700 2024-05-27T00:28:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d49e00640 1 -- 172.21.15.98:0/3532712034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d441f6d60 con 0x7f7d44152700 2024-05-27T00:28:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.942+0000 7f7d3b400640 1 -- 172.21.15.98:0/3532712034 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7d3c014bc0 con 0x7f7d44152700 2024-05-27T00:28:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.946+0000 7f7d3b400640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d240678f0 0x7f7d24069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.946+0000 7f7d3b400640 1 -- 172.21.15.98:0/3532712034 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f7d3c086980 con 0x7f7d44152700 2024-05-27T00:28:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.946+0000 7f7d42a00640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d240678f0 0x7f7d24069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:01.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.946+0000 7f7d42a00640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d240678f0 0x7f7d24069db0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7d34000f70 tx=0x7f7d34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:01.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:01.950+0000 7f7d3b400640 1 -- 172.21.15.98:0/3532712034 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7d3c011260 con 0x7f7d44152700 2024-05-27T00:28:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 -- 172.21.15.98:0/3532712034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d44075cb0 con 0x7f7d44152700 2024-05-27T00:28:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d3b400640 1 -- 172.21.15.98:0/3532712034 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6006 (secure 0 0 0) 0x7f7d3c0570a0 con 0x7f7d44152700 2024-05-27T00:28:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:02.175 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:55.271286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 53 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 -- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d240678f0 msgr2=0x7f7d24069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d240678f0 0x7f7d24069db0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7d34000f70 tx=0x7f7d34009040 comp rx=0 tx=0).stop 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 -- 172.21.15.98:0/3532712034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d44152700 msgr2=0x7f7d441f7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d44152700 0x7f7d441f7070 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c007e70 tx=0x7f7d3c00e4b0 comp rx=0 tx=0).stop 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 -- 172.21.15.98:0/3532712034 shutdown_connections 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d44152700 0x7f7d441f7070 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d240678f0 0x7f7d24069db0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d440ce800 0x7f7d441577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 --2- 172.21.15.98:0/3532712034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d440c8250 0x7f7d44161c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.170+0000 7f7d49e00640 1 -- 172.21.15.98:0/3532712034 >> 172.21.15.98:0/3532712034 conn(0x7f7d4406d680 msgr2=0x7f7d440ca5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.174+0000 7f7d49e00640 1 -- 172.21.15.98:0/3532712034 shutdown_connections 2024-05-27T00:28:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:02.174+0000 7f7d49e00640 1 -- 172.21.15.98:0/3532712034 wait complete. 2024-05-27T00:28:02.199 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:28:04.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f277d600640 1 Processor -- start 2024-05-27T00:28:04.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f277d600640 1 -- start start 2024-05-27T00:28:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f277d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780731f0 0x7f27780c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f277d600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27780c79f0 0x7f2778157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f277d600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27780c17d0 con 0x7f27780735d0 2024-05-27T00:28:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f277d600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2778071990 con 0x7f27780c79f0 2024-05-27T00:28:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f277d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2778070e10 con 0x7f27780731f0 2024-05-27T00:28:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f2777400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27780c79f0 0x7f2778157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f2777400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27780c79f0 0x7f2778157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55700/0 (socket says 172.21.15.98:55700) 2024-05-27T00:28:04.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f2777400640 1 -- 172.21.15.98:0/2170068508 learned_addr learned my addr 172.21.15.98:0/2170068508 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f2776a00640 1 --2- 172.21.15.98:0/2170068508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780731f0 0x7f27780c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f2777400640 1 -- 172.21.15.98:0/2170068508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780731f0 msgr2=0x7f27780c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f2777400640 1 --2- 172.21.15.98:0/2170068508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780731f0 0x7f27780c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f2777400640 1 -- 172.21.15.98:0/2170068508 >> v1:172.21.15.204:6789/0 conn(0x7f27780735d0 legacy=0x7f2778151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f2777400640 1 -- 172.21.15.98:0/2170068508 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27780710f0 con 0x7f27780c79f0 2024-05-27T00:28:04.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.350+0000 7f2776a00640 1 --2- 172.21.15.98:0/2170068508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780731f0 0x7f27780c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:04.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.354+0000 7f2777400640 1 --2- 172.21.15.98:0/2170068508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c79f0 0x7f2778157970 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2770009870 tx=0x7f277001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fb77e8090ec673 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:04.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.354+0000 7f2775600640 1 -- 172.21.15.98:0/2170068508 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27700b5070 con 0x7f27780c79f0 2024-05-27T00:28:04.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.354+0000 7f2775600640 1 -- 172.21.15.98:0/2170068508 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27700a8a50 con 0x7f27780c79f0 2024-05-27T00:28:04.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.354+0000 7f2775600640 1 -- 172.21.15.98:0/2170068508 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27700a8e20 con 0x7f27780c79f0 2024-05-27T00:28:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.354+0000 7f277d600640 1 -- 172.21.15.98:0/2170068508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c79f0 msgr2=0x7f2778157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.354+0000 7f277d600640 1 --2- 172.21.15.98:0/2170068508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c79f0 0x7f2778157970 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f2770009870 tx=0x7f277001ed50 comp rx=0 tx=0).stop 2024-05-27T00:28:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.354+0000 7f277d600640 1 -- 172.21.15.98:0/2170068508 shutdown_connections 2024-05-27T00:28:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.354+0000 7f277d600640 1 --2- 172.21.15.98:0/2170068508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780c79f0 0x7f2778157970 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.354+0000 7f277d600640 1 --2- 172.21.15.98:0/2170068508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780731f0 0x7f27780c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.358+0000 7f277d600640 1 -- 172.21.15.98:0/2170068508 >> 172.21.15.98:0/2170068508 conn(0x7f2778067530 msgr2=0x7f277806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.358+0000 7f277d600640 1 -- 172.21.15.98:0/2170068508 shutdown_connections 2024-05-27T00:28:04.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.358+0000 7f277d600640 1 -- 172.21.15.98:0/2170068508 wait complete. 2024-05-27T00:28:04.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.358+0000 7f277d600640 1 Processor -- start 2024-05-27T00:28:04.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.358+0000 7f277d600640 1 -- start start 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f277d600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780731f0 0x7f277814b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f277d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780735d0 0x7f27781f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f277d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27780c79f0 0x7f277814fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f277d600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27780c8400 con 0x7f27780731f0 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f277d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f27780716b0 con 0x7f27780c79f0 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f277d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f27780c13f0 con 0x7f27780735d0 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780731f0 0x7f277814b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780731f0 0x7f277814b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55704/0 (socket says 172.21.15.98:55704) 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776a00640 1 -- 172.21.15.98:0/1324099374 learned_addr learned my addr 172.21.15.98:0/1324099374 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776000640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780735d0 0x7f27781f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2777400640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27780c79f0 0x7f277814fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:04.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776a00640 1 -- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780735d0 msgr2=0x7f27781f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:04.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776a00640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780735d0 0x7f27781f2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:04.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776a00640 1 -- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27780c79f0 msgr2=0x7f277814fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:04.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776a00640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27780c79f0 0x7f277814fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:04.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776a00640 1 -- 172.21.15.98:0/1324099374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27781f2ad0 con 0x7f27780731f0 2024-05-27T00:28:04.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776000640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780735d0 0x7f27781f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:04.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f2776a00640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780731f0 0x7f277814b2e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2768009fc0 tx=0x7f276800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:04.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f276ea00640 1 -- 172.21.15.98:0/1324099374 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276801a070 con 0x7f27780731f0 2024-05-27T00:28:04.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f276ea00640 1 -- 172.21.15.98:0/1324099374 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2768003bc0 con 0x7f27780731f0 2024-05-27T00:28:04.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f276ea00640 1 -- 172.21.15.98:0/1324099374 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2768004030 con 0x7f27780731f0 2024-05-27T00:28:04.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f277d600640 1 -- 172.21.15.98:0/1324099374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f277814db60 con 0x7f27780731f0 2024-05-27T00:28:04.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f277d600640 1 -- 172.21.15.98:0/1324099374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27781ee850 con 0x7f27780731f0 2024-05-27T00:28:04.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.362+0000 7f277d600640 1 -- 172.21.15.98:0/1324099374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27781ee640 con 0x7f27780731f0 2024-05-27T00:28:04.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.366+0000 7f276ea00640 1 -- 172.21.15.98:0/1324099374 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f276801d070 con 0x7f27780731f0 2024-05-27T00:28:04.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.366+0000 7f276ea00640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 0x7f2758069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:04.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.366+0000 7f276ea00640 1 -- 172.21.15.98:0/1324099374 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f2768086200 con 0x7f27780731f0 2024-05-27T00:28:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.366+0000 7f2776000640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 0x7f2758069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:04.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.370+0000 7f2776000640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 0x7f2758069d80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f277815bb40 tx=0x7f2764009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:04.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.374+0000 7f276ea00640 1 -- 172.21.15.98:0/1324099374 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f2768010070 con 0x7f27780731f0 2024-05-27T00:28:04.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.590+0000 7f277d600640 1 -- 172.21.15.98:0/1324099374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2778075cb0 con 0x7f27780731f0 2024-05-27T00:28:04.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f276ea00640 1 -- 172.21.15.98:0/1324099374 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6006 (secure 0 0 0) 0x7f2768056370 con 0x7f27780731f0 2024-05-27T00:28:04.597 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:04.597 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:27:55.271286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:04.597 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 53 2024-05-27T00:28:04.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 -- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 msgr2=0x7f2758069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:04.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 0x7f2758069d80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f277815bb40 tx=0x7f2764009040 comp rx=0 tx=0).stop 2024-05-27T00:28:04.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 -- 172.21.15.98:0/1324099374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780731f0 msgr2=0x7f277814b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:04.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780731f0 0x7f277814b2e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2768009fc0 tx=0x7f276800d560 comp rx=0 tx=0).stop 2024-05-27T00:28:04.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 -- 172.21.15.98:0/1324099374 shutdown_connections 2024-05-27T00:28:04.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27580678c0 0x7f2758069d80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:04.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27780c79f0 0x7f277814fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:04.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27780735d0 0x7f27781f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:04.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 --2- 172.21.15.98:0/1324099374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27780731f0 0x7f277814b2e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:04.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 -- 172.21.15.98:0/1324099374 >> 172.21.15.98:0/1324099374 conn(0x7f2778067530 msgr2=0x7f277814e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:04.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 -- 172.21.15.98:0/1324099374 shutdown_connections 2024-05-27T00:28:04.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:04.594+0000 7f277d600640 1 -- 172.21.15.98:0/1324099374 wait complete. 2024-05-27T00:28:04.627 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:28:06.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:06.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd8aec00640 1 Processor -- start 2024-05-27T00:28:06.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd8aec00640 1 -- start start 2024-05-27T00:28:06.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd8aec00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8a8070fe0 0x7fd8a8154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:06.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd8aec00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a815a950 con 0x7fd8a80cb570 2024-05-27T00:28:06.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd8aec00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a815e060 con 0x7fd8a8070c00 2024-05-27T00:28:06.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd8aec00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd8a815e340 con 0x7fd8a8070fe0 2024-05-27T00:28:06.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd89f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd8a8070fe0 0x7fd8a8154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:06.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd8acc00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd8a80cb570 0x7fd8a815ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38800/0 (socket says 172.21.15.98:38800) 2024-05-27T00:28:06.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd8acc00640 1 -- 172.21.15.98:0/1034020092 learned_addr learned my addr 172.21.15.98:0/1034020092 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:06.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd89f400640 1 -- 172.21.15.98:0/1034020092 >> v1:172.21.15.204:6789/0 conn(0x7fd8a8070c00 legacy=0x7fd8a80c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:06.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.762+0000 7fd89f400640 1 -- 172.21.15.98:0/1034020092 >> v1:172.21.15.98:6789/0 conn(0x7fd8a80cb570 legacy=0x7fd8a815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:06.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8a7400640 1 -- 172.21.15.98:0/1034020092 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 934175065 0 0) 0x7fd8a815a950 con 0x7fd8a80cb570 2024-05-27T00:28:06.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd89f400640 1 -- 172.21.15.98:0/1034020092 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a815e620 con 0x7fd8a8070fe0 2024-05-27T00:28:06.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd89f400640 1 --2- 172.21.15.98:0/1034020092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070fe0 0x7fd8a8154780 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd8900097c0 tx=0x7fd89001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db8222850b4f958 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:06.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8a7400640 1 -- 172.21.15.98:0/1034020092 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8900b5070 con 0x7fd8a8070fe0 2024-05-27T00:28:06.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8a7400640 1 -- 172.21.15.98:0/1034020092 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8900a8990 con 0x7fd8a8070fe0 2024-05-27T00:28:06.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8a7400640 1 -- 172.21.15.98:0/1034020092 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8900a8d60 con 0x7fd8a8070fe0 2024-05-27T00:28:06.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8aec00640 1 -- 172.21.15.98:0/1034020092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070fe0 msgr2=0x7fd8a8154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:06.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8aec00640 1 --2- 172.21.15.98:0/1034020092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070fe0 0x7fd8a8154780 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd8900097c0 tx=0x7fd89001ec90 comp rx=0 tx=0).stop 2024-05-27T00:28:06.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8aec00640 1 -- 172.21.15.98:0/1034020092 shutdown_connections 2024-05-27T00:28:06.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8aec00640 1 --2- 172.21.15.98:0/1034020092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070fe0 0x7fd8a8154780 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:06.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8aec00640 1 -- 172.21.15.98:0/1034020092 >> 172.21.15.98:0/1034020092 conn(0x7fd8a8072f90 msgr2=0x7fd8a80733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:06.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8aec00640 1 -- 172.21.15.98:0/1034020092 shutdown_connections 2024-05-27T00:28:06.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.766+0000 7fd8aec00640 1 -- 172.21.15.98:0/1034020092 wait complete. 2024-05-27T00:28:06.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8aec00640 1 Processor -- start 2024-05-27T00:28:06.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8aec00640 1 -- start start 2024-05-27T00:28:06.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8aec00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070c00 0x7fd8a80ce720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:06.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8aec00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a8070fe0 0x7fd8a815cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:06.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8aec00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a80cb570 0x7fd8a81e4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8aec00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a80c9110 con 0x7fd8a8070c00 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8aec00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a815ebe0 con 0x7fd8a80cb570 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8aec00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a815b560 con 0x7fd8a8070fe0 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070c00 0x7fd8a80ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070c00 0x7fd8a80ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55714/0 (socket says 172.21.15.98:55714) 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a7e00640 1 -- 172.21.15.98:0/2707090532 learned_addr learned my addr 172.21.15.98:0/2707090532 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8acc00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a80cb570 0x7fd8a81e4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd89f400640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a8070fe0 0x7fd8a815cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a7e00640 1 -- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a8070fe0 msgr2=0x7fd8a815cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a7e00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a8070fe0 0x7fd8a815cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a7e00640 1 -- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a80cb570 msgr2=0x7fd8a81e4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a7e00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a80cb570 0x7fd8a81e4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:06.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a7e00640 1 -- 172.21.15.98:0/2707090532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a80cd980 con 0x7fd8a8070c00 2024-05-27T00:28:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd89f400640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a8070fe0 0x7fd8a815cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a7e00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070c00 0x7fd8a80ce720 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd89800c530 tx=0x7fd89800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a4c00640 1 -- 172.21.15.98:0/2707090532 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898016020 con 0x7fd8a8070c00 2024-05-27T00:28:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a4c00640 1 -- 172.21.15.98:0/2707090532 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd898005220 con 0x7fd8a8070c00 2024-05-27T00:28:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8a4c00640 1 -- 172.21.15.98:0/2707090532 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898007030 con 0x7fd8a8070c00 2024-05-27T00:28:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.770+0000 7fd8aec00640 1 -- 172.21.15.98:0/2707090532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a81f0fe0 con 0x7fd8a8070c00 2024-05-27T00:28:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.774+0000 7fd8aec00640 1 -- 172.21.15.98:0/2707090532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a81f3d40 con 0x7fd8a8070c00 2024-05-27T00:28:06.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.774+0000 7fd8aec00640 1 -- 172.21.15.98:0/2707090532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a8075cb0 con 0x7fd8a8070c00 2024-05-27T00:28:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.774+0000 7fd8a4c00640 1 -- 172.21.15.98:0/2707090532 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd8980071d0 con 0x7fd8a8070c00 2024-05-27T00:28:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.774+0000 7fd8a4c00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8840678f0 0x7fd884069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.774+0000 7fd8a4c00640 1 -- 172.21.15.98:0/2707090532 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7fd898085980 con 0x7fd8a8070c00 2024-05-27T00:28:06.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.774+0000 7fd89f400640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8840678f0 0x7fd884069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:06.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.774+0000 7fd89f400640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8840678f0 0x7fd884069db0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd8900b2000 tx=0x7fd890008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:06.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.778+0000 7fd8a4c00640 1 -- 172.21.15.98:0/2707090532 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd898020020 con 0x7fd8a8070c00 2024-05-27T00:28:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.970+0000 7fd8aec00640 1 -- 172.21.15.98:0/2707090532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8a80c9720 con 0x7fd8a8070c00 2024-05-27T00:28:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.970+0000 7fd8a4c00640 1 -- 172.21.15.98:0/2707090532 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+6010 (secure 0 0 0) 0x7fd8980074c0 con 0x7fd8a8070c00 2024-05-27T00:28:06.976 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:06.976 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:05.555961+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 54 2024-05-27T00:28:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 -- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8840678f0 msgr2=0x7fd884069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8840678f0 0x7fd884069db0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd8900b2000 tx=0x7fd890008040 comp rx=0 tx=0).stop 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 -- 172.21.15.98:0/2707090532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070c00 msgr2=0x7fd8a80ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070c00 0x7fd8a80ce720 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd89800c530 tx=0x7fd89800c9f0 comp rx=0 tx=0).stop 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 -- 172.21.15.98:0/2707090532 shutdown_connections 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8840678f0 0x7fd884069db0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a80cb570 0x7fd8a81e4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a8070fe0 0x7fd8a815cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 --2- 172.21.15.98:0/2707090532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8a8070c00 0x7fd8a80ce720 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 -- 172.21.15.98:0/2707090532 >> 172.21.15.98:0/2707090532 conn(0x7fd8a8072f90 msgr2=0x7fd8a8073370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 -- 172.21.15.98:0/2707090532 shutdown_connections 2024-05-27T00:28:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:06.974+0000 7fd8aec00640 1 -- 172.21.15.98:0/2707090532 wait complete. 2024-05-27T00:28:06.996 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:28:08.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:09.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80da200640 1 Processor -- start 2024-05-27T00:28:09.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80da200640 1 -- start start 2024-05-27T00:28:09.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80da200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f80d4154830 0x7f80d415b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:09.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80da200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80d40cbbb0 0x7f80d40cc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:09.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80da200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80d40c68c0 0x7f80d40c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:09.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80da200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f80d40c2b30 con 0x7f80d40cbbb0 2024-05-27T00:28:09.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80da200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f80d40c26a0 con 0x7f80d4154830 2024-05-27T00:28:09.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80da200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f80d40c1a90 con 0x7f80d40c68c0 2024-05-27T00:28:09.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80d2a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80d40cbbb0 0x7f80d40cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:09.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80d2a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80d40cbbb0 0x7f80d40cc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55718/0 (socket says 172.21.15.98:55718) 2024-05-27T00:28:09.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80d2a00640 1 -- 172.21.15.98:0/2062274588 learned_addr learned my addr 172.21.15.98:0/2062274588 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:09.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80d3e00640 1 --2- 172.21.15.98:0/2062274588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80d40c68c0 0x7f80d40c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:09.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80d3400640 1 --2- 172.21.15.98:0/2062274588 >> v2:172.21.15.204:3300/0 conn(0x7f80d4154830 0x7f80d415b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:09.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80d2a00640 1 -- 172.21.15.98:0/2062274588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80d40c68c0 msgr2=0x7f80d40c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:09.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80d2a00640 1 --2- 172.21.15.98:0/2062274588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80d40c68c0 0x7f80d40c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80d2a00640 1 -- 172.21.15.98:0/2062274588 >> v2:172.21.15.204:3300/0 conn(0x7f80d4154830 msgr2=0x7f80d415b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:09.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80d2a00640 1 --2- 172.21.15.98:0/2062274588 >> v2:172.21.15.204:3300/0 conn(0x7f80d4154830 0x7f80d415b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.130+0000 7f80d2a00640 1 -- 172.21.15.98:0/2062274588 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d40c1d70 con 0x7f80d40cbbb0 2024-05-27T00:28:09.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80d2a00640 1 --2- 172.21.15.98:0/2062274588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d40cbbb0 0x7f80d40cc010 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f80d40cc760 tx=0x7f80c001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38eca4fe8cd6aac1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:09.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80d2000640 1 -- 172.21.15.98:0/2062274588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c00b1070 con 0x7f80d40cbbb0 2024-05-27T00:28:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80d2000640 1 -- 172.21.15.98:0/2062274588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80c00ac630 con 0x7f80d40cbbb0 2024-05-27T00:28:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80d2000640 1 -- 172.21.15.98:0/2062274588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c00a8050 con 0x7f80d40cbbb0 2024-05-27T00:28:09.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80da200640 1 -- 172.21.15.98:0/2062274588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d40cbbb0 msgr2=0x7f80d40cc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80da200640 1 --2- 172.21.15.98:0/2062274588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d40cbbb0 0x7f80d40cc010 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f80d40cc760 tx=0x7f80c001c520 comp rx=0 tx=0).stop 2024-05-27T00:28:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80da200640 1 -- 172.21.15.98:0/2062274588 shutdown_connections 2024-05-27T00:28:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80da200640 1 --2- 172.21.15.98:0/2062274588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80d40c68c0 0x7f80d40c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80da200640 1 --2- 172.21.15.98:0/2062274588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d40cbbb0 0x7f80d40cc010 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80da200640 1 --2- 172.21.15.98:0/2062274588 >> v2:172.21.15.204:3300/0 conn(0x7f80d4154830 0x7f80d415b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80da200640 1 -- 172.21.15.98:0/2062274588 >> 172.21.15.98:0/2062274588 conn(0x7f80d4067530 msgr2=0x7f80d406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80da200640 1 -- 172.21.15.98:0/2062274588 shutdown_connections 2024-05-27T00:28:09.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.134+0000 7f80da200640 1 -- 172.21.15.98:0/2062274588 wait complete. 2024-05-27T00:28:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 Processor -- start 2024-05-27T00:28:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 -- start start 2024-05-27T00:28:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80d40c68c0 0x7f80d40af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80d40cbbb0 0x7f80d41f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d4154830 0x7f80d41575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80d415e040 con 0x7f80d4154830 2024-05-27T00:28:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f80d415dec0 con 0x7f80d40c68c0 2024-05-27T00:28:09.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f80d415e1c0 con 0x7f80d40cbbb0 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d4154830 0x7f80d41575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d4154830 0x7f80d41575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55720/0 (socket says 172.21.15.98:55720) 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3e00640 1 -- 172.21.15.98:0/496352520 learned_addr learned my addr 172.21.15.98:0/496352520 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3400640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80d40c68c0 0x7f80d40af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d2a00640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80d40cbbb0 0x7f80d41f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3e00640 1 -- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80d40cbbb0 msgr2=0x7f80d41f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3e00640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80d40cbbb0 0x7f80d41f4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3e00640 1 -- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80d40c68c0 msgr2=0x7f80d40af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3e00640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80d40c68c0 0x7f80d40af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3e00640 1 -- 172.21.15.98:0/496352520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d41f47c0 con 0x7f80d4154830 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3400640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80d40c68c0 0x7f80d40af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:09.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80d3e00640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d4154830 0x7f80d41575f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f80cc007e90 tx=0x7f80cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80cb400640 1 -- 172.21.15.98:0/496352520 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc014420 con 0x7f80d4154830 2024-05-27T00:28:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80cb400640 1 -- 172.21.15.98:0/496352520 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80cc019070 con 0x7f80d4154830 2024-05-27T00:28:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80cb400640 1 -- 172.21.15.98:0/496352520 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc014820 con 0x7f80d4154830 2024-05-27T00:28:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 -- 172.21.15.98:0/496352520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d40cb010 con 0x7f80d4154830 2024-05-27T00:28:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 -- 172.21.15.98:0/496352520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d4158d90 con 0x7f80d4154830 2024-05-27T00:28:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.138+0000 7f80da200640 1 -- 172.21.15.98:0/496352520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80d4075cb0 con 0x7f80d4154830 2024-05-27T00:28:09.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.142+0000 7f80cb400640 1 -- 172.21.15.98:0/496352520 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f80cc0149c0 con 0x7f80d4154830 2024-05-27T00:28:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.142+0000 7f80cb400640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f80b40678c0 0x7f80b4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.142+0000 7f80cb400640 1 -- 172.21.15.98:0/496352520 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f80cc086880 con 0x7f80d4154830 2024-05-27T00:28:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.142+0000 7f80d3400640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f80b40678c0 0x7f80b4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.142+0000 7f80d3400640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f80b40678c0 0x7f80b4069d80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f80c400b9f0 tx=0x7f80c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:09.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.146+0000 7f80cb400640 1 -- 172.21.15.98:0/496352520 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f80d40cb010 con 0x7f80d4154830 2024-05-27T00:28:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.338+0000 7f80da200640 1 -- 172.21.15.98:0/496352520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80d40c1300 con 0x7f80d4154830 2024-05-27T00:28:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.338+0000 7f80cb400640 1 -- 172.21.15.98:0/496352520 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+6010 (secure 0 0 0) 0x7f80cc056fb0 con 0x7f80d4154830 2024-05-27T00:28:09.345 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:09.345 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:05.555961+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 54 2024-05-27T00:28:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 -- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f80b40678c0 msgr2=0x7f80b4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f80b40678c0 0x7f80b4069d80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f80c400b9f0 tx=0x7f80c4008040 comp rx=0 tx=0).stop 2024-05-27T00:28:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 -- 172.21.15.98:0/496352520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d4154830 msgr2=0x7f80d41575f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d4154830 0x7f80d41575f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f80cc007e90 tx=0x7f80cc00e4b0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 -- 172.21.15.98:0/496352520 shutdown_connections 2024-05-27T00:28:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80d4154830 0x7f80d41575f0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80d40cbbb0 0x7f80d41f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f80b40678c0 0x7f80b4069d80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 --2- 172.21.15.98:0/496352520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f80d40c68c0 0x7f80d40af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 -- 172.21.15.98:0/496352520 >> 172.21.15.98:0/496352520 conn(0x7f80d4067530 msgr2=0x7f80d40c78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 -- 172.21.15.98:0/496352520 shutdown_connections 2024-05-27T00:28:09.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:09.342+0000 7f80da200640 1 -- 172.21.15.98:0/496352520 wait complete. 2024-05-27T00:28:09.364 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 54, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5943": { "addr": "172.21.15.98:6834/2148124202", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 2148124202, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 2148124202, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5943, "incarnation": 53, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:28:05.555961+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5943 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:28:11.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:11.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.498+0000 7f4f15000640 1 Processor -- start 2024-05-27T00:28:11.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.498+0000 7f4f15000640 1 -- start start 2024-05-27T00:28:11.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f15000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f100cc750 0x7f4f100c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:11.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f15000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f100c74f0 con 0x7f4f100ccb30 2024-05-27T00:28:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f15000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f1015a970 con 0x7f4f10154d30 2024-05-27T00:28:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f15000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4f1015b580 con 0x7f4f100cc750 2024-05-27T00:28:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f07e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4f100ccb30 0x7f4f10154620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38810/0 (socket says 172.21.15.98:38810) 2024-05-27T00:28:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f07e00640 1 -- 172.21.15.98:0/638483163 learned_addr learned my addr 172.21.15.98:0/638483163 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0f400640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f100cc750 0x7f4f100c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1552001810 0 0) 0x7f4f100c74f0 con 0x7f4f100ccb30 2024-05-27T00:28:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ef4003530 con 0x7f4f100ccb30 2024-05-27T00:28:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1022486443 0 0) 0x7f4f1015a970 con 0x7f4f10154d30 2024-05-27T00:28:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f100c74f0 con 0x7f4f10154d30 2024-05-27T00:28:11.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3436494753 0 0) 0x7f4ef4003530 con 0x7f4f100ccb30 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f100cc750 msgr2=0x7f4f100c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f100cc750 0x7f4f100c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 >> v1:172.21.15.204:6789/0 conn(0x7f4f10154d30 legacy=0x7f4f100c8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1015ac50 con 0x7f4f100ccb30 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f4ef8001680 con 0x7f4f100ccb30 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 >> v1:172.21.15.98:6789/0 conn(0x7f4f100ccb30 legacy=0x7f4f10154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4003b80 0x7f4ef4003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f100cc750 0x7f4ef4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ef4004d50 0x7f4ef40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1015b580 con 0x7f4ef4003b80 2024-05-27T00:28:11.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef400ba80 con 0x7f4ef4004d50 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef400bc80 con 0x7f4f100cc750 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0f400640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4003b80 0x7f4ef4003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f07e00640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f100cc750 0x7f4ef4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0fe00640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ef4004d50 0x7f4ef40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0f400640 1 -- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f100cc750 msgr2=0x7f4ef4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0f400640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f100cc750 0x7f4ef4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0f400640 1 -- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ef4004d50 msgr2=0x7f4ef40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0f400640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ef4004d50 0x7f4ef40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0f400640 1 -- 172.21.15.98:0/638483163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef4003530 con 0x7f4ef4003b80 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f0fe00640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ef4004d50 0x7f4ef40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:11.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.502+0000 7f4f07e00640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f100cc750 0x7f4ef4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f0f400640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4003b80 0x7f4ef4003f80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009ae0 tx=0x7f4f0001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f14f0575330a5c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f000b2070 con 0x7f4ef4003b80 2024-05-27T00:28:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f000a5660 con 0x7f4ef4003b80 2024-05-27T00:28:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f0ea00640 1 -- 172.21.15.98:0/638483163 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f000096b0 con 0x7f4ef4003b80 2024-05-27T00:28:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f15000640 1 -- 172.21.15.98:0/638483163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4003b80 msgr2=0x7f4ef4003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f15000640 1 --2- 172.21.15.98:0/638483163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ef4003b80 0x7f4ef4003f80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009ae0 tx=0x7f4f0001ca70 comp rx=0 tx=0).stop 2024-05-27T00:28:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f0fe00640 1 -- 172.21.15.98:0/638483163 reap_dead start 2024-05-27T00:28:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f15000640 1 -- 172.21.15.98:0/638483163 shutdown_connections 2024-05-27T00:28:11.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f15000640 1 -- 172.21.15.98:0/638483163 >> 172.21.15.98:0/638483163 conn(0x7f4f1006dd00 msgr2=0x7f4f1006b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:11.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f15000640 1 -- 172.21.15.98:0/638483163 shutdown_connections 2024-05-27T00:28:11.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f15000640 1 -- 172.21.15.98:0/638483163 wait complete. 2024-05-27T00:28:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f15000640 1 Processor -- start 2024-05-27T00:28:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.506+0000 7f4f15000640 1 -- start start 2024-05-27T00:28:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f15000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ef4003b80 0x7f4f10150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f15000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f100ca360 0x7f4f101f4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f15000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f101f48d0 0x7f4f10149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f15000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f100c8b90 con 0x7f4f100ca360 2024-05-27T00:28:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f15000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1015b210 con 0x7f4ef4003b80 2024-05-27T00:28:11.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f15000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4f100c8d10 con 0x7f4f101f48d0 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f100ca360 0x7f4f101f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f100ca360 0x7f4f101f4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55738/0 (socket says 172.21.15.98:55738) 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f0f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ef4003b80 0x7f4f10150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07e00640 1 -- 172.21.15.98:0/3256110077 learned_addr learned my addr 172.21.15.98:0/3256110077 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f0fe00640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f101f48d0 0x7f4f10149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07e00640 1 -- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f101f48d0 msgr2=0x7f4f10149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07e00640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f101f48d0 0x7f4f10149fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07e00640 1 -- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ef4003b80 msgr2=0x7f4f10150820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07e00640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ef4003b80 0x7f4f10150820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07e00640 1 -- 172.21.15.98:0/3256110077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1014cdd0 con 0x7f4f100ca360 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07e00640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f100ca360 0x7f4f101f4390 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800e7f0 tx=0x7f4ef800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07400640 1 -- 172.21.15.98:0/3256110077 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef800ee40 con 0x7f4f100ca360 2024-05-27T00:28:11.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07400640 1 -- 172.21.15.98:0/3256110077 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ef8018070 con 0x7f4f100ca360 2024-05-27T00:28:11.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f07400640 1 -- 172.21.15.98:0/3256110077 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef800cb10 con 0x7f4f100ca360 2024-05-27T00:28:11.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f15000640 1 -- 172.21.15.98:0/3256110077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f10159cb0 con 0x7f4f100ca360 2024-05-27T00:28:11.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f15000640 1 -- 172.21.15.98:0/3256110077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f1015a270 con 0x7f4f100ca360 2024-05-27T00:28:11.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.510+0000 7f4f15000640 1 -- 172.21.15.98:0/3256110077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f10075c40 con 0x7f4f100ca360 2024-05-27T00:28:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.514+0000 7f4f07400640 1 -- 172.21.15.98:0/3256110077 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4ef800ccb0 con 0x7f4f100ca360 2024-05-27T00:28:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.514+0000 7f4f07400640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eec0678f0 0x7f4eec069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.514+0000 7f4f07400640 1 -- 172.21.15.98:0/3256110077 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f4ef8085dc0 con 0x7f4f100ca360 2024-05-27T00:28:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.514+0000 7f4f0f400640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eec0678f0 0x7f4eec069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:11.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.514+0000 7f4f0f400640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eec0678f0 0x7f4eec069db0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4f000af000 tx=0x7f4f000046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:11.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.522+0000 7f4f07400640 1 -- 172.21.15.98:0/3256110077 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4ef8056570 con 0x7f4f100ca360 2024-05-27T00:28:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.710+0000 7f4f15000640 1 -- 172.21.15.98:0/3256110077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f100c7950 con 0x7f4f100ca360 2024-05-27T00:28:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f07400640 1 -- 172.21.15.98:0/3256110077 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6007 (secure 0 0 0) 0x7f4ef8055e70 con 0x7f4f100ca360 2024-05-27T00:28:11.717 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:11.717 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:10.685089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 55 2024-05-27T00:28:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 -- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eec0678f0 msgr2=0x7f4eec069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eec0678f0 0x7f4eec069db0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4f000af000 tx=0x7f4f000046d0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 -- 172.21.15.98:0/3256110077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f100ca360 msgr2=0x7f4f101f4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f100ca360 0x7f4f101f4390 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800e7f0 tx=0x7f4ef800ebf0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 -- 172.21.15.98:0/3256110077 shutdown_connections 2024-05-27T00:28:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f101f48d0 0x7f4f10149fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f100ca360 0x7f4f101f4390 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eec0678f0 0x7f4eec069db0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 --2- 172.21.15.98:0/3256110077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ef4003b80 0x7f4f10150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 -- 172.21.15.98:0/3256110077 >> 172.21.15.98:0/3256110077 conn(0x7f4f1006dd00 msgr2=0x7f4f1014e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 -- 172.21.15.98:0/3256110077 shutdown_connections 2024-05-27T00:28:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:11.714+0000 7f4f15000640 1 -- 172.21.15.98:0/3256110077 wait complete. 2024-05-27T00:28:11.738 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:28:13.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:13.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.870+0000 7feae3400640 1 Processor -- start 2024-05-27T00:28:13.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.870+0000 7feae3400640 1 -- start start 2024-05-27T00:28:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feadc0cc1f0 0x7feadc1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae3400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feadc073570 0x7feadc1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae3400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feadc0c1f60 con 0x7feadc0cbe10 2024-05-27T00:28:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae3400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7feadc159810 con 0x7feadc073570 2024-05-27T00:28:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae3400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7feadc0c6dd0 con 0x7feadc0cc1f0 2024-05-27T00:28:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae1400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feadc073570 0x7feadc1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae1400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feadc073570 0x7feadc1559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60726/0 (socket says 172.21.15.98:60726) 2024-05-27T00:28:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae1400640 1 -- 172.21.15.98:0/4190435769 learned_addr learned my addr 172.21.15.98:0/4190435769 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feadbe00640 1 --2- 172.21.15.98:0/4190435769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feadc0cc1f0 0x7feadc1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae1400640 1 -- 172.21.15.98:0/4190435769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feadc0cc1f0 msgr2=0x7feadc1592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae1400640 1 --2- 172.21.15.98:0/4190435769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feadc0cc1f0 0x7feadc1592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae1400640 1 -- 172.21.15.98:0/4190435769 >> v1:172.21.15.204:6789/0 conn(0x7feadc0cbe10 legacy=0x7feadc0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.874+0000 7feae1400640 1 -- 172.21.15.98:0/4190435769 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feadc0c70b0 con 0x7feadc073570 2024-05-27T00:28:13.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae1400640 1 --2- 172.21.15.98:0/4190435769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc073570 0x7feadc1559e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fead4009e50 tx=0x7fead401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df77e6fcf3dfb1d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:13.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feadb400640 1 -- 172.21.15.98:0/4190435769 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead40b4070 con 0x7feadc073570 2024-05-27T00:28:13.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feadb400640 1 -- 172.21.15.98:0/4190435769 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fead40af400 con 0x7feadc073570 2024-05-27T00:28:13.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feadb400640 1 -- 172.21.15.98:0/4190435769 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead40af7d0 con 0x7feadc073570 2024-05-27T00:28:13.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 -- 172.21.15.98:0/4190435769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc073570 msgr2=0x7feadc1559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:13.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 --2- 172.21.15.98:0/4190435769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc073570 0x7feadc1559e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fead4009e50 tx=0x7fead401f210 comp rx=0 tx=0).stop 2024-05-27T00:28:13.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 -- 172.21.15.98:0/4190435769 shutdown_connections 2024-05-27T00:28:13.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 --2- 172.21.15.98:0/4190435769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc073570 0x7feadc1559e0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:13.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 --2- 172.21.15.98:0/4190435769 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feadc0cc1f0 0x7feadc1592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:13.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 -- 172.21.15.98:0/4190435769 >> 172.21.15.98:0/4190435769 conn(0x7feadc067530 msgr2=0x7feadc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:13.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 -- 172.21.15.98:0/4190435769 shutdown_connections 2024-05-27T00:28:13.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 -- 172.21.15.98:0/4190435769 wait complete. 2024-05-27T00:28:13.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 Processor -- start 2024-05-27T00:28:13.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 -- start start 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feadc073570 0x7feadc1577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feadc0cbe10 0x7feadc0c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc0cc1f0 0x7feadc14e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feadc0c1330 con 0x7feadc0cc1f0 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feadc0c1050 con 0x7feadc0cbe10 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae3400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7feadc0c79d0 con 0x7feadc073570 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.878+0000 7feae1400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc0cc1f0 0x7feadc14e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae1400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc0cc1f0 0x7feadc14e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60730/0 (socket says 172.21.15.98:60730) 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae1400640 1 -- 172.21.15.98:0/3605610231 learned_addr learned my addr 172.21.15.98:0/3605610231 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae0a00640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feadc073570 0x7feadc1577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feadbe00640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feadc0cbe10 0x7feadc0c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:13.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae1400640 1 -- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feadc073570 msgr2=0x7feadc1577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae1400640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feadc073570 0x7feadc1577f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae1400640 1 -- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feadc0cbe10 msgr2=0x7feadc0c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae1400640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feadc0cbe10 0x7feadc0c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae1400640 1 -- 172.21.15.98:0/3605610231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feadc0c3710 con 0x7feadc0cc1f0 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feadbe00640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feadc0cbe10 0x7feadc0c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae1400640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc0cc1f0 0x7feadc14e2e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fead40b2040 tx=0x7fead401bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7fead8c00640 1 -- 172.21.15.98:0/3605610231 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead40be020 con 0x7feadc0cc1f0 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7fead8c00640 1 -- 172.21.15.98:0/3605610231 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fead40090d0 con 0x7feadc0cc1f0 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7fead8c00640 1 -- 172.21.15.98:0/3605610231 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead40b4040 con 0x7feadc0cc1f0 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae3400640 1 -- 172.21.15.98:0/3605610231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feadc0c39b0 con 0x7feadc0cc1f0 2024-05-27T00:28:13.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae3400640 1 -- 172.21.15.98:0/3605610231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feadc0c3f70 con 0x7feadc0cc1f0 2024-05-27T00:28:13.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7feae3400640 1 -- 172.21.15.98:0/3605610231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feadc075cb0 con 0x7feadc0cc1f0 2024-05-27T00:28:13.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7fead8c00640 1 -- 172.21.15.98:0/3605610231 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fead40a8d10 con 0x7feadc0cc1f0 2024-05-27T00:28:13.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7fead8c00640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feabc0678c0 0x7feabc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:13.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.882+0000 7fead8c00640 1 -- 172.21.15.98:0/3605610231 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7fead4123a30 con 0x7feadc0cc1f0 2024-05-27T00:28:13.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.886+0000 7feae0a00640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feabc0678c0 0x7feabc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:13.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.886+0000 7feae0a00640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feabc0678c0 0x7feabc069d80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fead0001810 tx=0x7fead0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:13.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:13.886+0000 7fead8c00640 1 -- 172.21.15.98:0/3605610231 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fead40c8060 con 0x7feadc0cc1f0 2024-05-27T00:28:14.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 -- 172.21.15.98:0/3605610231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feadc0b1d10 con 0x7feadc0cc1f0 2024-05-27T00:28:14.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7fead8c00640 1 -- 172.21.15.98:0/3605610231 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6007 (secure 0 0 0) 0x7fead4018d70 con 0x7feadc0cc1f0 2024-05-27T00:28:14.082 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:14.082 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:10.685089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:14.083 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 55 2024-05-27T00:28:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 -- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feabc0678c0 msgr2=0x7feabc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feabc0678c0 0x7feabc069d80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fead0001810 tx=0x7fead0009040 comp rx=0 tx=0).stop 2024-05-27T00:28:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 -- 172.21.15.98:0/3605610231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc0cc1f0 msgr2=0x7feadc14e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc0cc1f0 0x7feadc14e2e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fead40b2040 tx=0x7fead401bfb0 comp rx=0 tx=0).stop 2024-05-27T00:28:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 -- 172.21.15.98:0/3605610231 shutdown_connections 2024-05-27T00:28:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feadc0cc1f0 0x7feadc14e2e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7feabc0678c0 0x7feabc069d80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feadc0cbe10 0x7feadc0c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 --2- 172.21.15.98:0/3605610231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feadc073570 0x7feadc1577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:14.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 -- 172.21.15.98:0/3605610231 >> 172.21.15.98:0/3605610231 conn(0x7feadc067530 msgr2=0x7feadc154b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:14.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 -- 172.21.15.98:0/3605610231 shutdown_connections 2024-05-27T00:28:14.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:14.078+0000 7feae3400640 1 -- 172.21.15.98:0/3605610231 wait complete. 2024-05-27T00:28:14.103 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:28:16.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:16.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c08600640 1 Processor -- start 2024-05-27T00:28:16.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c08600640 1 -- start start 2024-05-27T00:28:16.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c08600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8c00157e10 0x7f8c0006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:16.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c08600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00072a50 0x7f8c00067130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:16.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c08600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c000c9340 con 0x7f8c00072500 2024-05-27T00:28:16.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c08600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8c0006b870 con 0x7f8c00157e10 2024-05-27T00:28:16.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c08600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8c00071fa0 con 0x7f8c00072a50 2024-05-27T00:28:16.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c05c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8c00157e10 0x7f8c0006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:16.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c05c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8c00157e10 0x7f8c0006c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60740/0 (socket says 172.21.15.98:60740) 2024-05-27T00:28:16.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c05c00640 1 -- 172.21.15.98:0/4181257596 learned_addr learned my addr 172.21.15.98:0/4181257596 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:16.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c05200640 1 --2- 172.21.15.98:0/4181257596 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00072a50 0x7f8c00067130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:16.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c05c00640 1 -- 172.21.15.98:0/4181257596 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00072a50 msgr2=0x7f8c00067130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:16.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c05c00640 1 --2- 172.21.15.98:0/4181257596 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00072a50 0x7f8c00067130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:16.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c05c00640 1 -- 172.21.15.98:0/4181257596 >> v1:172.21.15.204:6789/0 conn(0x7f8c00072500 legacy=0x7f8c0015dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:16.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.238+0000 7f8c05c00640 1 -- 172.21.15.98:0/4181257596 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c000c95e0 con 0x7f8c00157e10 2024-05-27T00:28:16.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8c05c00640 1 --2- 172.21.15.98:0/4181257596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00157e10 0x7f8c0006c640 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8c000c9510 tx=0x7f8bf401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693e250138e7e97c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:16.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8bffe00640 1 -- 172.21.15.98:0/4181257596 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf40b0070 con 0x7f8c00157e10 2024-05-27T00:28:16.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8bffe00640 1 -- 172.21.15.98:0/4181257596 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bf40ab600 con 0x7f8c00157e10 2024-05-27T00:28:16.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8bffe00640 1 -- 172.21.15.98:0/4181257596 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf40ab9d0 con 0x7f8c00157e10 2024-05-27T00:28:16.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8c08600640 1 -- 172.21.15.98:0/4181257596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00157e10 msgr2=0x7f8c0006c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:16.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8c08600640 1 --2- 172.21.15.98:0/4181257596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00157e10 0x7f8c0006c640 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8c000c9510 tx=0x7f8bf401c420 comp rx=0 tx=0).stop 2024-05-27T00:28:16.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8c08600640 1 -- 172.21.15.98:0/4181257596 shutdown_connections 2024-05-27T00:28:16.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8c08600640 1 --2- 172.21.15.98:0/4181257596 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00072a50 0x7f8c00067130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:16.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8c08600640 1 --2- 172.21.15.98:0/4181257596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00157e10 0x7f8c0006c640 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:16.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8c08600640 1 -- 172.21.15.98:0/4181257596 >> 172.21.15.98:0/4181257596 conn(0x7f8c000c9b70 msgr2=0x7f8c000c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:16.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8c08600640 1 -- 172.21.15.98:0/4181257596 shutdown_connections 2024-05-27T00:28:16.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.242+0000 7f8c08600640 1 -- 172.21.15.98:0/4181257596 wait complete. 2024-05-27T00:28:16.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 Processor -- start 2024-05-27T00:28:16.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 -- start start 2024-05-27T00:28:16.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c00072500 0x7f8c001f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00072a50 0x7f8c00156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00157e10 0x7f8c001418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c000c7c70 con 0x7f8c00072a50 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0006bd00 con 0x7f8c00072500 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8c000c78a0 con 0x7f8c00157e10 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c05200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00072a50 0x7f8c00156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c05200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00072a50 0x7f8c00156e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60756/0 (socket says 172.21.15.98:60756) 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c05200640 1 -- 172.21.15.98:0/2064091493 learned_addr learned my addr 172.21.15.98:0/2064091493 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c05c00640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c00072500 0x7f8c001f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c06600640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00157e10 0x7f8c001418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c05200640 1 -- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00157e10 msgr2=0x7f8c001418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c05200640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00157e10 0x7f8c001418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:16.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c05200640 1 -- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c00072500 msgr2=0x7f8c001f4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:16.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c05200640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c00072500 0x7f8c001f4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:16.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c05200640 1 -- 172.21.15.98:0/2064091493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c00145b10 con 0x7f8c00072a50 2024-05-27T00:28:16.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c06600640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00157e10 0x7f8c001418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:16.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c05200640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00072a50 0x7f8c00156e40 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00d620 tx=0x7f8bec00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8bfd600640 1 -- 172.21.15.98:0/2064091493 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bec00b620 con 0x7f8c00072a50 2024-05-27T00:28:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8bfd600640 1 -- 172.21.15.98:0/2064091493 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bec019070 con 0x7f8c00072a50 2024-05-27T00:28:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8bfd600640 1 -- 172.21.15.98:0/2064091493 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bec00b9e0 con 0x7f8c00072a50 2024-05-27T00:28:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 -- 172.21.15.98:0/2064091493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c00145df0 con 0x7f8c00072a50 2024-05-27T00:28:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 -- 172.21.15.98:0/2064091493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c001f7490 con 0x7f8c00072a50 2024-05-27T00:28:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.246+0000 7f8c08600640 1 -- 172.21.15.98:0/2064091493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c00075cb0 con 0x7f8c00072a50 2024-05-27T00:28:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.250+0000 7f8bfd600640 1 -- 172.21.15.98:0/2064091493 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8bec00bb80 con 0x7f8c00072a50 2024-05-27T00:28:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.250+0000 7f8bfd600640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8be00678c0 0x7f8be0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.250+0000 7f8bfd600640 1 -- 172.21.15.98:0/2064091493 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f8bec086450 con 0x7f8c00072a50 2024-05-27T00:28:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.250+0000 7f8c05c00640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8be00678c0 0x7f8be0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.250+0000 7f8c05c00640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8be00678c0 0x7f8be0069d80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8bf40ad000 tx=0x7f8bf4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.254+0000 7f8bfd600640 1 -- 172.21.15.98:0/2064091493 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8bec056b70 con 0x7f8c00072a50 2024-05-27T00:28:16.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.478+0000 7f8c08600640 1 -- 172.21.15.98:0/2064091493 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c0014e5d0 con 0x7f8c00072a50 2024-05-27T00:28:16.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8bfd600640 1 -- 172.21.15.98:0/2064091493 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6007 (secure 0 0 0) 0x7f8bec056470 con 0x7f8c00072a50 2024-05-27T00:28:16.485 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:16.485 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:10.685089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:16.486 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 55 2024-05-27T00:28:16.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 -- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8be00678c0 msgr2=0x7f8be0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:16.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8be00678c0 0x7f8be0069d80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8bf40ad000 tx=0x7f8bf4008040 comp rx=0 tx=0).stop 2024-05-27T00:28:16.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 -- 172.21.15.98:0/2064091493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00072a50 msgr2=0x7f8c00156e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:16.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00072a50 0x7f8c00156e40 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00d620 tx=0x7f8bec00dae0 comp rx=0 tx=0).stop 2024-05-27T00:28:16.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 -- 172.21.15.98:0/2064091493 shutdown_connections 2024-05-27T00:28:16.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c00157e10 0x7f8c001418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:16.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c00072a50 0x7f8c00156e40 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:16.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8be00678c0 0x7f8be0069d80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:16.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 --2- 172.21.15.98:0/2064091493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c00072500 0x7f8c001f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:16.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 -- 172.21.15.98:0/2064091493 >> 172.21.15.98:0/2064091493 conn(0x7f8c000c9b70 msgr2=0x7f8c00154d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:16.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 -- 172.21.15.98:0/2064091493 shutdown_connections 2024-05-27T00:28:16.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:16.482+0000 7f8c08600640 1 -- 172.21.15.98:0/2064091493 wait complete. 2024-05-27T00:28:16.508 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:28:18.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:18.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5f6a00640 1 Processor -- start 2024-05-27T00:28:18.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5f6a00640 1 -- start start 2024-05-27T00:28:18.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5f6a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5f00cbe10 0x7fb5f01592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:18.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5f6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5f00cc1f0 0x7fb5f00c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:18.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5f6a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5f00c6dd0 con 0x7fb5f0073570 2024-05-27T00:28:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5f6a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb5f0159810 con 0x7fb5f00cbe10 2024-05-27T00:28:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5f6a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f00c70b0 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5efe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5f00cbe10 0x7fb5f01592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5efe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5f00cbe10 0x7fb5f01592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60768/0 (socket says 172.21.15.98:60768) 2024-05-27T00:28:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5efe00640 1 -- 172.21.15.98:0/389245317 learned_addr learned my addr 172.21.15.98:0/389245317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5ef400640 1 --2- 172.21.15.98:0/389245317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5f00cc1f0 0x7fb5f00c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5efe00640 1 -- 172.21.15.98:0/389245317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5f00cc1f0 msgr2=0x7fb5f00c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5efe00640 1 --2- 172.21.15.98:0/389245317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5f00cc1f0 0x7fb5f00c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5efe00640 1 -- 172.21.15.98:0/389245317 >> v1:172.21.15.204:6789/0 conn(0x7fb5f0073570 legacy=0x7fb5f01559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:18.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.646+0000 7fb5efe00640 1 -- 172.21.15.98:0/389245317 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f00c7390 con 0x7fb5f00cbe10 2024-05-27T00:28:18.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5efe00640 1 --2- 172.21.15.98:0/389245317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cbe10 0x7fb5f01592d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009870 tx=0x7fb5dc01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784a1c5331ecae45 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:18.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5eea00640 1 -- 172.21.15.98:0/389245317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0b5070 con 0x7fb5f00cbe10 2024-05-27T00:28:18.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5eea00640 1 -- 172.21.15.98:0/389245317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5dc0a8a50 con 0x7fb5f00cbe10 2024-05-27T00:28:18.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5eea00640 1 -- 172.21.15.98:0/389245317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0a8e20 con 0x7fb5f00cbe10 2024-05-27T00:28:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5f6a00640 1 -- 172.21.15.98:0/389245317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cbe10 msgr2=0x7fb5f01592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5f6a00640 1 --2- 172.21.15.98:0/389245317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cbe10 0x7fb5f01592d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009870 tx=0x7fb5dc01ed50 comp rx=0 tx=0).stop 2024-05-27T00:28:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5f6a00640 1 -- 172.21.15.98:0/389245317 shutdown_connections 2024-05-27T00:28:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5f6a00640 1 --2- 172.21.15.98:0/389245317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5f00cc1f0 0x7fb5f00c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5f6a00640 1 --2- 172.21.15.98:0/389245317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cbe10 0x7fb5f01592d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5f6a00640 1 -- 172.21.15.98:0/389245317 >> 172.21.15.98:0/389245317 conn(0x7fb5f0067530 msgr2=0x7fb5f006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5f6a00640 1 -- 172.21.15.98:0/389245317 shutdown_connections 2024-05-27T00:28:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.650+0000 7fb5f6a00640 1 -- 172.21.15.98:0/389245317 wait complete. 2024-05-27T00:28:18.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f6a00640 1 Processor -- start 2024-05-27T00:28:18.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f6a00640 1 -- start start 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5f0073570 0x7fb5f0157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5f00cbe10 0x7fb5f01f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cc1f0 0x7fb5f00b0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f6a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f00c79d0 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f6a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f00c1330 con 0x7fb5f0073570 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f6a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f00c6990 con 0x7fb5f00cbe10 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f4a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cc1f0 0x7fb5f00b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f4a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cc1f0 0x7fb5f00b0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60772/0 (socket says 172.21.15.98:60772) 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f4a00640 1 -- 172.21.15.98:0/3550825996 learned_addr learned my addr 172.21.15.98:0/3550825996 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5efe00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5f0073570 0x7fb5f0157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:18.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5ef400640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5f00cbe10 0x7fb5f01f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:18.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f4a00640 1 -- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5f00cbe10 msgr2=0x7fb5f01f3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:18.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f4a00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5f00cbe10 0x7fb5f01f3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:18.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f4a00640 1 -- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5f0073570 msgr2=0x7fb5f0157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:18.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f4a00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5f0073570 0x7fb5f0157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:18.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f4a00640 1 -- 172.21.15.98:0/3550825996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f00aec00 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f4a00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cc1f0 0x7fb5f00b0470 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c7d0 tx=0x7fb5e800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:18.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5e7e00640 1 -- 172.21.15.98:0/3550825996 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8003b80 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5e7e00640 1 -- 172.21.15.98:0/3550825996 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5e8003d20 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f6a00640 1 -- 172.21.15.98:0/3550825996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f0156840 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5e7e00640 1 -- 172.21.15.98:0/3550825996 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e800aa30 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.654+0000 7fb5f6a00640 1 -- 172.21.15.98:0/3550825996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f01f01a0 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.658+0000 7fb5f6a00640 1 -- 172.21.15.98:0/3550825996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f0075cb0 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.658+0000 7fb5e7e00640 1 -- 172.21.15.98:0/3550825996 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb5e800abd0 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.658+0000 7fb5e7e00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb5d00678f0 0x7fb5d0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:18.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.658+0000 7fb5e7e00640 1 -- 172.21.15.98:0/3550825996 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7fb5e8085f60 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.658+0000 7fb5efe00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb5d00678f0 0x7fb5d0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:18.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.658+0000 7fb5efe00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb5d00678f0 0x7fb5d0069db0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0b2000 tx=0x7fb5dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:18.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.662+0000 7fb5e7e00640 1 -- 172.21.15.98:0/3550825996 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb5e8056680 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 -- 172.21.15.98:0/3550825996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5f00b1f60 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5e7e00640 1 -- 172.21.15.98:0/3550825996 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6014 (secure 0 0 0) 0x7fb5e8055f80 con 0x7fb5f00cc1f0 2024-05-27T00:28:18.859 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:18.860 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 56 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 -- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb5d00678f0 msgr2=0x7fb5d0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb5d00678f0 0x7fb5d0069db0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0b2000 tx=0x7fb5dc009040 comp rx=0 tx=0).stop 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 -- 172.21.15.98:0/3550825996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cc1f0 msgr2=0x7fb5f00b0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cc1f0 0x7fb5f00b0470 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c7d0 tx=0x7fb5e800cc90 comp rx=0 tx=0).stop 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 -- 172.21.15.98:0/3550825996 shutdown_connections 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5f00cc1f0 0x7fb5f00b0470 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5f00cbe10 0x7fb5f01f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb5d00678f0 0x7fb5d0069db0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 --2- 172.21.15.98:0/3550825996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5f0073570 0x7fb5f0157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 -- 172.21.15.98:0/3550825996 >> 172.21.15.98:0/3550825996 conn(0x7fb5f0067530 msgr2=0x7fb5f0153fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 -- 172.21.15.98:0/3550825996 shutdown_connections 2024-05-27T00:28:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:18.854+0000 7fb5f6a00640 1 -- 172.21.15.98:0/3550825996 wait complete. 2024-05-27T00:28:18.879 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5802 2024-05-27T00:28:18.880 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.c 2024-05-27T00:28:21.881 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-05-27T00:28:21.881 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-27T00:28:21.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-27T00:28:21.885 INFO:tasks.ceph.mds.c:Started 2024-05-27T00:28:21.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:21.970 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2024-05-27T00:28:22.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26af000640 1 Processor -- start 2024-05-27T00:28:22.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26af000640 1 -- start start 2024-05-27T00:28:22.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26af000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26a80cc1f0 0x7f26a81592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:22.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26af000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f26a8073570 0x7f26a81559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:22.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26af000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a80c1f60 con 0x7f26a80cbe10 2024-05-27T00:28:22.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26af000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f26a8159810 con 0x7f26a8073570 2024-05-27T00:28:22.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26af000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f26a80c6dd0 con 0x7f26a80cc1f0 2024-05-27T00:28:22.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26a7e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f26a80cbe10 0x7f26a80c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40676/0 (socket says 172.21.15.98:40676) 2024-05-27T00:28:22.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26a7e00640 1 -- 172.21.15.98:0/2144875262 learned_addr learned my addr 172.21.15.98:0/2144875262 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:22.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26ad000640 1 --2- 172.21.15.98:0/2144875262 >> v2:172.21.15.98:3300/0 conn(0x7f26a8073570 0x7f26a81559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:22.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f269fe00640 1 --2- 172.21.15.98:0/2144875262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26a80cc1f0 0x7f26a81592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:22.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26a7400640 1 -- 172.21.15.98:0/2144875262 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3560572743 0 0) 0x7f26a80c1f60 con 0x7f26a80cbe10 2024-05-27T00:28:22.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26a7400640 1 -- 172.21.15.98:0/2144875262 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f268c003530 con 0x7f26a80cbe10 2024-05-27T00:28:22.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26ad000640 1 -- 172.21.15.98:0/2144875262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26a80cc1f0 msgr2=0x7f26a81592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:22.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26ad000640 1 --2- 172.21.15.98:0/2144875262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26a80cc1f0 0x7f26a81592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:22.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26ad000640 1 -- 172.21.15.98:0/2144875262 >> v1:172.21.15.98:6789/0 conn(0x7f26a80cbe10 legacy=0x7f26a80c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:22.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.026+0000 7f26ad000640 1 -- 172.21.15.98:0/2144875262 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a80c70b0 con 0x7f26a8073570 2024-05-27T00:28:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.030+0000 7f26ad000640 1 --2- 172.21.15.98:0/2144875262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 0x7f26a81559e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009d90 tx=0x7f26a001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a990a33187543bee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.030+0000 7f26a7400640 1 -- 172.21.15.98:0/2144875262 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a00b4070 con 0x7f26a8073570 2024-05-27T00:28:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.030+0000 7f26a7400640 1 -- 172.21.15.98:0/2144875262 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26a00af600 con 0x7f26a8073570 2024-05-27T00:28:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.030+0000 7f26a7400640 1 -- 172.21.15.98:0/2144875262 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a00af9d0 con 0x7f26a8073570 2024-05-27T00:28:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 -- 172.21.15.98:0/2144875262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 msgr2=0x7f26a81559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 --2- 172.21.15.98:0/2144875262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 0x7f26a81559e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009d90 tx=0x7f26a001f150 comp rx=0 tx=0).stop 2024-05-27T00:28:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 -- 172.21.15.98:0/2144875262 shutdown_connections 2024-05-27T00:28:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 --2- 172.21.15.98:0/2144875262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 0x7f26a81559e0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 --2- 172.21.15.98:0/2144875262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26a80cc1f0 0x7f26a81592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:22.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 -- 172.21.15.98:0/2144875262 >> 172.21.15.98:0/2144875262 conn(0x7f26a8067530 msgr2=0x7f26a806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:22.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 -- 172.21.15.98:0/2144875262 shutdown_connections 2024-05-27T00:28:22.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 -- 172.21.15.98:0/2144875262 wait complete. 2024-05-27T00:28:22.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 Processor -- start 2024-05-27T00:28:22.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 -- start start 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 0x7f26a814c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26a80cbe10 0x7f26a80c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26a80cc1f0 0x7f26a814b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f26a80c1330 con 0x7f26a8073570 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f26a80c1050 con 0x7f26a80cbe10 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26af000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f26a80c79d0 con 0x7f26a80cc1f0 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26a7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 0x7f26a814c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26a7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 0x7f26a814c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60798/0 (socket says 172.21.15.98:60798) 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26a7e00640 1 -- 172.21.15.98:0/2410398048 learned_addr learned my addr 172.21.15.98:0/2410398048 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f269fe00640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26a80cbe10 0x7f26a80c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26ad000640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26a80cc1f0 0x7f26a814b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:22.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26a7e00640 1 -- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26a80cc1f0 msgr2=0x7f26a814b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26a7e00640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26a80cc1f0 0x7f26a814b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26a7e00640 1 -- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26a80cbe10 msgr2=0x7f26a80c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26a7e00640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26a80cbe10 0x7f26a80c7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26a7e00640 1 -- 172.21.15.98:0/2410398048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a81f0cc0 con 0x7f26a8073570 2024-05-27T00:28:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26a7e00640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 0x7f26a814c7c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f269800c480 tx=0x7f269800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.034+0000 7f26a4c00640 1 -- 172.21.15.98:0/2410398048 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2698017070 con 0x7f26a8073570 2024-05-27T00:28:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.038+0000 7f26a4c00640 1 -- 172.21.15.98:0/2410398048 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2698005cf0 con 0x7f26a8073570 2024-05-27T00:28:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.038+0000 7f26a4c00640 1 -- 172.21.15.98:0/2410398048 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2698007290 con 0x7f26a8073570 2024-05-27T00:28:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.038+0000 7f26af000640 1 -- 172.21.15.98:0/2410398048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a81f0fa0 con 0x7f26a8073570 2024-05-27T00:28:22.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.038+0000 7f26af000640 1 -- 172.21.15.98:0/2410398048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a80710f0 con 0x7f26a8073570 2024-05-27T00:28:22.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.038+0000 7f26af000640 1 -- 172.21.15.98:0/2410398048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a8075cb0 con 0x7f26a8073570 2024-05-27T00:28:22.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.038+0000 7f26a4c00640 1 -- 172.21.15.98:0/2410398048 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f269801a070 con 0x7f26a8073570 2024-05-27T00:28:22.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.038+0000 7f26a4c00640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f26840678c0 0x7f2684069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:22.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.038+0000 7f26a4c00640 1 -- 172.21.15.98:0/2410398048 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f2698085440 con 0x7f26a8073570 2024-05-27T00:28:22.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.038+0000 7f269fe00640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f26840678c0 0x7f2684069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:22.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.042+0000 7f269fe00640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f26840678c0 0x7f2684069d80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f26a8153160 tx=0x7f2690008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:22.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.042+0000 7f26a4c00640 1 -- 172.21.15.98:0/2410398048 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f2698055bf0 con 0x7f26a8073570 2024-05-27T00:28:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.238+0000 7f26af000640 1 -- 172.21.15.98:0/2410398048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26a814cd00 con 0x7f26a8073570 2024-05-27T00:28:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.238+0000 7f26a4c00640 1 -- 172.21.15.98:0/2410398048 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6014 (secure 0 0 0) 0x7f26980554f0 con 0x7f26a8073570 2024-05-27T00:28:22.241 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:22.241 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 56 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 -- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f26840678c0 msgr2=0x7f2684069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f26840678c0 0x7f2684069d80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f26a8153160 tx=0x7f2690008040 comp rx=0 tx=0).stop 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 -- 172.21.15.98:0/2410398048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 msgr2=0x7f26a814c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 0x7f26a814c7c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f269800c480 tx=0x7f269800c940 comp rx=0 tx=0).stop 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 -- 172.21.15.98:0/2410398048 shutdown_connections 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26a80cc1f0 0x7f26a814b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f26840678c0 0x7f2684069d80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26a80cbe10 0x7f26a80c7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 --2- 172.21.15.98:0/2410398048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26a8073570 0x7f26a814c7c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 -- 172.21.15.98:0/2410398048 >> 172.21.15.98:0/2410398048 conn(0x7f26a8067530 msgr2=0x7f26a80ca470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 -- 172.21.15.98:0/2410398048 shutdown_connections 2024-05-27T00:28:22.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:22.242+0000 7f26af000640 1 -- 172.21.15.98:0/2410398048 wait complete. 2024-05-27T00:28:22.261 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-05-27T00:28:24.043 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 79% (145/183) Updating files: 80% (147/183) Updating files: 81% (149/183) Updating files: 82% (151/183) Updating files: 83% (152/183) Updating files: 84% (154/183) Updating files: 85% (156/183) Updating files: 86% (158/183) Updating files: 87% (160/183) Updating files: 88% (162/183) Updating files: 89% (163/183) Updating files: 90% (165/183) Updating files: 91% (167/183) Updating files: 92% (169/183) Updating files: 93% (171/183) Updating files: 94% (173/183) Updating files: 95% (174/183) Updating files: 96% (176/183) Updating files: 97% (178/183) Updating files: 98% (180/183) Updating files: 99% (182/183) Updating files: 100% (183/183) Updating files: 100% (183/183), done. 2024-05-27T00:28:24.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:24.263 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at e48859474c4 v0.52 2024-05-27T00:28:24.268 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.52 2024-05-27T00:28:24.287 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 53 -eq 48 ] 2024-05-27T00:28:24.287 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.53 2024-05-27T00:28:24.287 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.53 2024-05-27T00:28:24.287 INFO:tasks.workunit.client.0.smithi098.stdout:v0.53 2024-05-27T00:28:24.288 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.53 2024-05-27T00:28:24.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.410+0000 7f1b64600640 1 Processor -- start 2024-05-27T00:28:24.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.410+0000 7f1b64600640 1 -- start start 2024-05-27T00:28:24.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.410+0000 7f1b64600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b5c06c620 0x7f1b5c067510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:24.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.410+0000 7f1b64600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1b5c0c78a0 0x7f1b5c0c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:24.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.410+0000 7f1b64600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b5c06cad0 con 0x7f1b5c067130 2024-05-27T00:28:24.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.410+0000 7f1b64600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1b5c06cdb0 con 0x7f1b5c06c620 2024-05-27T00:28:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.410+0000 7f1b64600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f1b5c06c220 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.410+0000 7f1b61200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b5c06c620 0x7f1b5c067510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.410+0000 7f1b61200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b5c06c620 0x7f1b5c067510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47090/0 (socket says 172.21.15.98:47090) 2024-05-27T00:28:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.410+0000 7f1b61200640 1 -- 172.21.15.98:0/1768199481 learned_addr learned my addr 172.21.15.98:0/1768199481 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.414+0000 7f1b62600640 1 --2- 172.21.15.98:0/1768199481 >> v2:172.21.15.204:3300/0 conn(0x7f1b5c0c78a0 0x7f1b5c0c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.414+0000 7f1b61200640 1 -- 172.21.15.98:0/1768199481 >> v2:172.21.15.204:3300/0 conn(0x7f1b5c0c78a0 msgr2=0x7f1b5c0c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:24.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.414+0000 7f1b61200640 1 --2- 172.21.15.98:0/1768199481 >> v2:172.21.15.204:3300/0 conn(0x7f1b5c0c78a0 0x7f1b5c0c92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:24.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.414+0000 7f1b61200640 1 -- 172.21.15.98:0/1768199481 >> v1:172.21.15.204:6789/0 conn(0x7f1b5c067130 legacy=0x7f1b5c0732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:24.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.414+0000 7f1b61200640 1 -- 172.21.15.98:0/1768199481 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c06bc60 con 0x7f1b5c06c620 2024-05-27T00:28:24.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.414+0000 7f1b61200640 1 --2- 172.21.15.98:0/1768199481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c06c620 0x7f1b5c067510 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c06f340 tx=0x7f1b4801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779ba0c5b6438e2b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:24.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b5be00640 1 -- 172.21.15.98:0/1768199481 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b480b0070 con 0x7f1b5c06c620 2024-05-27T00:28:24.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b5be00640 1 -- 172.21.15.98:0/1768199481 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b480ab600 con 0x7f1b5c06c620 2024-05-27T00:28:24.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b5be00640 1 -- 172.21.15.98:0/1768199481 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b480ab9d0 con 0x7f1b5c06c620 2024-05-27T00:28:24.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b64600640 1 -- 172.21.15.98:0/1768199481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c06c620 msgr2=0x7f1b5c067510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:24.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b64600640 1 --2- 172.21.15.98:0/1768199481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c06c620 0x7f1b5c067510 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c06f340 tx=0x7f1b4801c520 comp rx=0 tx=0).stop 2024-05-27T00:28:24.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b64600640 1 -- 172.21.15.98:0/1768199481 shutdown_connections 2024-05-27T00:28:24.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b64600640 1 --2- 172.21.15.98:0/1768199481 >> v2:172.21.15.204:3300/0 conn(0x7f1b5c0c78a0 0x7f1b5c0c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:24.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b64600640 1 --2- 172.21.15.98:0/1768199481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c06c620 0x7f1b5c067510 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:24.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b64600640 1 -- 172.21.15.98:0/1768199481 >> 172.21.15.98:0/1768199481 conn(0x7f1b5c0c9b70 msgr2=0x7f1b5c0c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:24.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b64600640 1 -- 172.21.15.98:0/1768199481 shutdown_connections 2024-05-27T00:28:24.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b64600640 1 -- 172.21.15.98:0/1768199481 wait complete. 2024-05-27T00:28:24.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b64600640 1 Processor -- start 2024-05-27T00:28:24.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.418+0000 7f1b64600640 1 -- start start 2024-05-27T00:28:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b64600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b5c067130 0x7f1b5c0cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b64600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b5c06c620 0x7f1b5c0b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b64600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c0c78a0 0x7f1b5c14ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b64600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c15a930 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b64600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c0c97e0 con 0x7f1b5c067130 2024-05-27T00:28:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b64600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1b5c154f90 con 0x7f1b5c06c620 2024-05-27T00:28:24.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b62600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c0c78a0 0x7f1b5c14ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b62600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c0c78a0 0x7f1b5c14ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47106/0 (socket says 172.21.15.98:47106) 2024-05-27T00:28:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b62600640 1 -- 172.21.15.98:0/551598964 learned_addr learned my addr 172.21.15.98:0/551598964 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b61200640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b5c06c620 0x7f1b5c0b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:24.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b61c00640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b5c067130 0x7f1b5c0cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b62600640 1 -- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b5c06c620 msgr2=0x7f1b5c0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b62600640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b5c06c620 0x7f1b5c0b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b62600640 1 -- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b5c067130 msgr2=0x7f1b5c0cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b62600640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b5c067130 0x7f1b5c0cc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b62600640 1 -- 172.21.15.98:0/551598964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c0b1550 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b62600640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c0c78a0 0x7f1b5c14ecd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1b54007d80 tx=0x7f1b5400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b59600640 1 -- 172.21.15.98:0/551598964 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b54014420 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b59600640 1 -- 172.21.15.98:0/551598964 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b54019070 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b59600640 1 -- 172.21.15.98:0/551598964 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b54014800 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b64600640 1 -- 172.21.15.98:0/551598964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5c1f6da0 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b64600640 1 -- 172.21.15.98:0/551598964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5c1f7360 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.422+0000 7f1b64600640 1 -- 172.21.15.98:0/551598964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5c075cb0 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.426+0000 7f1b59600640 1 -- 172.21.15.98:0/551598964 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1b540149a0 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.426+0000 7f1b59600640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b3c0678c0 0x7f1b3c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:24.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.426+0000 7f1b59600640 1 -- 172.21.15.98:0/551598964 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f1b540866f0 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.426+0000 7f1b61c00640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b3c0678c0 0x7f1b3c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:24.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.426+0000 7f1b61c00640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b3c0678c0 0x7f1b3c069d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1b50007fd0 tx=0x7f1b50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:24.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.430+0000 7f1b59600640 1 -- 172.21.15.98:0/551598964 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1b54056860 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.674+0000 7f1b64600640 1 -- 172.21.15.98:0/551598964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b5c0b24a0 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.674+0000 7f1b59600640 1 -- 172.21.15.98:0/551598964 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6839 (secure 0 0 0) 0x7f1b5405ab60 con 0x7f1b5c0c78a0 2024-05-27T00:28:24.675 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:24.675 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:24.675 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 57 2024-05-27T00:28:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 -- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b3c0678c0 msgr2=0x7f1b3c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b3c0678c0 0x7f1b3c069d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1b50007fd0 tx=0x7f1b50009040 comp rx=0 tx=0).stop 2024-05-27T00:28:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 -- 172.21.15.98:0/551598964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c0c78a0 msgr2=0x7f1b5c14ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:24.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c0c78a0 0x7f1b5c14ecd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1b54007d80 tx=0x7f1b5400e4b0 comp rx=0 tx=0).stop 2024-05-27T00:28:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 -- 172.21.15.98:0/551598964 shutdown_connections 2024-05-27T00:28:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c0c78a0 0x7f1b5c14ecd0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b5c06c620 0x7f1b5c0b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b3c0678c0 0x7f1b3c069d80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 --2- 172.21.15.98:0/551598964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b5c067130 0x7f1b5c0cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 -- 172.21.15.98:0/551598964 >> 172.21.15.98:0/551598964 conn(0x7f1b5c0c9b70 msgr2=0x7f1b5c0c8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 -- 172.21.15.98:0/551598964 shutdown_connections 2024-05-27T00:28:24.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:24.678+0000 7f1b64600640 1 -- 172.21.15.98:0/551598964 wait complete. 2024-05-27T00:28:24.705 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-05-27T00:28:24.706 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 10 secs before thrashing 2024-05-27T00:28:26.560 INFO:tasks.ceph.mds.a.smithi098.stderr:2024-05-27T00:28:26.558+0000 7f0366000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:28:26.907 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 62% (241/386) Updating files: 63% (244/386) Updating files: 64% (248/386) Updating files: 65% (251/386) Updating files: 66% (255/386) Updating files: 67% (259/386) Updating files: 68% (263/386) Updating files: 69% (267/386) Updating files: 70% (271/386) Updating files: 71% (275/386) Updating files: 72% (278/386) Updating files: 73% (282/386) Updating files: 74% (286/386) Updating files: 75% (290/386) Updating files: 76% (294/386) Updating files: 77% (298/386) Updating files: 78% (302/386) Updating files: 79% (305/386) Updating files: 80% (309/386) Updating files: 81% (313/386) Updating files: 82% (317/386) Updating files: 83% (321/386) Updating files: 84% (325/386) Updating files: 85% (329/386) Updating files: 86% (332/386) Updating files: 87% (336/386) Updating files: 88% (340/386) Updating files: 89% (344/386) Updating files: 90% (348/386) Updating files: 91% (352/386) Updating files: 92% (356/386) Updating files: 93% (359/386) Updating files: 94% (363/386) Updating files: 95% (367/386) Updating files: 96% (371/386) Updating files: 97% (375/386) Updating files: 98% (379/386) Updating files: 99% (383/386) Updating files: 100% (386/386) Updating files: 100% (386/386), done. 2024-05-27T00:28:26.951 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 2528b5ee105 v0.43 2024-05-27T00:28:26.956 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.53 2024-05-27T00:28:26.987 INFO:tasks.workunit.client.0.smithi098.stdout:v0.54 2024-05-27T00:28:26.987 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 54 -eq 48 ] 2024-05-27T00:28:26.987 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.54 2024-05-27T00:28:26.987 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.54 2024-05-27T00:28:26.987 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.54 2024-05-27T00:28:29.834 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 67% (144/212) Updating files: 68% (145/212) Updating files: 69% (147/212) Updating files: 70% (149/212) Updating files: 71% (151/212) Updating files: 72% (153/212) Updating files: 73% (155/212) Updating files: 74% (157/212) Updating files: 75% (159/212) Updating files: 76% (162/212) Updating files: 77% (164/212) Updating files: 78% (166/212) Updating files: 79% (168/212) Updating files: 80% (170/212) Updating files: 81% (172/212) Updating files: 82% (174/212) Updating files: 83% (176/212) Updating files: 84% (179/212) Updating files: 85% (181/212) Updating files: 86% (183/212) Updating files: 87% (185/212) Updating files: 88% (187/212) Updating files: 89% (189/212) Updating files: 90% (191/212) Updating files: 91% (193/212) Updating files: 92% (196/212) Updating files: 93% (198/212) Updating files: 94% (200/212) Updating files: 95% (202/212) Updating files: 96% (204/212) Updating files: 97% (206/212) Updating files: 98% (208/212) Updating files: 99% (210/212) Updating files: 100% (212/212) Updating files: 100% (212/212), done. 2024-05-27T00:28:29.915 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 60b84b095b1 v0.54 2024-05-27T00:28:29.926 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.54 2024-05-27T00:28:29.981 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 55 -eq 48 ] 2024-05-27T00:28:29.981 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.55 2024-05-27T00:28:29.982 INFO:tasks.workunit.client.0.smithi098.stdout:v0.55 2024-05-27T00:28:29.982 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.55 2024-05-27T00:28:29.982 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.55 2024-05-27T00:28:31.970 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 57% (309/540) Updating files: 58% (314/540) Updating files: 59% (319/540) Updating files: 60% (324/540) Updating files: 61% (330/540) Updating files: 62% (335/540) Updating files: 63% (341/540) Updating files: 64% (346/540) Updating files: 65% (351/540) Updating files: 66% (357/540) Updating files: 67% (362/540) Updating files: 68% (368/540) Updating files: 69% (373/540) Updating files: 70% (378/540) Updating files: 71% (384/540) Updating files: 72% (389/540) Updating files: 73% (395/540) Updating files: 74% (400/540) Updating files: 75% (405/540) Updating files: 76% (411/540) Updating files: 77% (416/540) Updating files: 78% (422/540) Updating files: 79% (427/540) Updating files: 80% (432/540) Updating files: 81% (438/540) Updating files: 82% (443/540) Updating files: 83% (449/540) Updating files: 84% (454/540) Updating files: 85% (459/540) Updating files: 86% (465/540) Updating files: 87% (470/540) Updating files: 88% (476/540) Updating files: 89% (481/540) Updating files: 90% (486/540) Updating files: 91% (492/540) Updating files: 92% (497/540) Updating files: 93% (503/540) Updating files: 94% (508/540) Updating files: 95% (513/540) Updating files: 96% (519/540) Updating files: 97% (524/540) Updating files: 98% (530/540) Updating files: 99% (535/540) Updating files: 100% (540/540) Updating files: 100% (540/540), done. 2024-05-27T00:28:32.014 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 690f8175606 v0.55 2024-05-27T00:28:32.021 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.55 2024-05-27T00:28:32.143 INFO:tasks.workunit.client.0.smithi098.stdout:v0.56 2024-05-27T00:28:32.143 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 56 -eq 48 ] 2024-05-27T00:28:32.143 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.56 2024-05-27T00:28:32.143 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.56 2024-05-27T00:28:32.143 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.56 2024-05-27T00:28:34.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f7185000640 1 Processor -- start 2024-05-27T00:28:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f7185000640 1 -- start start 2024-05-27T00:28:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f7185000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7180073570 0x7f71801592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f7185000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71800c1f60 con 0x7f71800cbe10 2024-05-27T00:28:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f7185000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71800c6dd0 con 0x7f71800cc1f0 2024-05-27T00:28:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f7185000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7180159810 con 0x7f7180073570 2024-05-27T00:28:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717de00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f71800cbe10 0x7f71800c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51196/0 (socket says 172.21.15.98:51196) 2024-05-27T00:28:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717de00640 1 -- 172.21.15.98:0/1196607260 learned_addr learned my addr 172.21.15.98:0/1196607260 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:34.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717e800640 1 --2- 172.21.15.98:0/1196607260 >> v2:172.21.15.204:3300/0 conn(0x7f7180073570 0x7f71801592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:34.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3126038197 0 0) 0x7f71800c1f60 con 0x7f71800cbe10 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7164003530 con 0x7f71800cbe10 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2407700151 0 0) 0x7f71800c6dd0 con 0x7f71800cc1f0 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71800c1f60 con 0x7f71800cc1f0 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1195593278 0 0) 0x7f7164003530 con 0x7f71800cbe10 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 >> v2:172.21.15.204:3300/0 conn(0x7f7180073570 msgr2=0x7f71801592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 --2- 172.21.15.98:0/1196607260 >> v2:172.21.15.204:3300/0 conn(0x7f7180073570 0x7f71801592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 >> v1:172.21.15.204:6789/0 conn(0x7f71800cc1f0 legacy=0x7f71801559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71800c70b0 con 0x7f71800cbe10 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f7170001680 con 0x7f71800cbe10 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 >> v1:172.21.15.98:6789/0 conn(0x7f71800cbe10 legacy=0x7f71800c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7164003b50 0x7f7164003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7164004810 0x7f7164004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640051d0 0x7f716400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7180159810 con 0x7f71640051d0 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f716400bfc0 con 0x7f7164003b50 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f716400c1c0 con 0x7f7164004810 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.846+0000 7f717e800640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640051d0 0x7f716400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717de00640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7164003b50 0x7f7164003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717d400640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7164004810 0x7f7164004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717e800640 1 -- 172.21.15.98:0/1196607260 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7164004810 msgr2=0x7f7164004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717e800640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7164004810 0x7f7164004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717e800640 1 -- 172.21.15.98:0/1196607260 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7164003b50 msgr2=0x7f7164003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717e800640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7164003b50 0x7f7164003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717e800640 1 -- 172.21.15.98:0/1196607260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7164003530 con 0x7f71640051d0 2024-05-27T00:28:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717e800640 1 -- 172.21.15.98:0/1196607260 reap_dead start 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717e800640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640051d0 0x7f716400b9d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7178009f90 tx=0x7f717801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498fca6ed828617e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178009c10 con 0x7f71640051d0 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71780a5c90 con 0x7f71640051d0 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f717ca00640 1 -- 172.21.15.98:0/1196607260 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178003be0 con 0x7f71640051d0 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f7185000640 1 -- 172.21.15.98:0/1196607260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640051d0 msgr2=0x7f716400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f7185000640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640051d0 0x7f716400b9d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7178009f90 tx=0x7f717801cc80 comp rx=0 tx=0).stop 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f7185000640 1 -- 172.21.15.98:0/1196607260 shutdown_connections 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.850+0000 7f7185000640 1 --2- 172.21.15.98:0/1196607260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71640051d0 0x7f716400b9d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.854+0000 7f7185000640 1 -- 172.21.15.98:0/1196607260 >> 172.21.15.98:0/1196607260 conn(0x7f7180067530 msgr2=0x7f718006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.854+0000 7f7185000640 1 -- 172.21.15.98:0/1196607260 shutdown_connections 2024-05-27T00:28:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.854+0000 7f7185000640 1 -- 172.21.15.98:0/1196607260 wait complete. 2024-05-27T00:28:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.854+0000 7f7185000640 1 Processor -- start 2024-05-27T00:28:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.854+0000 7f7185000640 1 -- start start 2024-05-27T00:28:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.854+0000 7f7185000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71640051d0 0x7f71800c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.854+0000 7f7185000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71800b0fc0 0x7f718015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.854+0000 7f7185000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71801edf00 0x7f71801ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f7185000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f71800c1330 con 0x7f71800b0fc0 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f7185000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f71800c1050 con 0x7f71801edf00 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f7185000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f71800c79d0 con 0x7f71640051d0 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71800b0fc0 0x7f718015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71800b0fc0 0x7f718015a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54988/0 (socket says 172.21.15.98:54988) 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717d400640 1 -- 172.21.15.98:0/2075328058 learned_addr learned my addr 172.21.15.98:0/2075328058 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717e800640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71801edf00 0x7f71801ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717de00640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71640051d0 0x7f71800c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717d400640 1 -- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71640051d0 msgr2=0x7f71800c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717d400640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71640051d0 0x7f71800c6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717d400640 1 -- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71801edf00 msgr2=0x7f71801ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717d400640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71801edf00 0x7f71801ee380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717d400640 1 -- 172.21.15.98:0/2075328058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718015c5c0 con 0x7f71800b0fc0 2024-05-27T00:28:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f717d400640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71800b0fc0 0x7f718015a7e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f716c00f7a0 tx=0x7f716c00fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f7176000640 1 -- 172.21.15.98:0/2075328058 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c01a070 con 0x7f71800b0fc0 2024-05-27T00:28:34.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f7176000640 1 -- 172.21.15.98:0/2075328058 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f716c005320 con 0x7f71800b0fc0 2024-05-27T00:28:34.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f7176000640 1 -- 172.21.15.98:0/2075328058 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c006a20 con 0x7f71800b0fc0 2024-05-27T00:28:34.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f7185000640 1 -- 172.21.15.98:0/2075328058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71801ee8c0 con 0x7f71800b0fc0 2024-05-27T00:28:34.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f7185000640 1 -- 172.21.15.98:0/2075328058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7180159c70 con 0x7f71800b0fc0 2024-05-27T00:28:34.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f7185000640 1 -- 172.21.15.98:0/2075328058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7180152220 con 0x7f71800b0fc0 2024-05-27T00:28:34.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.858+0000 7f7176000640 1 -- 172.21.15.98:0/2075328058 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f716c006bc0 con 0x7f71800b0fc0 2024-05-27T00:28:34.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.862+0000 7f7176000640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71600678c0 0x7f7160069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:34.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.862+0000 7f7176000640 1 -- 172.21.15.98:0/2075328058 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f716c0856c0 con 0x7f71800b0fc0 2024-05-27T00:28:34.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.862+0000 7f717de00640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71600678c0 0x7f7160069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:34.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.862+0000 7f717de00640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71600678c0 0x7f7160069d80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7170001060 tx=0x7f7170001e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:34.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:34.866+0000 7f7176000640 1 -- 172.21.15.98:0/2075328058 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f71801ee8c0 con 0x7f71800b0fc0 2024-05-27T00:28:35.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.054+0000 7f7185000640 1 -- 172.21.15.98:0/2075328058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7180075cb0 con 0x7f71800b0fc0 2024-05-27T00:28:35.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.054+0000 7f7176000640 1 -- 172.21.15.98:0/2075328058 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6839 (secure 0 0 0) 0x7f716c055820 con 0x7f71800b0fc0 2024-05-27T00:28:35.056 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:35.056 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:35.057 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 57 2024-05-27T00:28:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 -- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71600678c0 msgr2=0x7f7160069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71600678c0 0x7f7160069d80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7170001060 tx=0x7f7170001e50 comp rx=0 tx=0).stop 2024-05-27T00:28:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 -- 172.21.15.98:0/2075328058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71800b0fc0 msgr2=0x7f718015a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71800b0fc0 0x7f718015a7e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f716c00f7a0 tx=0x7f716c00fba0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 -- 172.21.15.98:0/2075328058 shutdown_connections 2024-05-27T00:28:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f71600678c0 0x7f7160069d80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71801edf00 0x7f71801ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71800b0fc0 0x7f718015a7e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 --2- 172.21.15.98:0/2075328058 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71640051d0 0x7f71800c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 -- 172.21.15.98:0/2075328058 >> 172.21.15.98:0/2075328058 conn(0x7f7180067530 msgr2=0x7f718006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 -- 172.21.15.98:0/2075328058 shutdown_connections 2024-05-27T00:28:35.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.058+0000 7f7185000640 1 -- 172.21.15.98:0/2075328058 wait complete. 2024-05-27T00:28:35.077 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2024-05-27T00:28:35.077 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-27T00:28:35.078 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:28:35.110 INFO:tasks.ceph.mds.a:Stopped 2024-05-27T00:28:35.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026fc00640 1 Processor -- start 2024-05-27T00:28:35.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026fc00640 1 -- start start 2024-05-27T00:28:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026fc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02680cb7e0 0x7f0268158750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026fc00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026815be40 con 0x7f0268152a50 2024-05-27T00:28:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026fc00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02680c50e0 con 0x7f02680b3ed0 2024-05-27T00:28:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026fc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f02680c8a00 con 0x7f02680cb7e0 2024-05-27T00:28:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026d200640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0268152a50 0x7f02680709a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51200/0 (socket says 172.21.15.98:51200) 2024-05-27T00:28:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026d200640 1 -- 172.21.15.98:0/1238078864 learned_addr learned my addr 172.21.15.98:0/1238078864 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267e00640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02680cb7e0 0x7f0268158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 557133344 0 0) 0x7f026815be40 con 0x7f0268152a50 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0250003530 con 0x7f0268152a50 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 707532555 0 0) 0x7f02680c50e0 con 0x7f02680b3ed0 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f026815be40 con 0x7f02680b3ed0 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 752549535 0 0) 0x7f0250003530 con 0x7f0268152a50 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02680cb7e0 msgr2=0x7f0268158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02680cb7e0 0x7f0268158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 >> v1:172.21.15.204:6789/0 conn(0x7f02680b3ed0 legacy=0x7f0268162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02680c8c50 con 0x7f0268152a50 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f025c0016a0 con 0x7f0268152a50 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 >> v1:172.21.15.98:6789/0 conn(0x7f0268152a50 legacy=0x7f02680709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0250003b80 0x7f0250003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02680cb7e0 0x7f0250004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0250004d50 0x7f02500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02680c8a00 con 0x7f0250004d50 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f025000bb60 con 0x7f0250003b80 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f025000bd30 con 0x7f02680cb7e0 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026dc00640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0250004d50 0x7f02500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026d200640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0250003b80 0x7f0250003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267e00640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02680cb7e0 0x7f0250004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026dc00640 1 -- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02680cb7e0 msgr2=0x7f0250004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026dc00640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02680cb7e0 0x7f0250004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026dc00640 1 -- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0250003b80 msgr2=0x7f0250003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026dc00640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0250003b80 0x7f0250003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f026dc00640 1 -- 172.21.15.98:0/1238078864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0250003530 con 0x7f0250004d50 2024-05-27T00:28:35.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.290+0000 7f0267e00640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02680cb7e0 0x7f0250004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:35.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f026dc00640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0250004d50 0x7f02500051d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0260002000 tx=0x7f026001f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa908c4265bc75 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0260001c20 con 0x7f0250004d50 2024-05-27T00:28:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02600a8cb0 con 0x7f0250004d50 2024-05-27T00:28:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f0267400640 1 -- 172.21.15.98:0/1238078864 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02600a8e60 con 0x7f0250004d50 2024-05-27T00:28:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f026fc00640 1 -- 172.21.15.98:0/1238078864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0250004d50 msgr2=0x7f02500051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f026fc00640 1 --2- 172.21.15.98:0/1238078864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0250004d50 0x7f02500051d0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0260002000 tx=0x7f026001f070 comp rx=0 tx=0).stop 2024-05-27T00:28:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f026dc00640 1 -- 172.21.15.98:0/1238078864 reap_dead start 2024-05-27T00:28:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f026fc00640 1 -- 172.21.15.98:0/1238078864 shutdown_connections 2024-05-27T00:28:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f026fc00640 1 -- 172.21.15.98:0/1238078864 >> 172.21.15.98:0/1238078864 conn(0x7f0268072f90 msgr2=0x7f02680733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f026fc00640 1 -- 172.21.15.98:0/1238078864 shutdown_connections 2024-05-27T00:28:35.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.294+0000 7f026fc00640 1 -- 172.21.15.98:0/1238078864 wait complete. 2024-05-27T00:28:35.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026fc00640 1 Processor -- start 2024-05-27T00:28:35.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026fc00640 1 -- start start 2024-05-27T00:28:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026fc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0250004d50 0x7f026814c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026fc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026806c1c0 0x7f02680ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026fc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f026814e470 0x7f026814bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026fc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02680c94f0 con 0x7f026806c1c0 2024-05-27T00:28:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026fc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f02680c9210 con 0x7f026814e470 2024-05-27T00:28:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026fc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f02680cb570 con 0x7f0250004d50 2024-05-27T00:28:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0267e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026806c1c0 0x7f02680ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0267e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026806c1c0 0x7f02680ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55004/0 (socket says 172.21.15.98:55004) 2024-05-27T00:28:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0267e00640 1 -- 172.21.15.98:0/1291713343 learned_addr learned my addr 172.21.15.98:0/1291713343 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026dc00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f026814e470 0x7f026814bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026d200640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0250004d50 0x7f026814c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0267e00640 1 -- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0250004d50 msgr2=0x7f026814c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0267e00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0250004d50 0x7f026814c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0267e00640 1 -- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f026814e470 msgr2=0x7f026814bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0267e00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f026814e470 0x7f026814bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0267e00640 1 -- 172.21.15.98:0/1291713343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02680ae9a0 con 0x7f026806c1c0 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026d200640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0250004d50 0x7f026814c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0267e00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026806c1c0 0x7f02680ce310 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0254011660 tx=0x7f0254011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0264c00640 1 -- 172.21.15.98:0/1291713343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0254004230 con 0x7f026806c1c0 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0264c00640 1 -- 172.21.15.98:0/1291713343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02540043d0 con 0x7f026806c1c0 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f0264c00640 1 -- 172.21.15.98:0/1291713343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025400fa00 con 0x7f026806c1c0 2024-05-27T00:28:35.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026fc00640 1 -- 172.21.15.98:0/1291713343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f026814cdd0 con 0x7f026806c1c0 2024-05-27T00:28:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.298+0000 7f026fc00640 1 -- 172.21.15.98:0/1291713343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f026814ed10 con 0x7f026806c1c0 2024-05-27T00:28:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.302+0000 7f026fc00640 1 -- 172.21.15.98:0/1291713343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0268075cb0 con 0x7f026806c1c0 2024-05-27T00:28:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.302+0000 7f0264c00640 1 -- 172.21.15.98:0/1291713343 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f025400fba0 con 0x7f026806c1c0 2024-05-27T00:28:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.302+0000 7f0264c00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02480678f0 0x7f0248069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:35.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.302+0000 7f0264c00640 1 -- 172.21.15.98:0/1291713343 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f0254086320 con 0x7f026806c1c0 2024-05-27T00:28:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.302+0000 7f026d200640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02480678f0 0x7f0248069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:35.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.302+0000 7f026d200640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02480678f0 0x7f0248069db0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f025c000c10 tx=0x7f025c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:35.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.306+0000 7f0264c00640 1 -- 172.21.15.98:0/1291713343 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f0254019070 con 0x7f026806c1c0 2024-05-27T00:28:35.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.498+0000 7f026fc00640 1 -- 172.21.15.98:0/1291713343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0268073540 con 0x7f026806c1c0 2024-05-27T00:28:35.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f0264c00640 1 -- 172.21.15.98:0/1291713343 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6839 (secure 0 0 0) 0x7f0254056ae0 con 0x7f026806c1c0 2024-05-27T00:28:35.503 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:35.503 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:35.503 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 57 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f026fc00640 1 -- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02480678f0 msgr2=0x7f0248069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f026fc00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02480678f0 0x7f0248069db0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f025c000c10 tx=0x7f025c009040 comp rx=0 tx=0).stop 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f026fc00640 1 -- 172.21.15.98:0/1291713343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026806c1c0 msgr2=0x7f02680ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f026fc00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026806c1c0 0x7f02680ce310 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0254011660 tx=0x7f0254011b20 comp rx=0 tx=0).stop 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f026fc00640 1 -- 172.21.15.98:0/1291713343 shutdown_connections 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f026fc00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02480678f0 0x7f0248069db0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f026fc00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f026814e470 0x7f026814bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f026fc00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f026806c1c0 0x7f02680ce310 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f026fc00640 1 --2- 172.21.15.98:0/1291713343 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0250004d50 0x7f026814c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.502+0000 7f026fc00640 1 -- 172.21.15.98:0/1291713343 >> 172.21.15.98:0/1291713343 conn(0x7f0268072f90 msgr2=0x7f02680c78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:35.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.506+0000 7f026fc00640 1 -- 172.21.15.98:0/1291713343 shutdown_connections 2024-05-27T00:28:35.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:35.506+0000 7f026fc00640 1 -- 172.21.15.98:0/1291713343 wait complete. 2024-05-27T00:28:35.524 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:28:37.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f6490c00640 1 Processor -- start 2024-05-27T00:28:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f6490c00640 1 -- start start 2024-05-27T00:28:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f6490c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f648c08cbd0 0x7f648c16ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f6490c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f648c164d30 0x7f648c16b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f6490c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648c0c6a10 con 0x7f648c06e7a0 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f6490c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f648c0c6cf0 con 0x7f648c08cbd0 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f6490c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f648c0c6fd0 con 0x7f648c164d30 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f6483e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f648c08cbd0 0x7f648c16ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f648b400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f648c06e7a0 0x7f648c164620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51206/0 (socket says 172.21.15.98:51206) 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f648b400640 1 -- 172.21.15.98:0/1827256744 learned_addr learned my addr 172.21.15.98:0/1827256744 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f648be00640 1 --2- 172.21.15.98:0/1827256744 >> v2:172.21.15.204:3300/0 conn(0x7f648c164d30 0x7f648c16b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.666+0000 7f648aa00640 1 -- 172.21.15.98:0/1827256744 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2390301530 0 0) 0x7f648c0c6a10 con 0x7f648c06e7a0 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.698+0000 7f648aa00640 1 -- 172.21.15.98:0/1827256744 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6470003530 con 0x7f648c06e7a0 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.698+0000 7f6483e00640 1 -- 172.21.15.98:0/1827256744 >> v2:172.21.15.204:3300/0 conn(0x7f648c164d30 msgr2=0x7f648c16b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.698+0000 7f6483e00640 1 --2- 172.21.15.98:0/1827256744 >> v2:172.21.15.204:3300/0 conn(0x7f648c164d30 0x7f648c16b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.698+0000 7f648aa00640 1 -- 172.21.15.98:0/1827256744 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1815732569 0 0) 0x7f6470003530 con 0x7f648c06e7a0 2024-05-27T00:28:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.698+0000 7f6483e00640 1 -- 172.21.15.98:0/1827256744 >> v1:172.21.15.98:6789/0 conn(0x7f648c06e7a0 legacy=0x7f648c164620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.698+0000 7f6483e00640 1 -- 172.21.15.98:0/1827256744 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c0c72b0 con 0x7f648c08cbd0 2024-05-27T00:28:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f6483e00640 1 --2- 172.21.15.98:0/1827256744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c08cbd0 0x7f648c16ac20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f647400a960 tx=0x7f647401d6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07e2684f230a835 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f648aa00640 1 -- 172.21.15.98:0/1827256744 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64740ad070 con 0x7f648c08cbd0 2024-05-27T00:28:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f648aa00640 1 -- 172.21.15.98:0/1827256744 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64740a7470 con 0x7f648c08cbd0 2024-05-27T00:28:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f648aa00640 1 -- 172.21.15.98:0/1827256744 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c001680 con 0x7f648c08cbd0 2024-05-27T00:28:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f6490c00640 1 -- 172.21.15.98:0/1827256744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c08cbd0 msgr2=0x7f648c16ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f6490c00640 1 --2- 172.21.15.98:0/1827256744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c08cbd0 0x7f648c16ac20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f647400a960 tx=0x7f647401d6c0 comp rx=0 tx=0).stop 2024-05-27T00:28:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f6490c00640 1 -- 172.21.15.98:0/1827256744 shutdown_connections 2024-05-27T00:28:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f6490c00640 1 --2- 172.21.15.98:0/1827256744 >> v2:172.21.15.204:3300/0 conn(0x7f648c164d30 0x7f648c16b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f6490c00640 1 --2- 172.21.15.98:0/1827256744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c08cbd0 0x7f648c16ac20 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f6490c00640 1 -- 172.21.15.98:0/1827256744 >> 172.21.15.98:0/1827256744 conn(0x7f648c0cbe00 msgr2=0x7f648c0cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f6490c00640 1 -- 172.21.15.98:0/1827256744 shutdown_connections 2024-05-27T00:28:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.702+0000 7f6490c00640 1 -- 172.21.15.98:0/1827256744 wait complete. 2024-05-27T00:28:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6490c00640 1 Processor -- start 2024-05-27T00:28:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6490c00640 1 -- start start 2024-05-27T00:28:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6490c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c06e7a0 0x7f648c072380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6490c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f648c08cbd0 0x7f648c168570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6490c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f648c164d30 0x7f648c1671f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6490c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f648c06f390 con 0x7f648c06e7a0 2024-05-27T00:28:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6490c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f648c1744d0 con 0x7f648c08cbd0 2024-05-27T00:28:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6490c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f648c15e880 con 0x7f648c164d30 2024-05-27T00:28:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f648b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c06e7a0 0x7f648c072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f648b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c06e7a0 0x7f648c072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55030/0 (socket says 172.21.15.98:55030) 2024-05-27T00:28:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f648b400640 1 -- 172.21.15.98:0/3160041107 learned_addr learned my addr 172.21.15.98:0/3160041107 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f648be00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f648c164d30 0x7f648c1671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6483e00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f648c08cbd0 0x7f648c168570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f648b400640 1 -- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f648c164d30 msgr2=0x7f648c1671f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f648b400640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f648c164d30 0x7f648c1671f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f648b400640 1 -- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f648c08cbd0 msgr2=0x7f648c168570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f648b400640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f648c08cbd0 0x7f648c168570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f648b400640 1 -- 172.21.15.98:0/3160041107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c16a190 con 0x7f648c06e7a0 2024-05-27T00:28:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6483e00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f648c08cbd0 0x7f648c168570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:37.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f648b400640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c06e7a0 0x7f648c072380 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f647c00c780 tx=0x7f647c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:37.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6483400640 1 -- 172.21.15.98:0/3160041107 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c005ae0 con 0x7f648c06e7a0 2024-05-27T00:28:37.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6483400640 1 -- 172.21.15.98:0/3160041107 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f647c005ca0 con 0x7f648c06e7a0 2024-05-27T00:28:37.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6490c00640 1 -- 172.21.15.98:0/3160041107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c168180 con 0x7f648c06e7a0 2024-05-27T00:28:37.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6483400640 1 -- 172.21.15.98:0/3160041107 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c0072b0 con 0x7f648c06e7a0 2024-05-27T00:28:37.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.706+0000 7f6490c00640 1 -- 172.21.15.98:0/3160041107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c204d60 con 0x7f648c06e7a0 2024-05-27T00:28:37.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.710+0000 7f6490c00640 1 -- 172.21.15.98:0/3160041107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648c075c40 con 0x7f648c06e7a0 2024-05-27T00:28:37.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.710+0000 7f6483400640 1 -- 172.21.15.98:0/3160041107 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f647c007450 con 0x7f648c06e7a0 2024-05-27T00:28:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.710+0000 7f6483400640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f64680678f0 0x7f6468069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:37.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.710+0000 7f6483400640 1 -- 172.21.15.98:0/3160041107 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f647c085fc0 con 0x7f648c06e7a0 2024-05-27T00:28:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.710+0000 7f6483e00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f64680678f0 0x7f6468069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.714+0000 7f6483e00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f64680678f0 0x7f6468069db0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f64740af000 tx=0x7f64740b4040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:37.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.714+0000 7f6483400640 1 -- 172.21.15.98:0/3160041107 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f647c056760 con 0x7f648c06e7a0 2024-05-27T00:28:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.906+0000 7f6490c00640 1 -- 172.21.15.98:0/3160041107 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f648c0c2ac0 con 0x7f648c06e7a0 2024-05-27T00:28:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.906+0000 7f6483400640 1 -- 172.21.15.98:0/3160041107 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6839 (secure 0 0 0) 0x7f647c056060 con 0x7f648c06e7a0 2024-05-27T00:28:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:37.909 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 57 2024-05-27T00:28:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 -- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f64680678f0 msgr2=0x7f6468069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f64680678f0 0x7f6468069db0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f64740af000 tx=0x7f64740b4040 comp rx=0 tx=0).stop 2024-05-27T00:28:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 -- 172.21.15.98:0/3160041107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c06e7a0 msgr2=0x7f648c072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c06e7a0 0x7f648c072380 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f647c00c780 tx=0x7f647c00cc40 comp rx=0 tx=0).stop 2024-05-27T00:28:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 -- 172.21.15.98:0/3160041107 shutdown_connections 2024-05-27T00:28:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f648c164d30 0x7f648c1671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f64680678f0 0x7f6468069db0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f648c08cbd0 0x7f648c168570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 --2- 172.21.15.98:0/3160041107 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f648c06e7a0 0x7f648c072380 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 -- 172.21.15.98:0/3160041107 >> 172.21.15.98:0/3160041107 conn(0x7f648c0cbe00 msgr2=0x7f648c0c1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 -- 172.21.15.98:0/3160041107 shutdown_connections 2024-05-27T00:28:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:37.910+0000 7f6490c00640 1 -- 172.21.15.98:0/3160041107 wait complete. 2024-05-27T00:28:37.931 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:28:39.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c1be00640 1 Processor -- start 2024-05-27T00:28:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c1be00640 1 -- start start 2024-05-27T00:28:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c1be00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8c14158e60 0x7f8c14164000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c1be00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c1415be40 con 0x7f8c140cb7e0 2024-05-27T00:28:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c1be00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c140c50e0 con 0x7f8c14152a50 2024-05-27T00:28:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c1be00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f8c140c8a00 con 0x7f8c14158e60 2024-05-27T00:28:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c18a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8c140cb7e0 0x7f8c14158750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51222/0 (socket says 172.21.15.98:51222) 2024-05-27T00:28:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c18a00640 1 -- 172.21.15.98:0/722852356 learned_addr learned my addr 172.21.15.98:0/722852356 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c19e00640 1 --2- 172.21.15.98:0/722852356 >> v2:172.21.15.204:3300/0 conn(0x7f8c14158e60 0x7f8c14164000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 214257256 0 0) 0x7f8c1415be40 con 0x7f8c140cb7e0 2024-05-27T00:28:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bfc003530 con 0x7f8c140cb7e0 2024-05-27T00:28:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2026603711 0 0) 0x7f8bfc003530 con 0x7f8c140cb7e0 2024-05-27T00:28:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 >> v2:172.21.15.204:3300/0 conn(0x7f8c14158e60 msgr2=0x7f8c14164000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 --2- 172.21.15.98:0/722852356 >> v2:172.21.15.204:3300/0 conn(0x7f8c14158e60 0x7f8c14164000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 >> v1:172.21.15.204:6789/0 conn(0x7f8c14152a50 legacy=0x7f8c140709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c140c8c50 con 0x7f8c140cb7e0 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f8c000016a0 con 0x7f8c140cb7e0 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 >> v1:172.21.15.98:6789/0 conn(0x7f8c140cb7e0 legacy=0x7f8c14158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bfc0039c0 0x7f8bfc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bfc0043c0 0x7f8bfc004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bfc005130 0x7f8bfc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c140c8a00 con 0x7f8bfc0039c0 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8bfc00c070 con 0x7f8bfc0043c0 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8bfc00c270 con 0x7f8bfc005130 2024-05-27T00:28:40.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c19400640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bfc0039c0 0x7f8bfc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c18a00640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bfc0043c0 0x7f8bfc004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.074+0000 7f8c19e00640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bfc005130 0x7f8bfc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c19400640 1 -- 172.21.15.98:0/722852356 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bfc005130 msgr2=0x7f8bfc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c19400640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8bfc005130 0x7f8bfc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c19400640 1 -- 172.21.15.98:0/722852356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bfc0043c0 msgr2=0x7f8bfc004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c19400640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bfc0043c0 0x7f8bfc004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c19400640 1 -- 172.21.15.98:0/722852356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bfc003530 con 0x7f8bfc0039c0 2024-05-27T00:28:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c19e00640 1 -- 172.21.15.98:0/722852356 reap_dead start 2024-05-27T00:28:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c18a00640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bfc0043c0 0x7f8bfc004890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c19400640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bfc0039c0 0x7f8bfc003da0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8c08001910 tx=0x7f8c0801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b703029971de91 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c08001510 con 0x7f8bfc0039c0 2024-05-27T00:28:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c080b0070 con 0x7f8bfc0039c0 2024-05-27T00:28:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c13e00640 1 -- 172.21.15.98:0/722852356 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c080ab8b0 con 0x7f8bfc0039c0 2024-05-27T00:28:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c1be00640 1 -- 172.21.15.98:0/722852356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bfc0039c0 msgr2=0x7f8bfc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c1be00640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bfc0039c0 0x7f8bfc003da0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8c08001910 tx=0x7f8c0801c530 comp rx=0 tx=0).stop 2024-05-27T00:28:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c1be00640 1 -- 172.21.15.98:0/722852356 shutdown_connections 2024-05-27T00:28:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c1be00640 1 --2- 172.21.15.98:0/722852356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bfc0039c0 0x7f8bfc003da0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c1be00640 1 -- 172.21.15.98:0/722852356 >> 172.21.15.98:0/722852356 conn(0x7f8c14072f90 msgr2=0x7f8c140733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c1be00640 1 -- 172.21.15.98:0/722852356 shutdown_connections 2024-05-27T00:28:40.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.078+0000 7f8c1be00640 1 -- 172.21.15.98:0/722852356 wait complete. 2024-05-27T00:28:40.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c1be00640 1 Processor -- start 2024-05-27T00:28:40.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c1be00640 1 -- start start 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c1be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bfc0039c0 0x7f8c1406c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c1be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c141f6980 0x7f8c141f6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c1be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c141575f0 0x7f8c14157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c1be00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c140711b0 con 0x7f8c141575f0 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c1be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8c140cb570 con 0x7f8bfc0039c0 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c1be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8c14152e30 con 0x7f8c141f6980 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c19e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c141575f0 0x7f8c14157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c19e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c141575f0 0x7f8c14157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55044/0 (socket says 172.21.15.98:55044) 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c19e00640 1 -- 172.21.15.98:0/3862819799 learned_addr learned my addr 172.21.15.98:0/3862819799 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c19400640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bfc0039c0 0x7f8c1406c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:40.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c18a00640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c141f6980 0x7f8c141f6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c19e00640 1 -- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c141f6980 msgr2=0x7f8c141f6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c19e00640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c141f6980 0x7f8c141f6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c19e00640 1 -- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bfc0039c0 msgr2=0x7f8c1406c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c19e00640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bfc0039c0 0x7f8c1406c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c19e00640 1 -- 172.21.15.98:0/3862819799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1406d490 con 0x7f8c141575f0 2024-05-27T00:28:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c19e00640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c141575f0 0x7f8c14157aa0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8c140c86a0 tx=0x7f8c0c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c11600640 1 -- 172.21.15.98:0/3862819799 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c011e10 con 0x7f8c141575f0 2024-05-27T00:28:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c11600640 1 -- 172.21.15.98:0/3862819799 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c0c01b070 con 0x7f8c141575f0 2024-05-27T00:28:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c11600640 1 -- 172.21.15.98:0/3862819799 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0039a0 con 0x7f8c141575f0 2024-05-27T00:28:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.082+0000 7f8c1be00640 1 -- 172.21.15.98:0/3862819799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c1415ba10 con 0x7f8c141575f0 2024-05-27T00:28:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.086+0000 7f8c1be00640 1 -- 172.21.15.98:0/3862819799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c1415b1b0 con 0x7f8c141575f0 2024-05-27T00:28:40.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.086+0000 7f8c1be00640 1 -- 172.21.15.98:0/3862819799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c141f7320 con 0x7f8c141575f0 2024-05-27T00:28:40.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.086+0000 7f8c11600640 1 -- 172.21.15.98:0/3862819799 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8c0c003b40 con 0x7f8c141575f0 2024-05-27T00:28:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.086+0000 7f8c11600640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8bf40678f0 0x7f8bf4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.086+0000 7f8c11600640 1 -- 172.21.15.98:0/3862819799 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7f8c0c0867e0 con 0x7f8c141575f0 2024-05-27T00:28:40.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.086+0000 7f8c19400640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8bf40678f0 0x7f8bf4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:40.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.090+0000 7f8c19400640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8bf40678f0 0x7f8bf4069db0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8c080af000 tx=0x7f8c08001330 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:40.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.090+0000 7f8c11600640 1 -- 172.21.15.98:0/3862819799 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8c0c025020 con 0x7f8c141575f0 2024-05-27T00:28:40.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.282+0000 7f8c1be00640 1 -- 172.21.15.98:0/3862819799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c141f66e0 con 0x7f8c141575f0 2024-05-27T00:28:40.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c11600640 1 -- 172.21.15.98:0/3862819799 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6839 (secure 0 0 0) 0x7f8c0c0568b0 con 0x7f8c141575f0 2024-05-27T00:28:40.285 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:40.285 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:40.285 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 57 2024-05-27T00:28:40.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 -- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8bf40678f0 msgr2=0x7f8bf4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:40.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8bf40678f0 0x7f8bf4069db0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8c080af000 tx=0x7f8c08001330 comp rx=0 tx=0).stop 2024-05-27T00:28:40.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 -- 172.21.15.98:0/3862819799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c141575f0 msgr2=0x7f8c14157aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:40.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c141575f0 0x7f8c14157aa0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8c140c86a0 tx=0x7f8c0c0115a0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 -- 172.21.15.98:0/3862819799 shutdown_connections 2024-05-27T00:28:40.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c141575f0 0x7f8c14157aa0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c141f6980 0x7f8c141f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8bf40678f0 0x7f8bf4069db0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 --2- 172.21.15.98:0/3862819799 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bfc0039c0 0x7f8c1406c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:40.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 -- 172.21.15.98:0/3862819799 >> 172.21.15.98:0/3862819799 conn(0x7f8c14072f90 msgr2=0x7f8c14073370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:40.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 -- 172.21.15.98:0/3862819799 shutdown_connections 2024-05-27T00:28:40.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:40.286+0000 7f8c1be00640 1 -- 172.21.15.98:0/3862819799 wait complete. 2024-05-27T00:28:40.306 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:28:42.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.442+0000 7fb4c7200640 1 Processor -- start 2024-05-27T00:28:42.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.442+0000 7fb4c7200640 1 -- start start 2024-05-27T00:28:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4c7200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb4c81526c0 0x7fb4c815b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4c7200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4c8152aa0 con 0x7fb4c80b3e30 2024-05-27T00:28:42.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4c7200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4c815c0e0 con 0x7fb4c8159880 2024-05-27T00:28:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4c7200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb4c815bec0 con 0x7fb4c81526c0 2024-05-27T00:28:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4bfe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb4c80b3e30 0x7fb4c8158da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51230/0 (socket says 172.21.15.98:51230) 2024-05-27T00:28:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4bfe00640 1 -- 172.21.15.98:0/215483418 learned_addr learned my addr 172.21.15.98:0/215483418 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4c5e00640 1 --2- 172.21.15.98:0/215483418 >> v2:172.21.15.98:3300/0 conn(0x7fb4c81526c0 0x7fb4c815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4c5400640 1 -- 172.21.15.98:0/215483418 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2945984140 0 0) 0x7fb4b0001120 con 0x7fb4c80b3e30 2024-05-27T00:28:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4c5400640 1 -- 172.21.15.98:0/215483418 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4ac003530 con 0x7fb4c80b3e30 2024-05-27T00:28:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4c5e00640 1 -- 172.21.15.98:0/215483418 >> v1:172.21.15.204:6789/0 conn(0x7fb4c8159880 legacy=0x7fb4c80cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4c5e00640 1 -- 172.21.15.98:0/215483418 >> v1:172.21.15.98:6789/0 conn(0x7fb4c80b3e30 legacy=0x7fb4c8158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:42.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.446+0000 7fb4c5e00640 1 -- 172.21.15.98:0/215483418 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c815c3c0 con 0x7fb4c81526c0 2024-05-27T00:28:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c5e00640 1 --2- 172.21.15.98:0/215483418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c81526c0 0x7fb4c815b980 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80097c0 tx=0x7fb4b801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f49d4151311147 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:42.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c5400640 1 -- 172.21.15.98:0/215483418 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b80b5070 con 0x7fb4c81526c0 2024-05-27T00:28:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c5400640 1 -- 172.21.15.98:0/215483418 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4b80a8990 con 0x7fb4c81526c0 2024-05-27T00:28:42.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c5400640 1 -- 172.21.15.98:0/215483418 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b80a8d60 con 0x7fb4c81526c0 2024-05-27T00:28:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 -- 172.21.15.98:0/215483418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c81526c0 msgr2=0x7fb4c815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 --2- 172.21.15.98:0/215483418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c81526c0 0x7fb4c815b980 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80097c0 tx=0x7fb4b801ec90 comp rx=0 tx=0).stop 2024-05-27T00:28:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 -- 172.21.15.98:0/215483418 shutdown_connections 2024-05-27T00:28:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 --2- 172.21.15.98:0/215483418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c81526c0 0x7fb4c815b980 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 -- 172.21.15.98:0/215483418 >> 172.21.15.98:0/215483418 conn(0x7fb4c8072f90 msgr2=0x7fb4c80733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 -- 172.21.15.98:0/215483418 shutdown_connections 2024-05-27T00:28:42.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 -- 172.21.15.98:0/215483418 wait complete. 2024-05-27T00:28:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 Processor -- start 2024-05-27T00:28:42.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 -- start start 2024-05-27T00:28:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4c80b3e30 0x7fb4c81f0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:42.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4c81526c0 0x7fb4c814b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8159880 0x7fb4c814a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81585a0 con 0x7fb4c8159880 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c8158910 con 0x7fb4c80b3e30 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.450+0000 7fb4c7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c80ce5c0 con 0x7fb4c81526c0 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8159880 0x7fb4c814a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c6800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8159880 0x7fb4c814a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55062/0 (socket says 172.21.15.98:55062) 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c6800640 1 -- 172.21.15.98:0/2044286522 learned_addr learned my addr 172.21.15.98:0/2044286522 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4bfe00640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4c81526c0 0x7fb4c814b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c5e00640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4c80b3e30 0x7fb4c81f0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c6800640 1 -- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4c81526c0 msgr2=0x7fb4c814b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c6800640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4c81526c0 0x7fb4c814b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c6800640 1 -- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4c80b3e30 msgr2=0x7fb4c81f0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c6800640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4c80b3e30 0x7fb4c81f0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c6800640 1 -- 172.21.15.98:0/2044286522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c81f3970 con 0x7fb4c8159880 2024-05-27T00:28:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c6800640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8159880 0x7fb4c814a020 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000c770 tx=0x7fb4c000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4be000640 1 -- 172.21.15.98:0/2044286522 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0003b80 con 0x7fb4c8159880 2024-05-27T00:28:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4be000640 1 -- 172.21.15.98:0/2044286522 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4c0003d20 con 0x7fb4c8159880 2024-05-27T00:28:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c7200640 1 -- 172.21.15.98:0/2044286522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c81f3c30 con 0x7fb4c8159880 2024-05-27T00:28:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4be000640 1 -- 172.21.15.98:0/2044286522 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c000aa90 con 0x7fb4c8159880 2024-05-27T00:28:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c7200640 1 -- 172.21.15.98:0/2044286522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c81f0510 con 0x7fb4c8159880 2024-05-27T00:28:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4c7200640 1 -- 172.21.15.98:0/2044286522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4c8075c40 con 0x7fb4c8159880 2024-05-27T00:28:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.454+0000 7fb4be000640 1 -- 172.21.15.98:0/2044286522 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb4c000ac30 con 0x7fb4c8159880 2024-05-27T00:28:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.458+0000 7fb4be000640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4a40678c0 0x7fb4a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.458+0000 7fb4be000640 1 -- 172.21.15.98:0/2044286522 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7fb4c0085e00 con 0x7fb4c8159880 2024-05-27T00:28:42.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.458+0000 7fb4c5e00640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4a40678c0 0x7fb4a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.458+0000 7fb4c5e00640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4a40678c0 0x7fb4a4069d80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80b2000 tx=0x7fb4b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:42.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.462+0000 7fb4be000640 1 -- 172.21.15.98:0/2044286522 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb4c0055f60 con 0x7fb4c8159880 2024-05-27T00:28:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.662+0000 7fb4c7200640 1 -- 172.21.15.98:0/2044286522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4c80b3a00 con 0x7fb4c8159880 2024-05-27T00:28:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.662+0000 7fb4be000640 1 -- 172.21.15.98:0/2044286522 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6839 (secure 0 0 0) 0x7fb4c005a260 con 0x7fb4c8159880 2024-05-27T00:28:42.664 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:42.664 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 57 2024-05-27T00:28:42.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 -- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4a40678c0 msgr2=0x7fb4a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:42.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4a40678c0 0x7fb4a4069d80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80b2000 tx=0x7fb4b8008040 comp rx=0 tx=0).stop 2024-05-27T00:28:42.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 -- 172.21.15.98:0/2044286522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8159880 msgr2=0x7fb4c814a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:42.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8159880 0x7fb4c814a020 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000c770 tx=0x7fb4c000cc30 comp rx=0 tx=0).stop 2024-05-27T00:28:42.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 -- 172.21.15.98:0/2044286522 shutdown_connections 2024-05-27T00:28:42.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c8159880 0x7fb4c814a020 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:42.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4c81526c0 0x7fb4c814b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:42.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb4a40678c0 0x7fb4a4069d80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 --2- 172.21.15.98:0/2044286522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4c80b3e30 0x7fb4c81f0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 -- 172.21.15.98:0/2044286522 >> 172.21.15.98:0/2044286522 conn(0x7fb4c8072f90 msgr2=0x7fb4c81515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 -- 172.21.15.98:0/2044286522 shutdown_connections 2024-05-27T00:28:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:42.666+0000 7fb4c7200640 1 -- 172.21.15.98:0/2044286522 wait complete. 2024-05-27T00:28:42.685 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:28:44.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:44.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2ff200640 1 Processor -- start 2024-05-27T00:28:44.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2ff200640 1 -- start start 2024-05-27T00:28:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2ff200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2f8152850 0x7fc2f80c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2ff200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fc2f80c9f30 0x7fc2f8070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2ff200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2f8070ba0 0x7fc2f815bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2ff200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f80c58f0 con 0x7fc2f8070ba0 2024-05-27T00:28:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2ff200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f815bab0 con 0x7fc2f80c9f30 2024-05-27T00:28:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2ff200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f80ca440 con 0x7fc2f8152850 2024-05-27T00:28:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2fe800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2f8070ba0 0x7fc2f815bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2fe800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2f8070ba0 0x7fc2f815bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40792/0 (socket says 172.21.15.98:40792) 2024-05-27T00:28:44.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2fe800640 1 -- 172.21.15.98:0/882063863 learned_addr learned my addr 172.21.15.98:0/882063863 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2fde00640 1 --2- 172.21.15.98:0/882063863 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2f8152850 0x7fc2f80c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2fd400640 1 --2- 172.21.15.98:0/882063863 >> v2:172.21.15.204:3300/0 conn(0x7fc2f80c9f30 0x7fc2f8070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2fe800640 1 -- 172.21.15.98:0/882063863 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2f8152850 msgr2=0x7fc2f80c53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2fe800640 1 --2- 172.21.15.98:0/882063863 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2f8152850 0x7fc2f80c53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2fe800640 1 -- 172.21.15.98:0/882063863 >> v2:172.21.15.204:3300/0 conn(0x7fc2f80c9f30 msgr2=0x7fc2f8070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2fe800640 1 --2- 172.21.15.98:0/882063863 >> v2:172.21.15.204:3300/0 conn(0x7fc2f80c9f30 0x7fc2f8070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:44.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.826+0000 7fc2fe800640 1 -- 172.21.15.98:0/882063863 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f8152c30 con 0x7fc2f8070ba0 2024-05-27T00:28:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.830+0000 7fc2fe800640 1 --2- 172.21.15.98:0/882063863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 0x7fc2f815bf40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc2f80cde10 tx=0x7fc2f401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609545fc4df7da28 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.830+0000 7fc2fca00640 1 -- 172.21.15.98:0/882063863 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f40b4070 con 0x7fc2f8070ba0 2024-05-27T00:28:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.830+0000 7fc2fca00640 1 -- 172.21.15.98:0/882063863 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f40af400 con 0x7fc2f8070ba0 2024-05-27T00:28:44.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.830+0000 7fc2fca00640 1 -- 172.21.15.98:0/882063863 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f40af7d0 con 0x7fc2f8070ba0 2024-05-27T00:28:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.830+0000 7fc2ff200640 1 -- 172.21.15.98:0/882063863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 msgr2=0x7fc2f815bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.830+0000 7fc2ff200640 1 --2- 172.21.15.98:0/882063863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 0x7fc2f815bf40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc2f80cde10 tx=0x7fc2f401f1e0 comp rx=0 tx=0).stop 2024-05-27T00:28:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 -- 172.21.15.98:0/882063863 shutdown_connections 2024-05-27T00:28:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 --2- 172.21.15.98:0/882063863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 0x7fc2f815bf40 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:44.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 --2- 172.21.15.98:0/882063863 >> v2:172.21.15.204:3300/0 conn(0x7fc2f80c9f30 0x7fc2f8070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 --2- 172.21.15.98:0/882063863 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2f8152850 0x7fc2f80c53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 -- 172.21.15.98:0/882063863 >> 172.21.15.98:0/882063863 conn(0x7fc2f80cdeb0 msgr2=0x7fc2f80ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 -- 172.21.15.98:0/882063863 shutdown_connections 2024-05-27T00:28:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 -- 172.21.15.98:0/882063863 wait complete. 2024-05-27T00:28:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 Processor -- start 2024-05-27T00:28:44.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 -- start start 2024-05-27T00:28:44.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 0x7fc2f81f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2f80c9f30 0x7fc2f81e4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2f8152850 0x7fc2f81ee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f806fb50 con 0x7fc2f8070ba0 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f806f870 con 0x7fc2f8152850 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f806fe30 con 0x7fc2f80c9f30 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fde00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 0x7fc2f81f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fde00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 0x7fc2f81f2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40798/0 (socket says 172.21.15.98:40798) 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fde00640 1 -- 172.21.15.98:0/3819255543 learned_addr learned my addr 172.21.15.98:0/3819255543 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fd400640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2f80c9f30 0x7fc2f81e4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fe800640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2f8152850 0x7fc2f81ee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fde00640 1 -- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2f80c9f30 msgr2=0x7fc2f81e4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fde00640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2f80c9f30 0x7fc2f81e4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fde00640 1 -- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2f8152850 msgr2=0x7fc2f81ee3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fde00640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2f8152850 0x7fc2f81ee3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fde00640 1 -- 172.21.15.98:0/3819255543 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f80cbd50 con 0x7fc2f8070ba0 2024-05-27T00:28:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2fde00640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 0x7fc2f81f2060 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00e870 tx=0x7fc2ec00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2f2000640 1 -- 172.21.15.98:0/3819255543 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec017070 con 0x7fc2f8070ba0 2024-05-27T00:28:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2f2000640 1 -- 172.21.15.98:0/3819255543 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ec00c940 con 0x7fc2f8070ba0 2024-05-27T00:28:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2f2000640 1 -- 172.21.15.98:0/3819255543 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00ccf0 con 0x7fc2f8070ba0 2024-05-27T00:28:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.834+0000 7fc2ff200640 1 -- 172.21.15.98:0/3819255543 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f81f25a0 con 0x7fc2f8070ba0 2024-05-27T00:28:44.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.838+0000 7fc2ff200640 1 -- 172.21.15.98:0/3819255543 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f80cbfd0 con 0x7fc2f8070ba0 2024-05-27T00:28:44.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.838+0000 7fc2ff200640 1 -- 172.21.15.98:0/3819255543 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f81ee1a0 con 0x7fc2f8070ba0 2024-05-27T00:28:44.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.838+0000 7fc2f2000640 1 -- 172.21.15.98:0/3819255543 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc2ec01a070 con 0x7fc2f8070ba0 2024-05-27T00:28:44.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.838+0000 7fc2f2000640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2dc0678c0 0x7fc2dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:44.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.838+0000 7fc2f2000640 1 -- 172.21.15.98:0/3819255543 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7fc2ec085f90 con 0x7fc2f8070ba0 2024-05-27T00:28:44.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.838+0000 7fc2fd400640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2dc0678c0 0x7fc2dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:44.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.842+0000 7fc2fd400640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2dc0678c0 0x7fc2dc069d80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc2f814afd0 tx=0x7fc2e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:44.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:44.842+0000 7fc2f2000640 1 -- 172.21.15.98:0/3819255543 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc2f80cbd50 con 0x7fc2f8070ba0 2024-05-27T00:28:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 -- 172.21.15.98:0/3819255543 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2f8075c40 con 0x7fc2f8070ba0 2024-05-27T00:28:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2f2000640 1 -- 172.21.15.98:0/3819255543 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6839 (secure 0 0 0) 0x7fc2ec0566b0 con 0x7fc2f8070ba0 2024-05-27T00:28:45.042 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:45.042 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 57 2024-05-27T00:28:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 -- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2dc0678c0 msgr2=0x7fc2dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2dc0678c0 0x7fc2dc069d80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc2f814afd0 tx=0x7fc2e8008040 comp rx=0 tx=0).stop 2024-05-27T00:28:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 -- 172.21.15.98:0/3819255543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 msgr2=0x7fc2f81f2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:45.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 0x7fc2f81f2060 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00e870 tx=0x7fc2ec00ec70 comp rx=0 tx=0).stop 2024-05-27T00:28:45.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 -- 172.21.15.98:0/3819255543 shutdown_connections 2024-05-27T00:28:45.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2dc0678c0 0x7fc2dc069d80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:45.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2f8152850 0x7fc2f81ee3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:45.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2f80c9f30 0x7fc2f81e4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:45.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 --2- 172.21.15.98:0/3819255543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2f8070ba0 0x7fc2f81f2060 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:45.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 -- 172.21.15.98:0/3819255543 >> 172.21.15.98:0/3819255543 conn(0x7fc2f80cdeb0 msgr2=0x7fc2f80ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:45.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 -- 172.21.15.98:0/3819255543 shutdown_connections 2024-05-27T00:28:45.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:45.042+0000 7fc2ff200640 1 -- 172.21.15.98:0/3819255543 wait complete. 2024-05-27T00:28:45.062 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:28:47.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:47.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.198+0000 7fdb46c00640 1 Processor -- start 2024-05-27T00:28:47.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.198+0000 7fdb46c00640 1 -- start start 2024-05-27T00:28:47.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb40151f00 0x7fdb40152390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb4015bb10 con 0x7fdb400c6a60 2024-05-27T00:28:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46c00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb4015bdf0 con 0x7fdb400c6e40 2024-05-27T00:28:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdb4015c0d0 con 0x7fdb40151f00 2024-05-27T00:28:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb45800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdb400c6a60 0x7fdb40073340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44090/0 (socket says 172.21.15.98:44090) 2024-05-27T00:28:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb45800640 1 -- 172.21.15.98:0/3147267293 learned_addr learned my addr 172.21.15.98:0/3147267293 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46200640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb40151f00 0x7fdb40152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2699927398 0 0) 0x7fdb4015bb10 con 0x7fdb400c6a60 2024-05-27T00:28:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb28003530 con 0x7fdb400c6a60 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 562898108 0 0) 0x7fdb28003530 con 0x7fdb400c6a60 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb40151f00 msgr2=0x7fdb40152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb40151f00 0x7fdb40152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 >> v1:172.21.15.204:6789/0 conn(0x7fdb400c6e40 legacy=0x7fdb40158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4015c3b0 con 0x7fdb400c6a60 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fdb34001680 con 0x7fdb400c6a60 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 >> v1:172.21.15.98:6789/0 conn(0x7fdb400c6a60 legacy=0x7fdb40073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb40151f00 0x7fdb280039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb28003f70 0x7fdb28004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 0x7fdb280051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4015c0d0 con 0x7fdb28004d50 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2800bab0 con 0x7fdb28003f70 2024-05-27T00:28:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdb2800bcb0 con 0x7fdb40151f00 2024-05-27T00:28:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46200640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 0x7fdb280051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb45800640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb40151f00 0x7fdb280039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb44e00640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb28003f70 0x7fdb28004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46200640 1 -- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb40151f00 msgr2=0x7fdb280039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46200640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb40151f00 0x7fdb280039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46200640 1 -- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb28003f70 msgr2=0x7fdb28004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46200640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb28003f70 0x7fdb28004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb46200640 1 -- 172.21.15.98:0/3147267293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb28003530 con 0x7fdb28004d50 2024-05-27T00:28:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.202+0000 7fdb44e00640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb28003f70 0x7fdb28004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:28:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb46200640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 0x7fdb280051d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009fc0 tx=0x7fdb3801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a260153a1aa58483 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb380b2070 con 0x7fdb28004d50 2024-05-27T00:28:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb380a5c90 con 0x7fdb28004d50 2024-05-27T00:28:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb3fe00640 1 -- 172.21.15.98:0/3147267293 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb38009c10 con 0x7fdb28004d50 2024-05-27T00:28:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb46c00640 1 -- 172.21.15.98:0/3147267293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 msgr2=0x7fdb280051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb46c00640 1 --2- 172.21.15.98:0/3147267293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 0x7fdb280051d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009fc0 tx=0x7fdb3801cc80 comp rx=0 tx=0).stop 2024-05-27T00:28:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb46200640 1 -- 172.21.15.98:0/3147267293 reap_dead start 2024-05-27T00:28:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb46c00640 1 -- 172.21.15.98:0/3147267293 shutdown_connections 2024-05-27T00:28:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb46c00640 1 -- 172.21.15.98:0/3147267293 >> 172.21.15.98:0/3147267293 conn(0x7fdb400cbe00 msgr2=0x7fdb400cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb46c00640 1 -- 172.21.15.98:0/3147267293 shutdown_connections 2024-05-27T00:28:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.206+0000 7fdb46c00640 1 -- 172.21.15.98:0/3147267293 wait complete. 2024-05-27T00:28:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 Processor -- start 2024-05-27T00:28:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 -- start start 2024-05-27T00:28:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 0x7fdb401569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb401f3d90 0x7fdb4014b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb401509b0 0x7fdb4014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb40152510 con 0x7fdb28004d50 2024-05-27T00:28:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdb40152390 con 0x7fdb401509b0 2024-05-27T00:28:47.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdb40152690 con 0x7fdb401f3d90 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb45800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 0x7fdb401569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb45800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 0x7fdb401569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40826/0 (socket says 172.21.15.98:40826) 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb45800640 1 -- 172.21.15.98:0/209975774 learned_addr learned my addr 172.21.15.98:0/209975774 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb44e00640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb401f3d90 0x7fdb4014b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46200640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb401509b0 0x7fdb4014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb45800640 1 -- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb401f3d90 msgr2=0x7fdb4014b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb45800640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb401f3d90 0x7fdb4014b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb45800640 1 -- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb401509b0 msgr2=0x7fdb4014f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb45800640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb401509b0 0x7fdb4014f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb45800640 1 -- 172.21.15.98:0/209975774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb400cc970 con 0x7fdb28004d50 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb44e00640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb401f3d90 0x7fdb4014b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46200640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb401509b0 0x7fdb4014f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:47.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb45800640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 0x7fdb401569b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdb34001ec0 tx=0x7fdb3400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb3d600640 1 -- 172.21.15.98:0/209975774 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3400c8a0 con 0x7fdb28004d50 2024-05-27T00:28:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb3d600640 1 -- 172.21.15.98:0/209975774 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb34018070 con 0x7fdb28004d50 2024-05-27T00:28:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb3d600640 1 -- 172.21.15.98:0/209975774 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3400cca0 con 0x7fdb28004d50 2024-05-27T00:28:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 -- 172.21.15.98:0/209975774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4006edf0 con 0x7fdb28004d50 2024-05-27T00:28:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 -- 172.21.15.98:0/209975774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb401eff70 con 0x7fdb28004d50 2024-05-27T00:28:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.210+0000 7fdb46c00640 1 -- 172.21.15.98:0/209975774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb40075c40 con 0x7fdb28004d50 2024-05-27T00:28:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.214+0000 7fdb3d600640 1 -- 172.21.15.98:0/209975774 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdb3401d460 con 0x7fdb28004d50 2024-05-27T00:28:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.214+0000 7fdb3d600640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb200678f0 0x7fdb20069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.214+0000 7fdb3d600640 1 -- 172.21.15.98:0/209975774 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7fdb34085fe0 con 0x7fdb28004d50 2024-05-27T00:28:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.214+0000 7fdb44e00640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb200678f0 0x7fdb20069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.214+0000 7fdb44e00640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb200678f0 0x7fdb20069db0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdb400c2cf0 tx=0x7fdb2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:47.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.218+0000 7fdb3d600640 1 -- 172.21.15.98:0/209975774 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fdb34056690 con 0x7fdb28004d50 2024-05-27T00:28:47.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 -- 172.21.15.98:0/209975774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb40071ea0 con 0x7fdb28004d50 2024-05-27T00:28:47.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb3d600640 1 -- 172.21.15.98:0/209975774 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6839 (secure 0 0 0) 0x7fdb34055f90 con 0x7fdb28004d50 2024-05-27T00:28:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:47.410 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 57 2024-05-27T00:28:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 -- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb200678f0 msgr2=0x7fdb20069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb200678f0 0x7fdb20069db0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdb400c2cf0 tx=0x7fdb2c009040 comp rx=0 tx=0).stop 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 -- 172.21.15.98:0/209975774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 msgr2=0x7fdb401569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 0x7fdb401569b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdb34001ec0 tx=0x7fdb3400eca0 comp rx=0 tx=0).stop 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 -- 172.21.15.98:0/209975774 shutdown_connections 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdb200678f0 0x7fdb20069db0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb401509b0 0x7fdb4014f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdb401f3d90 0x7fdb4014b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 --2- 172.21.15.98:0/209975774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb28004d50 0x7fdb401569b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 -- 172.21.15.98:0/209975774 >> 172.21.15.98:0/209975774 conn(0x7fdb400cbe00 msgr2=0x7fdb40158b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 -- 172.21.15.98:0/209975774 shutdown_connections 2024-05-27T00:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:47.410+0000 7fdb46c00640 1 -- 172.21.15.98:0/209975774 wait complete. 2024-05-27T00:28:47.431 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:28:49.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:49.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.574+0000 7eff99600640 1 Processor -- start 2024-05-27T00:28:49.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.574+0000 7eff99600640 1 -- start start 2024-05-27T00:28:49.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff99600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff940cbe10 0x7eff941592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:49.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff99600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7eff940cc1f0 0x7eff940c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:49.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff99600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff940c6dd0 con 0x7eff94073570 2024-05-27T00:28:49.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff99600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7eff94159810 con 0x7eff940cc1f0 2024-05-27T00:28:49.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff99600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7eff940c70b0 con 0x7eff940cbe10 2024-05-27T00:28:49.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff8a000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7eff940cc1f0 0x7eff940c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:49.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff93400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7eff94073570 0x7eff941559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44098/0 (socket says 172.21.15.98:44098) 2024-05-27T00:28:49.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff93400640 1 -- 172.21.15.98:0/1047118812 learned_addr learned my addr 172.21.15.98:0/1047118812 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:49.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff92a00640 1 --2- 172.21.15.98:0/1047118812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff940cbe10 0x7eff941592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:49.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff8a000640 1 -- 172.21.15.98:0/1047118812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff940cbe10 msgr2=0x7eff941592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:49.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff8a000640 1 --2- 172.21.15.98:0/1047118812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff940cbe10 0x7eff941592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:49.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff8a000640 1 -- 172.21.15.98:0/1047118812 >> v1:172.21.15.98:6789/0 conn(0x7eff94073570 legacy=0x7eff941559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:49.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff8a000640 1 -- 172.21.15.98:0/1047118812 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff940c7390 con 0x7eff940cc1f0 2024-05-27T00:28:49.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.578+0000 7eff92000640 1 -- 172.21.15.98:0/1047118812 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 585420173 0 0) 0x7eff940c6dd0 con 0x7eff94073570 2024-05-27T00:28:49.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff8a000640 1 --2- 172.21.15.98:0/1047118812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cc1f0 0x7eff940c1850 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0097c0 tx=0x7eff7c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5179063e7b2f23bc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:49.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff92000640 1 -- 172.21.15.98:0/1047118812 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0b5070 con 0x7eff940cc1f0 2024-05-27T00:28:49.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff92000640 1 -- 172.21.15.98:0/1047118812 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff7c0a8990 con 0x7eff940cc1f0 2024-05-27T00:28:49.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff92000640 1 -- 172.21.15.98:0/1047118812 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0a8d60 con 0x7eff940cc1f0 2024-05-27T00:28:49.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff99600640 1 -- 172.21.15.98:0/1047118812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cc1f0 msgr2=0x7eff940c1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:49.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff99600640 1 --2- 172.21.15.98:0/1047118812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cc1f0 0x7eff940c1850 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0097c0 tx=0x7eff7c01ec90 comp rx=0 tx=0).stop 2024-05-27T00:28:49.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff99600640 1 -- 172.21.15.98:0/1047118812 shutdown_connections 2024-05-27T00:28:49.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff99600640 1 --2- 172.21.15.98:0/1047118812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cc1f0 0x7eff940c1850 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:49.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff99600640 1 --2- 172.21.15.98:0/1047118812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff940cbe10 0x7eff941592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:49.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff99600640 1 -- 172.21.15.98:0/1047118812 >> 172.21.15.98:0/1047118812 conn(0x7eff94067530 msgr2=0x7eff9406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:49.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff99600640 1 -- 172.21.15.98:0/1047118812 shutdown_connections 2024-05-27T00:28:49.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff99600640 1 -- 172.21.15.98:0/1047118812 wait complete. 2024-05-27T00:28:49.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff99600640 1 Processor -- start 2024-05-27T00:28:49.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.582+0000 7eff99600640 1 -- start start 2024-05-27T00:28:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff99600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff94073570 0x7eff940c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff99600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cbe10 0x7eff94071740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff99600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff940cc1f0 0x7eff94071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff99600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff940c79d0 con 0x7eff940cbe10 2024-05-27T00:28:49.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff99600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff940c1330 con 0x7eff94073570 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff99600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7eff940c6990 con 0x7eff940cc1f0 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8a000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cbe10 0x7eff94071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8a000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cbe10 0x7eff94071740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40848/0 (socket says 172.21.15.98:40848) 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8a000640 1 -- 172.21.15.98:0/872491620 learned_addr learned my addr 172.21.15.98:0/872491620 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff92a00640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff94073570 0x7eff940c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff93400640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff940cc1f0 0x7eff94071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8a000640 1 -- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff940cc1f0 msgr2=0x7eff94071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8a000640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff940cc1f0 0x7eff94071c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8a000640 1 -- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff94073570 msgr2=0x7eff940c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8a000640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff94073570 0x7eff940c49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8a000640 1 -- 172.21.15.98:0/872491620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9414ed10 con 0x7eff940cbe10 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8a000640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cbe10 0x7eff94071740 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7eff7c01d340 tx=0x7eff7c01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8b400640 1 -- 172.21.15.98:0/872491620 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0c1070 con 0x7eff940cbe10 2024-05-27T00:28:49.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8b400640 1 -- 172.21.15.98:0/872491620 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff7c0b0780 con 0x7eff940cbe10 2024-05-27T00:28:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8b400640 1 -- 172.21.15.98:0/872491620 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0b5040 con 0x7eff940cbe10 2024-05-27T00:28:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff99600640 1 -- 172.21.15.98:0/872491620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff941f11e0 con 0x7eff940cbe10 2024-05-27T00:28:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff99600640 1 -- 172.21.15.98:0/872491620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff94071220 con 0x7eff940cbe10 2024-05-27T00:28:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff99600640 1 -- 172.21.15.98:0/872491620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff94075cb0 con 0x7eff940cbe10 2024-05-27T00:28:49.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.586+0000 7eff8b400640 1 -- 172.21.15.98:0/872491620 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7eff7c0c4070 con 0x7eff940cbe10 2024-05-27T00:28:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.590+0000 7eff8b400640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7eff700678c0 0x7eff70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.590+0000 7eff8b400640 1 -- 172.21.15.98:0/872491620 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6599+0+0 (secure 0 0 0) 0x7eff7c122e80 con 0x7eff940cbe10 2024-05-27T00:28:49.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.590+0000 7eff92a00640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7eff700678c0 0x7eff70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:49.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.590+0000 7eff92a00640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7eff700678c0 0x7eff70069d80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7eff8400b9d0 tx=0x7eff84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:49.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.594+0000 7eff8b400640 1 -- 172.21.15.98:0/872491620 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7eff7c0187f0 con 0x7eff940cbe10 2024-05-27T00:28:49.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.782+0000 7eff99600640 1 -- 172.21.15.98:0/872491620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff94158fb0 con 0x7eff940cbe10 2024-05-27T00:28:49.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.782+0000 7eff8b400640 1 -- 172.21.15.98:0/872491620 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6839 (secure 0 0 0) 0x7eff7c0f3630 con 0x7eff940cbe10 2024-05-27T00:28:49.785 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:49.785 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6078,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:17.424503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5943},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5943":{"gid":5943,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":14,"addr":"172.21.15.98:6834/2148124202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2148124202},{"type":"v1","addr":"172.21.15.98:6834","nonce":2148124202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5943,"qdb_cluster":[5943]},"id":1}]} 2024-05-27T00:28:49.787 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 57 2024-05-27T00:28:49.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 -- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7eff700678c0 msgr2=0x7eff70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:49.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7eff700678c0 0x7eff70069d80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7eff8400b9d0 tx=0x7eff84008040 comp rx=0 tx=0).stop 2024-05-27T00:28:49.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 -- 172.21.15.98:0/872491620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cbe10 msgr2=0x7eff94071740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:49.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cbe10 0x7eff94071740 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7eff7c01d340 tx=0x7eff7c01ba30 comp rx=0 tx=0).stop 2024-05-27T00:28:49.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 -- 172.21.15.98:0/872491620 shutdown_connections 2024-05-27T00:28:49.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff940cc1f0 0x7eff94071c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:49.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff940cbe10 0x7eff94071740 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:49.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7eff700678c0 0x7eff70069d80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:49.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 --2- 172.21.15.98:0/872491620 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff94073570 0x7eff940c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:49.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 -- 172.21.15.98:0/872491620 >> 172.21.15.98:0/872491620 conn(0x7eff94067530 msgr2=0x7eff9406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:49.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 -- 172.21.15.98:0/872491620 shutdown_connections 2024-05-27T00:28:49.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:49.786+0000 7eff99600640 1 -- 172.21.15.98:0/872491620 wait complete. 2024-05-27T00:28:49.805 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:28:51.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:51.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.942+0000 7fdc12e00640 1 Processor -- start 2024-05-27T00:28:51.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.942+0000 7fdc12e00640 1 -- start start 2024-05-27T00:28:51.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc12e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fdc0c152520 0x7fdc0c0c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:51.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc12e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc0c0c70e0 0x7fdc0c06b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:51.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc12e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc0c06cc70 con 0x7fdc0c073760 2024-05-27T00:28:51.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc12e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fdc0c072fa0 con 0x7fdc0c152520 2024-05-27T00:28:51.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc12e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdc0c152900 con 0x7fdc0c0c70e0 2024-05-27T00:28:51.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc03e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdc0c073760 0x7fdc0c071e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44106/0 (socket says 172.21.15.98:44106) 2024-05-27T00:28:51.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc03e00640 1 -- 172.21.15.98:0/3766672977 learned_addr learned my addr 172.21.15.98:0/3766672977 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:51.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0be00640 1 --2- 172.21.15.98:0/3766672977 >> v2:172.21.15.204:3300/0 conn(0x7fdc0c152520 0x7fdc0c0c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:51.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc10e00640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc0c0c70e0 0x7fdc0c06b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:51.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1162917235 0 0) 0x7fdc0c06cc70 con 0x7fdc0c073760 2024-05-27T00:28:51.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbf0003530 con 0x7fdc0c073760 2024-05-27T00:28:51.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1655391835 0 0) 0x7fdbf0003530 con 0x7fdc0c073760 2024-05-27T00:28:51.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc0c0c70e0 msgr2=0x7fdc0c06b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:51.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc0c0c70e0 0x7fdc0c06b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:51.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 >> v2:172.21.15.204:3300/0 conn(0x7fdc0c152520 msgr2=0x7fdc0c0c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:51.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 --2- 172.21.15.98:0/3766672977 >> v2:172.21.15.204:3300/0 conn(0x7fdc0c152520 0x7fdc0c0c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:51.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0c06c6b0 con 0x7fdc0c073760 2024-05-27T00:28:51.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fdbf4001680 con 0x7fdc0c073760 2024-05-27T00:28:51.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 >> v1:172.21.15.98:6789/0 conn(0x7fdc0c073760 legacy=0x7fdc0c071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:51.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc0c0c70e0 0x7fdbf00039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:51.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbf0003f40 0x7fdbf0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:51.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdbf0004d50 0x7fdbf00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:51.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0c072fa0 con 0x7fdbf0003f40 2024-05-27T00:28:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0c152900 con 0x7fdbf0004d50 2024-05-27T00:28:51.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdbf000b9d0 con 0x7fdc0c0c70e0 2024-05-27T00:28:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc03e00640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbf0003f40 0x7fdbf0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc0be00640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc0c0c70e0 0x7fdbf00039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:51.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc10e00640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdbf0004d50 0x7fdbf00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc03e00640 1 -- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc0c0c70e0 msgr2=0x7fdbf00039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc03e00640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc0c0c70e0 0x7fdbf00039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc03e00640 1 -- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdbf0004d50 msgr2=0x7fdbf00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc03e00640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdbf0004d50 0x7fdbf00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.946+0000 7fdc03e00640 1 -- 172.21.15.98:0/3766672977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf0003530 con 0x7fdbf0003f40 2024-05-27T00:28:51.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc03e00640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbf0003f40 0x7fdbf0004810 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400c240 tx=0x7fdbf401f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9310cf889d11395a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4007d40 con 0x7fdbf0003f40 2024-05-27T00:28:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbf4005b00 con 0x7fdbf0003f40 2024-05-27T00:28:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc0b400640 1 -- 172.21.15.98:0/3766672977 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4007070 con 0x7fdbf0003f40 2024-05-27T00:28:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc12e00640 1 -- 172.21.15.98:0/3766672977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbf0003f40 msgr2=0x7fdbf0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:51.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc12e00640 1 --2- 172.21.15.98:0/3766672977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbf0003f40 0x7fdbf0004810 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400c240 tx=0x7fdbf401f3b0 comp rx=0 tx=0).stop 2024-05-27T00:28:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc10e00640 1 -- 172.21.15.98:0/3766672977 reap_dead start 2024-05-27T00:28:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc12e00640 1 -- 172.21.15.98:0/3766672977 shutdown_connections 2024-05-27T00:28:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc12e00640 1 -- 172.21.15.98:0/3766672977 >> 172.21.15.98:0/3766672977 conn(0x7fdc0c0c9b70 msgr2=0x7fdc0c0c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc12e00640 1 -- 172.21.15.98:0/3766672977 shutdown_connections 2024-05-27T00:28:51.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc12e00640 1 -- 172.21.15.98:0/3766672977 wait complete. 2024-05-27T00:28:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.950+0000 7fdc12e00640 1 Processor -- start 2024-05-27T00:28:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc12e00640 1 -- start start 2024-05-27T00:28:51.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc12e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdbf0003f40 0x7fdc0c1f14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc12e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc0c150de0 0x7fdc0c14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc12e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc0c145440 0x7fdc0c144550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc12e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0c0c78a0 con 0x7fdc0c145440 2024-05-27T00:28:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc12e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0c073240 con 0x7fdc0c150de0 2024-05-27T00:28:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc12e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdc0c0c7a20 con 0x7fdbf0003f40 2024-05-27T00:28:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc10e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc0c145440 0x7fdc0c144550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:51.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc10e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc0c145440 0x7fdc0c144550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40862/0 (socket says 172.21.15.98:40862) 2024-05-27T00:28:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc10e00640 1 -- 172.21.15.98:0/2887436522 learned_addr learned my addr 172.21.15.98:0/2887436522 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc0be00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdbf0003f40 0x7fdc0c1f14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc03e00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc0c150de0 0x7fdc0c14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc10e00640 1 -- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdbf0003f40 msgr2=0x7fdc0c1f14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc10e00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdbf0003f40 0x7fdc0c1f14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc10e00640 1 -- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc0c150de0 msgr2=0x7fdc0c14fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:51.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc10e00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc0c150de0 0x7fdc0c14fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc10e00640 1 -- 172.21.15.98:0/2887436522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0c1458c0 con 0x7fdc0c145440 2024-05-27T00:28:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc10e00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc0c145440 0x7fdc0c144550 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c06f370 tx=0x7fdc0400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc08c00640 1 -- 172.21.15.98:0/2887436522 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400fce0 con 0x7fdc0c145440 2024-05-27T00:28:51.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc08c00640 1 -- 172.21.15.98:0/2887436522 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc0400fe80 con 0x7fdc0c145440 2024-05-27T00:28:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc08c00640 1 -- 172.21.15.98:0/2887436522 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04015670 con 0x7fdc0c145440 2024-05-27T00:28:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc12e00640 1 -- 172.21.15.98:0/2887436522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0c1503d0 con 0x7fdc0c145440 2024-05-27T00:28:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc12e00640 1 -- 172.21.15.98:0/2887436522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc0c150990 con 0x7fdc0c145440 2024-05-27T00:28:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.954+0000 7fdc12e00640 1 -- 172.21.15.98:0/2887436522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc0c075cb0 con 0x7fdc0c145440 2024-05-27T00:28:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.958+0000 7fdc08c00640 1 -- 172.21.15.98:0/2887436522 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdc0403c940 con 0x7fdc0c145440 2024-05-27T00:28:51.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.958+0000 7fdc08c00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbe80678f0 0x7fdbe8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.958+0000 7fdc08c00640 1 -- 172.21.15.98:0/2887436522 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7fdc04086820 con 0x7fdc0c145440 2024-05-27T00:28:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.958+0000 7fdc0be00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbe80678f0 0x7fdbe8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.958+0000 7fdc0be00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbe80678f0 0x7fdbe8069db0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009be0 tx=0x7fdbfc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:51.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:51.962+0000 7fdc08c00640 1 -- 172.21.15.98:0/2887436522 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fdc04024030 con 0x7fdc0c145440 2024-05-27T00:28:52.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.150+0000 7fdc12e00640 1 -- 172.21.15.98:0/2887436522 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc0c0c6cb0 con 0x7fdc0c145440 2024-05-27T00:28:52.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.150+0000 7fdc08c00640 1 -- 172.21.15.98:0/2887436522 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+6006 (secure 0 0 0) 0x7fdc0c1458c0 con 0x7fdc0c145440 2024-05-27T00:28:52.154 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:52.154 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:50.280035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:52.155 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 58 2024-05-27T00:28:52.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 -- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbe80678f0 msgr2=0x7fdbe8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:52.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbe80678f0 0x7fdbe8069db0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009be0 tx=0x7fdbfc009190 comp rx=0 tx=0).stop 2024-05-27T00:28:52.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 -- 172.21.15.98:0/2887436522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc0c145440 msgr2=0x7fdc0c144550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:52.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc0c145440 0x7fdc0c144550 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c06f370 tx=0x7fdc0400f1b0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 -- 172.21.15.98:0/2887436522 shutdown_connections 2024-05-27T00:28:52.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc0c145440 0x7fdc0c144550 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdbe80678f0 0x7fdbe8069db0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc0c150de0 0x7fdc0c14fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 --2- 172.21.15.98:0/2887436522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdbf0003f40 0x7fdc0c1f14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 -- 172.21.15.98:0/2887436522 >> 172.21.15.98:0/2887436522 conn(0x7fdc0c0c9b70 msgr2=0x7fdc0c157d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:52.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 -- 172.21.15.98:0/2887436522 shutdown_connections 2024-05-27T00:28:52.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.154+0000 7fdc12e00640 1 -- 172.21.15.98:0/2887436522 wait complete. 2024-05-27T00:28:52.182 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-05-27T00:28:52.182 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:28:52.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.318+0000 7f3cdbe00640 1 Processor -- start 2024-05-27T00:28:52.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdbe00640 1 -- start start 2024-05-27T00:28:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdbe00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3cdc15b110 0x7f3cdc166020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdbe00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cdc0cb190 con 0x7f3cdc154690 2024-05-27T00:28:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdbe00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cdc15dd20 con 0x7f3cdc0cc290 2024-05-27T00:28:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdbe00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f3cdc0caf40 con 0x7f3cdc15b110 2024-05-27T00:28:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdaa00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3cdc154690 0x7f3cdc0c4f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44118/0 (socket says 172.21.15.98:44118) 2024-05-27T00:28:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdaa00640 1 -- 172.21.15.98:0/2432737714 learned_addr learned my addr 172.21.15.98:0/2432737714 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:52.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdb400640 1 --2- 172.21.15.98:0/2432737714 >> v2:172.21.15.204:3300/0 conn(0x7f3cdc15b110 0x7f3cdc166020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:52.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3867184177 0 0) 0x7f3cdc0cb190 con 0x7f3cdc154690 2024-05-27T00:28:52.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc0003530 con 0x7f3cdc154690 2024-05-27T00:28:52.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3531910446 0 0) 0x7f3cc0003530 con 0x7f3cdc154690 2024-05-27T00:28:52.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 >> v2:172.21.15.204:3300/0 conn(0x7f3cdc15b110 msgr2=0x7f3cdc166020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:52.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 --2- 172.21.15.98:0/2432737714 >> v2:172.21.15.204:3300/0 conn(0x7f3cdc15b110 0x7f3cdc166020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 >> v1:172.21.15.204:6789/0 conn(0x7f3cdc0cc290 legacy=0x7f3cdc0ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cdc15e000 con 0x7f3cdc154690 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f3ccc001680 con 0x7f3cdc154690 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 >> v1:172.21.15.98:6789/0 conn(0x7f3cdc154690 legacy=0x7f3cdc0c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cc0003a10 0x7f3cc0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cc0004810 0x7f3cc0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cc00051d0 0x7f3cc000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3cdc0caf40 con 0x7f3cc0004810 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc000c0b0 con 0x7f3cc0003a10 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3cc000c2b0 con 0x7f3cc00051d0 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cda000640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cc0004810 0x7f3cc0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdaa00640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cc0003a10 0x7f3cc0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdb400640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cc00051d0 0x7f3cc000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:52.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cda000640 1 -- 172.21.15.98:0/2432737714 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cc00051d0 msgr2=0x7f3cc000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:52.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cda000640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cc00051d0 0x7f3cc000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cda000640 1 -- 172.21.15.98:0/2432737714 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cc0003a10 msgr2=0x7f3cc0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:52.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cda000640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cc0003a10 0x7f3cc0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cda000640 1 -- 172.21.15.98:0/2432737714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc0003530 con 0x7f3cc0004810 2024-05-27T00:28:52.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.322+0000 7f3cdb400640 1 -- 172.21.15.98:0/2432737714 reap_dead start 2024-05-27T00:28:52.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cda000640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cc0004810 0x7f3cc0004c90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8001780 tx=0x7f3cc801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f708e08af380b81 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:52.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc80013a0 con 0x7f3cc0004810 2024-05-27T00:28:52.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc80052f0 con 0x7f3cc0004810 2024-05-27T00:28:52.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cd9600640 1 -- 172.21.15.98:0/2432737714 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8003840 con 0x7f3cc0004810 2024-05-27T00:28:52.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cdbe00640 1 -- 172.21.15.98:0/2432737714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cc0004810 msgr2=0x7f3cc0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:52.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cdbe00640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cc0004810 0x7f3cc0004c90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8001780 tx=0x7f3cc801c400 comp rx=0 tx=0).stop 2024-05-27T00:28:52.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cdbe00640 1 -- 172.21.15.98:0/2432737714 shutdown_connections 2024-05-27T00:28:52.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cdbe00640 1 --2- 172.21.15.98:0/2432737714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cc0004810 0x7f3cc0004c90 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cdbe00640 1 -- 172.21.15.98:0/2432737714 >> 172.21.15.98:0/2432737714 conn(0x7f3cdc06c740 msgr2=0x7f3cdc06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:52.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cdbe00640 1 -- 172.21.15.98:0/2432737714 shutdown_connections 2024-05-27T00:28:52.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.326+0000 7f3cdbe00640 1 -- 172.21.15.98:0/2432737714 wait complete. 2024-05-27T00:28:52.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cdbe00640 1 Processor -- start 2024-05-27T00:28:52.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cdbe00640 1 -- start start 2024-05-27T00:28:52.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cdbe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cc0004810 0x7f3cdc1e6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:52.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cdbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cdc1f02a0 0x7f3cdc0cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:52.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cdbe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cdc0cf4e0 0x7f3cdc1f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:52.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cdbe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3cdc0cb7e0 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cdbe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3cdc15e5c0 con 0x7f3cdc0cf4e0 2024-05-27T00:28:52.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cdbe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3cdc15eae0 con 0x7f3cc0004810 2024-05-27T00:28:52.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cda000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cdc1f02a0 0x7f3cdc0cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:52.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cda000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cdc1f02a0 0x7f3cdc0cefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40874/0 (socket says 172.21.15.98:40874) 2024-05-27T00:28:52.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cda000640 1 -- 172.21.15.98:0/1865945893 learned_addr learned my addr 172.21.15.98:0/1865945893 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:52.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cdb400640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cdc0cf4e0 0x7f3cdc1f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:52.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cdaa00640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cc0004810 0x7f3cdc1e6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:52.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cda000640 1 -- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cc0004810 msgr2=0x7f3cdc1e6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:52.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cda000640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cc0004810 0x7f3cdc1e6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cda000640 1 -- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cdc0cf4e0 msgr2=0x7f3cdc1f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:52.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cda000640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cdc0cf4e0 0x7f3cdc1f3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cda000640 1 -- 172.21.15.98:0/1865945893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cdc153670 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.330+0000 7f3cda000640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cdc1f02a0 0x7f3cdc0cefa0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3cc801c230 tx=0x7f3cc0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:52.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.334+0000 7f3cd2a00640 1 -- 172.21.15.98:0/1865945893 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8007a40 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.334+0000 7f3cd2a00640 1 -- 172.21.15.98:0/1865945893 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc80ba070 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.334+0000 7f3cdbe00640 1 -- 172.21.15.98:0/1865945893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cdc153910 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.334+0000 7f3cd2a00640 1 -- 172.21.15.98:0/1865945893 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8007250 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.334+0000 7f3cdbe00640 1 -- 172.21.15.98:0/1865945893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cdc0cf980 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.334+0000 7f3cdbe00640 1 -- 172.21.15.98:0/1865945893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cdc075c40 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.334+0000 7f3cd2a00640 1 -- 172.21.15.98:0/1865945893 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3cc80052f0 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.334+0000 7f3cd2a00640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 0x7f3cbc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:52.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.334+0000 7f3cd2a00640 1 -- 172.21.15.98:0/1865945893 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f3cc81202d0 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.338+0000 7f3cdaa00640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 0x7f3cbc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:52.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.338+0000 7f3cdaa00640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 0x7f3cbc069d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc000c10 tx=0x7f3ccc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:52.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.338+0000 7f3cd2a00640 1 -- 172.21.15.98:0/1865945893 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f3cc8016110 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.530+0000 7f3cdbe00640 1 -- 172.21.15.98:0/1865945893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cdc153ec0 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cd2a00640 1 -- 172.21.15.98:0/1865945893 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+6006 (secure 0 0 0) 0x7f3cc80f0910 con 0x7f3cdc1f02a0 2024-05-27T00:28:52.533 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:52.533 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:50.280035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:52.533 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 58 2024-05-27T00:28:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 -- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 msgr2=0x7f3cbc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 0x7f3cbc069d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc000c10 tx=0x7f3ccc009040 comp rx=0 tx=0).stop 2024-05-27T00:28:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 -- 172.21.15.98:0/1865945893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cdc1f02a0 msgr2=0x7f3cdc0cefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cdc1f02a0 0x7f3cdc0cefa0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3cc801c230 tx=0x7f3cc0005840 comp rx=0 tx=0).stop 2024-05-27T00:28:52.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 -- 172.21.15.98:0/1865945893 shutdown_connections 2024-05-27T00:28:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3cbc0678c0 0x7f3cbc069d80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cdc0cf4e0 0x7f3cdc1f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3cdc1f02a0 0x7f3cdc0cefa0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 --2- 172.21.15.98:0/1865945893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3cc0004810 0x7f3cdc1e6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 -- 172.21.15.98:0/1865945893 >> 172.21.15.98:0/1865945893 conn(0x7f3cdc06c740 msgr2=0x7f3cdc158bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 -- 172.21.15.98:0/1865945893 shutdown_connections 2024-05-27T00:28:52.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:52.534+0000 7f3cdbe00640 1 -- 172.21.15.98:0/1865945893 wait complete. 2024-05-27T00:28:52.556 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 58, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 58, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6078": { "addr": "172.21.15.98:6837/1802902856", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 1802902856, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 1802902856, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6078, "incarnation": 58, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:28:50.280035+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6078 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:28:54.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:54.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.698+0000 7f4582200640 1 Processor -- start 2024-05-27T00:28:54.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.698+0000 7f4582200640 1 -- start start 2024-05-27T00:28:54.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f4582200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f457c0cbe10 0x7f457c1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f4582200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f457c073570 0x7f457c1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f4582200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f457c0c1f60 con 0x7f457c0cc1f0 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f4582200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f457c159810 con 0x7f457c0cbe10 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f4582200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f457c0c6dd0 con 0x7f457c073570 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f457b400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f457c0cbe10 0x7f457c1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f457aa00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f457c0cc1f0 0x7f457c0c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55238/0 (socket says 172.21.15.98:55238) 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f457aa00640 1 -- 172.21.15.98:0/554885819 learned_addr learned my addr 172.21.15.98:0/554885819 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f457be00640 1 --2- 172.21.15.98:0/554885819 >> v2:172.21.15.204:3300/0 conn(0x7f457c073570 0x7f457c1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f457b400640 1 -- 172.21.15.98:0/554885819 >> v2:172.21.15.204:3300/0 conn(0x7f457c073570 msgr2=0x7f457c1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f457b400640 1 --2- 172.21.15.98:0/554885819 >> v2:172.21.15.204:3300/0 conn(0x7f457c073570 0x7f457c1559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:54.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f457b400640 1 -- 172.21.15.98:0/554885819 >> v1:172.21.15.98:6789/0 conn(0x7f457c0cc1f0 legacy=0x7f457c0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:54.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f457a000640 1 -- 172.21.15.98:0/554885819 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 896206475 0 0) 0x7f457c0c1f60 con 0x7f457c0cc1f0 2024-05-27T00:28:54.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.702+0000 7f457b400640 1 -- 172.21.15.98:0/554885819 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457c0c70b0 con 0x7f457c0cbe10 2024-05-27T00:28:54.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f457b400640 1 --2- 172.21.15.98:0/554885819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c0cbe10 0x7f457c1592d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f457c06e230 tx=0x7f456c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fef1eb057fe88 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:54.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f457a000640 1 -- 172.21.15.98:0/554885819 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c0b5070 con 0x7f457c0cbe10 2024-05-27T00:28:54.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f457a000640 1 -- 172.21.15.98:0/554885819 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f456c0a8970 con 0x7f457c0cbe10 2024-05-27T00:28:54.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f457a000640 1 -- 172.21.15.98:0/554885819 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c0a8d40 con 0x7f457c0cbe10 2024-05-27T00:28:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 -- 172.21.15.98:0/554885819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c0cbe10 msgr2=0x7f457c1592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 --2- 172.21.15.98:0/554885819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c0cbe10 0x7f457c1592d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f457c06e230 tx=0x7f456c01ec70 comp rx=0 tx=0).stop 2024-05-27T00:28:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 -- 172.21.15.98:0/554885819 shutdown_connections 2024-05-27T00:28:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 --2- 172.21.15.98:0/554885819 >> v2:172.21.15.204:3300/0 conn(0x7f457c073570 0x7f457c1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 --2- 172.21.15.98:0/554885819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c0cbe10 0x7f457c1592d0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 -- 172.21.15.98:0/554885819 >> 172.21.15.98:0/554885819 conn(0x7f457c067530 msgr2=0x7f457c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:54.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 -- 172.21.15.98:0/554885819 shutdown_connections 2024-05-27T00:28:54.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 -- 172.21.15.98:0/554885819 wait complete. 2024-05-27T00:28:54.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 Processor -- start 2024-05-27T00:28:54.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 -- start start 2024-05-27T00:28:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c073570 0x7f457c1effd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f457c0cbe10 0x7f457c154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f457c0cc1f0 0x7f457c0c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f457c0c1330 con 0x7f457c073570 2024-05-27T00:28:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f457c0c1050 con 0x7f457c0cc1f0 2024-05-27T00:28:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.706+0000 7f4582200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f457c0c79d0 con 0x7f457c0cbe10 2024-05-27T00:28:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c073570 0x7f457c1effd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:54.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c073570 0x7f457c1effd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54026/0 (socket says 172.21.15.98:54026) 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457b400640 1 -- 172.21.15.98:0/3297974928 learned_addr learned my addr 172.21.15.98:0/3297974928 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457be00640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f457c0cc1f0 0x7f457c0c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457aa00640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f457c0cbe10 0x7f457c154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457b400640 1 -- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f457c0cbe10 msgr2=0x7f457c154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457b400640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f457c0cbe10 0x7f457c154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457b400640 1 -- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f457c0cc1f0 msgr2=0x7f457c0c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457b400640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f457c0cc1f0 0x7f457c0c7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457b400640 1 -- 172.21.15.98:0/3297974928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457c1554b0 con 0x7f457c073570 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457aa00640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f457c0cbe10 0x7f457c154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f457b400640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c073570 0x7f457c1effd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f456c0b3040 tx=0x7f456c01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f4573400640 1 -- 172.21.15.98:0/3297974928 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c0bf070 con 0x7f457c073570 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f4573400640 1 -- 172.21.15.98:0/3297974928 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f456c0b0740 con 0x7f457c073570 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f4573400640 1 -- 172.21.15.98:0/3297974928 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c0b5040 con 0x7f457c073570 2024-05-27T00:28:54.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f4582200640 1 -- 172.21.15.98:0/3297974928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f457c15c900 con 0x7f457c073570 2024-05-27T00:28:54.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f4582200640 1 -- 172.21.15.98:0/3297974928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f457c0b1520 con 0x7f457c073570 2024-05-27T00:28:54.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f4582200640 1 -- 172.21.15.98:0/3297974928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f457c075cb0 con 0x7f457c073570 2024-05-27T00:28:54.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f4573400640 1 -- 172.21.15.98:0/3297974928 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f456c0b0b30 con 0x7f457c073570 2024-05-27T00:28:54.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.710+0000 7f4573400640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f455c0678f0 0x7f455c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:54.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.714+0000 7f4573400640 1 -- 172.21.15.98:0/3297974928 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f456c122a90 con 0x7f457c073570 2024-05-27T00:28:54.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.714+0000 7f457aa00640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f455c0678f0 0x7f455c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:54.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.714+0000 7f457aa00640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f455c0678f0 0x7f455c069db0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f457c0b14b0 tx=0x7f4568009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:54.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.714+0000 7f4573400640 1 -- 172.21.15.98:0/3297974928 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f457c1554b0 con 0x7f457c073570 2024-05-27T00:28:54.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.902+0000 7f4582200640 1 -- 172.21.15.98:0/3297974928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f457c0c3710 con 0x7f457c073570 2024-05-27T00:28:54.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.902+0000 7f4573400640 1 -- 172.21.15.98:0/3297974928 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+6006 (secure 0 0 0) 0x7f456c0187d0 con 0x7f457c073570 2024-05-27T00:28:54.904 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:54.904 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:50.280035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:54.905 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 58 2024-05-27T00:28:54.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 -- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f455c0678f0 msgr2=0x7f455c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:54.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f455c0678f0 0x7f455c069db0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f457c0b14b0 tx=0x7f4568009040 comp rx=0 tx=0).stop 2024-05-27T00:28:54.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 -- 172.21.15.98:0/3297974928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c073570 msgr2=0x7f457c1effd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:54.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c073570 0x7f457c1effd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f456c0b3040 tx=0x7f456c01ba10 comp rx=0 tx=0).stop 2024-05-27T00:28:54.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 -- 172.21.15.98:0/3297974928 shutdown_connections 2024-05-27T00:28:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f455c0678f0 0x7f455c069db0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f457c0cc1f0 0x7f457c0c7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f457c0cbe10 0x7f457c154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 --2- 172.21.15.98:0/3297974928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f457c073570 0x7f457c1effd0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 -- 172.21.15.98:0/3297974928 >> 172.21.15.98:0/3297974928 conn(0x7f457c067530 msgr2=0x7f457c06bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 -- 172.21.15.98:0/3297974928 shutdown_connections 2024-05-27T00:28:54.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:54.906+0000 7f4582200640 1 -- 172.21.15.98:0/3297974928 wait complete. 2024-05-27T00:28:54.927 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:28:56.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.098+0000 7fce70800640 1 Processor -- start 2024-05-27T00:28:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.098+0000 7fce70800640 1 -- start start 2024-05-27T00:28:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.098+0000 7fce70800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fce680cbe10 0x7fce681592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce70800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce680cc1f0 0x7fce680c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce70800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce680c6dd0 con 0x7fce68073570 2024-05-27T00:28:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce70800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fce68159810 con 0x7fce680cbe10 2024-05-27T00:28:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce70800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fce680c70b0 con 0x7fce680cc1f0 2024-05-27T00:28:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6e800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fce68073570 0x7fce681559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55246/0 (socket says 172.21.15.98:55246) 2024-05-27T00:28:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6e800640 1 -- 172.21.15.98:0/1425135434 learned_addr learned my addr 172.21.15.98:0/1425135434 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6de00640 1 --2- 172.21.15.98:0/1425135434 >> v2:172.21.15.204:3300/0 conn(0x7fce680cbe10 0x7fce681592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6d400640 1 --2- 172.21.15.98:0/1425135434 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce680cc1f0 0x7fce680c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6ca00640 1 -- 172.21.15.98:0/1425135434 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4252496938 0 0) 0x7fce680c6dd0 con 0x7fce68073570 2024-05-27T00:28:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6ca00640 1 -- 172.21.15.98:0/1425135434 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce50003530 con 0x7fce68073570 2024-05-27T00:28:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6de00640 1 -- 172.21.15.98:0/1425135434 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce680cc1f0 msgr2=0x7fce680c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6de00640 1 --2- 172.21.15.98:0/1425135434 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce680cc1f0 0x7fce680c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6de00640 1 -- 172.21.15.98:0/1425135434 >> v1:172.21.15.98:6789/0 conn(0x7fce68073570 legacy=0x7fce681559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6ca00640 1 -- 172.21.15.98:0/1425135434 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3072695720 0 0) 0x7fce50003530 con 0x7fce68073570 2024-05-27T00:28:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6de00640 1 -- 172.21.15.98:0/1425135434 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce680c7390 con 0x7fce680cbe10 2024-05-27T00:28:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.102+0000 7fce6d400640 1 --2- 172.21.15.98:0/1425135434 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce680cc1f0 0x7fce680c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:28:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce6de00640 1 --2- 172.21.15.98:0/1425135434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce680cbe10 0x7fce681592d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0097c0 tx=0x7fce5c01ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bece21cd22e4bed0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce6ca00640 1 -- 172.21.15.98:0/1425135434 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c0b5070 con 0x7fce680cbe10 2024-05-27T00:28:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce6ca00640 1 -- 172.21.15.98:0/1425135434 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce5c0a85f0 con 0x7fce680cbe10 2024-05-27T00:28:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce6ca00640 1 -- 172.21.15.98:0/1425135434 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64001a00 con 0x7fce680cbe10 2024-05-27T00:28:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce70800640 1 -- 172.21.15.98:0/1425135434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce680cbe10 msgr2=0x7fce681592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce70800640 1 --2- 172.21.15.98:0/1425135434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce680cbe10 0x7fce681592d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0097c0 tx=0x7fce5c01ebb0 comp rx=0 tx=0).stop 2024-05-27T00:28:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce70800640 1 -- 172.21.15.98:0/1425135434 shutdown_connections 2024-05-27T00:28:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce70800640 1 --2- 172.21.15.98:0/1425135434 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce680cc1f0 0x7fce680c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce70800640 1 --2- 172.21.15.98:0/1425135434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce680cbe10 0x7fce681592d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce70800640 1 -- 172.21.15.98:0/1425135434 >> 172.21.15.98:0/1425135434 conn(0x7fce68067530 msgr2=0x7fce6806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce70800640 1 -- 172.21.15.98:0/1425135434 shutdown_connections 2024-05-27T00:28:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce70800640 1 -- 172.21.15.98:0/1425135434 wait complete. 2024-05-27T00:28:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce70800640 1 Processor -- start 2024-05-27T00:28:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.106+0000 7fce70800640 1 -- start start 2024-05-27T00:28:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce70800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce68073570 0x7fce68151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce70800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce680cbe10 0x7fce680ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce70800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce680cc1f0 0x7fce680b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce70800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce680c79d0 con 0x7fce680cbe10 2024-05-27T00:28:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce70800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fce680c1330 con 0x7fce680cc1f0 2024-05-27T00:28:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce70800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fce680c6990 con 0x7fce68073570 2024-05-27T00:28:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce680cbe10 0x7fce680ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce680cbe10 0x7fce680ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54042/0 (socket says 172.21.15.98:54042) 2024-05-27T00:28:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6d400640 1 -- 172.21.15.98:0/536727489 learned_addr learned my addr 172.21.15.98:0/536727489 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6de00640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce68073570 0x7fce68151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6e800640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce680cc1f0 0x7fce680b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6d400640 1 -- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce68073570 msgr2=0x7fce68151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6d400640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce68073570 0x7fce68151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6d400640 1 -- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce680cc1f0 msgr2=0x7fce680b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6d400640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce680cc1f0 0x7fce680b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6d400640 1 -- 172.21.15.98:0/536727489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce681f3ab0 con 0x7fce680cbe10 2024-05-27T00:28:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6e800640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce680cc1f0 0x7fce680b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:28:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce6d400640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce680cbe10 0x7fce680ae8c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fce5800e9b0 tx=0x7fce5800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce62000640 1 -- 172.21.15.98:0/536727489 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58017070 con 0x7fce680cbe10 2024-05-27T00:28:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce62000640 1 -- 172.21.15.98:0/536727489 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce5800ca00 con 0x7fce680cbe10 2024-05-27T00:28:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce70800640 1 -- 172.21.15.98:0/536727489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce681f3d90 con 0x7fce680cbe10 2024-05-27T00:28:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce62000640 1 -- 172.21.15.98:0/536727489 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5800cdd0 con 0x7fce680cbe10 2024-05-27T00:28:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce70800640 1 -- 172.21.15.98:0/536727489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce681f4350 con 0x7fce680cbe10 2024-05-27T00:28:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.110+0000 7fce70800640 1 -- 172.21.15.98:0/536727489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce68075cb0 con 0x7fce680cbe10 2024-05-27T00:28:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.114+0000 7fce62000640 1 -- 172.21.15.98:0/536727489 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fce5801a070 con 0x7fce680cbe10 2024-05-27T00:28:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.114+0000 7fce62000640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 0x7fce4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.114+0000 7fce62000640 1 -- 172.21.15.98:0/536727489 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7fce580862c0 con 0x7fce680cbe10 2024-05-27T00:28:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.114+0000 7fce6de00640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 0x7fce4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.114+0000 7fce6de00640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 0x7fce4c069d80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0b2000 tx=0x7fce5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:57.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.118+0000 7fce62000640 1 -- 172.21.15.98:0/536727489 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fce58056340 con 0x7fce680cbe10 2024-05-27T00:28:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.334+0000 7fce70800640 1 -- 172.21.15.98:0/536727489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce681f4550 con 0x7fce680cbe10 2024-05-27T00:28:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce62000640 1 -- 172.21.15.98:0/536727489 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+6006 (secure 0 0 0) 0x7fce5805a640 con 0x7fce680cbe10 2024-05-27T00:28:57.337 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:57.337 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:50.280035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 58 2024-05-27T00:28:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 -- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 msgr2=0x7fce4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 0x7fce4c069d80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0b2000 tx=0x7fce5c008040 comp rx=0 tx=0).stop 2024-05-27T00:28:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 -- 172.21.15.98:0/536727489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce680cbe10 msgr2=0x7fce680ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce680cbe10 0x7fce680ae8c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fce5800e9b0 tx=0x7fce5800ee70 comp rx=0 tx=0).stop 2024-05-27T00:28:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 -- 172.21.15.98:0/536727489 shutdown_connections 2024-05-27T00:28:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 0x7fce4c069d80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce680cc1f0 0x7fce680b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce680cbe10 0x7fce680ae8c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 --2- 172.21.15.98:0/536727489 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce68073570 0x7fce68151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 -- 172.21.15.98:0/536727489 >> 172.21.15.98:0/536727489 conn(0x7fce68067530 msgr2=0x7fce68154ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 -- 172.21.15.98:0/536727489 shutdown_connections 2024-05-27T00:28:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:57.338+0000 7fce70800640 1 -- 172.21.15.98:0/536727489 wait complete. 2024-05-27T00:28:57.361 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:28:59.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:28:59.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.510+0000 7fc292000640 1 Processor -- start 2024-05-27T00:28:59.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.510+0000 7fc292000640 1 -- start start 2024-05-27T00:28:59.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.510+0000 7fc292000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc28c0731f0 0x7fc28c0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:59.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc292000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc28c0c17d0 con 0x7fc28c0c79f0 2024-05-27T00:28:59.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc292000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc28c070e10 con 0x7fc28c0735d0 2024-05-27T00:28:59.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc292000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc28c071990 con 0x7fc28c0731f0 2024-05-27T00:28:59.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc28b400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc28c0731f0 0x7fc28c0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:59.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc28be00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc28c0c79f0 0x7fc28c157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55248/0 (socket says 172.21.15.98:55248) 2024-05-27T00:28:59.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc28be00640 1 -- 172.21.15.98:0/2504566444 learned_addr learned my addr 172.21.15.98:0/2504566444 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:28:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc28b400640 1 -- 172.21.15.98:0/2504566444 >> v1:172.21.15.204:6789/0 conn(0x7fc28c0735d0 legacy=0x7fc28c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:28:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc28b400640 1 -- 172.21.15.98:0/2504566444 >> v1:172.21.15.98:6789/0 conn(0x7fc28c0c79f0 legacy=0x7fc28c157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc28b400640 1 -- 172.21.15.98:0/2504566444 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28c0710f0 con 0x7fc28c0731f0 2024-05-27T00:28:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc28aa00640 1 -- 172.21.15.98:0/2504566444 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446320122 0 0) 0x7fc28c0c17d0 con 0x7fc28c0c79f0 2024-05-27T00:28:59.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.514+0000 7fc28b400640 1 --2- 172.21.15.98:0/2504566444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0731f0 0x7fc28c0c0eb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc28c071b10 tx=0x7fc27c01eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89995bc54073679 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc28aa00640 1 -- 172.21.15.98:0/2504566444 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c0b4070 con 0x7fc28c0731f0 2024-05-27T00:28:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc28aa00640 1 -- 172.21.15.98:0/2504566444 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc27c0a7ad0 con 0x7fc28c0731f0 2024-05-27T00:28:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc28aa00640 1 -- 172.21.15.98:0/2504566444 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c0af440 con 0x7fc28c0731f0 2024-05-27T00:28:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 -- 172.21.15.98:0/2504566444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0731f0 msgr2=0x7fc28c0c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 --2- 172.21.15.98:0/2504566444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0731f0 0x7fc28c0c0eb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc28c071b10 tx=0x7fc27c01eb00 comp rx=0 tx=0).stop 2024-05-27T00:28:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 -- 172.21.15.98:0/2504566444 shutdown_connections 2024-05-27T00:28:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 --2- 172.21.15.98:0/2504566444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0731f0 0x7fc28c0c0eb0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 -- 172.21.15.98:0/2504566444 >> 172.21.15.98:0/2504566444 conn(0x7fc28c067530 msgr2=0x7fc28c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 -- 172.21.15.98:0/2504566444 shutdown_connections 2024-05-27T00:28:59.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 -- 172.21.15.98:0/2504566444 wait complete. 2024-05-27T00:28:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 Processor -- start 2024-05-27T00:28:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 -- start start 2024-05-27T00:28:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc28c0731f0 0x7fc28c155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc28c0735d0 0x7fc28c13fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0c79f0 0x7fc28c140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc28c0c1570 con 0x7fc28c0c79f0 2024-05-27T00:28:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc28c0c13f0 con 0x7fc28c0731f0 2024-05-27T00:28:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.518+0000 7fc292000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc28c151b40 con 0x7fc28c0735d0 2024-05-27T00:28:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc28be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0c79f0 0x7fc28c140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc28be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0c79f0 0x7fc28c140020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54064/0 (socket says 172.21.15.98:54064) 2024-05-27T00:28:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc28be00640 1 -- 172.21.15.98:0/1263260240 learned_addr learned my addr 172.21.15.98:0/1263260240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:28:59.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc28b400640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc28c0731f0 0x7fc28c155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:59.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc282a00640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc28c0735d0 0x7fc28c13fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc28be00640 1 -- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc28c0735d0 msgr2=0x7fc28c13fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc28be00640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc28c0735d0 0x7fc28c13fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc28be00640 1 -- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc28c0731f0 msgr2=0x7fc28c155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc28be00640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc28c0731f0 0x7fc28c155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc28be00640 1 -- 172.21.15.98:0/1263260240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28c142780 con 0x7fc28c0c79f0 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc28be00640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0c79f0 0x7fc28c140020 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc28400c7d0 tx=0x7fc28400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc283e00640 1 -- 172.21.15.98:0/1263260240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284018070 con 0x7fc28c0c79f0 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc283e00640 1 -- 172.21.15.98:0/1263260240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc28400a640 con 0x7fc28c0c79f0 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc283e00640 1 -- 172.21.15.98:0/1263260240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28400aa70 con 0x7fc28c0c79f0 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc292000640 1 -- 172.21.15.98:0/1263260240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc28c1411b0 con 0x7fc28c0c79f0 2024-05-27T00:28:59.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc292000640 1 -- 172.21.15.98:0/1263260240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc28c152380 con 0x7fc28c0c79f0 2024-05-27T00:28:59.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.522+0000 7fc292000640 1 -- 172.21.15.98:0/1263260240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc28c075cb0 con 0x7fc28c0c79f0 2024-05-27T00:28:59.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.526+0000 7fc283e00640 1 -- 172.21.15.98:0/1263260240 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc28400ac10 con 0x7fc28c0c79f0 2024-05-27T00:28:59.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.526+0000 7fc283e00640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2680678f0 0x7fc268069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:28:59.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.526+0000 7fc283e00640 1 -- 172.21.15.98:0/1263260240 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7fc2840855a0 con 0x7fc28c0c79f0 2024-05-27T00:28:59.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.526+0000 7fc28b400640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2680678f0 0x7fc268069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:28:59.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.526+0000 7fc28b400640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2680678f0 0x7fc268069db0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0b1000 tx=0x7fc27c0a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:28:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.530+0000 7fc283e00640 1 -- 172.21.15.98:0/1263260240 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc284055590 con 0x7fc28c0c79f0 2024-05-27T00:28:59.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.734+0000 7fc292000640 1 -- 172.21.15.98:0/1263260240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc28c162c10 con 0x7fc28c0c79f0 2024-05-27T00:28:59.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.734+0000 7fc283e00640 1 -- 172.21.15.98:0/1263260240 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+6006 (secure 0 0 0) 0x7fc284059890 con 0x7fc28c0c79f0 2024-05-27T00:28:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:28:59.735 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:28:50.280035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:28:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 58 2024-05-27T00:28:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 -- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2680678f0 msgr2=0x7fc268069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2680678f0 0x7fc268069db0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0b1000 tx=0x7fc27c0a9070 comp rx=0 tx=0).stop 2024-05-27T00:28:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 -- 172.21.15.98:0/1263260240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0c79f0 msgr2=0x7fc28c140020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:28:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0c79f0 0x7fc28c140020 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc28400c7d0 tx=0x7fc28400cc90 comp rx=0 tx=0).stop 2024-05-27T00:28:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 -- 172.21.15.98:0/1263260240 shutdown_connections 2024-05-27T00:28:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc28c0c79f0 0x7fc28c140020 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc28c0735d0 0x7fc28c13fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2680678f0 0x7fc268069db0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 --2- 172.21.15.98:0/1263260240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc28c0731f0 0x7fc28c155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:28:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 -- 172.21.15.98:0/1263260240 >> 172.21.15.98:0/1263260240 conn(0x7fc28c067530 msgr2=0x7fc28c147710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:28:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 -- 172.21.15.98:0/1263260240 shutdown_connections 2024-05-27T00:28:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:28:59.738+0000 7fc292000640 1 -- 172.21.15.98:0/1263260240 wait complete. 2024-05-27T00:28:59.757 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:29:01.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:29:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4877200640 1 Processor -- start 2024-05-27T00:29:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4877200640 1 -- start start 2024-05-27T00:29:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4877200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48780cac80 0x7f48780c9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4877200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4878152ad0 0x7f4878162e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4877200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f487815bf90 con 0x7f487815bb00 2024-05-27T00:29:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4877200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f48780c84f0 con 0x7f4878152ad0 2024-05-27T00:29:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4877200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f48780c8990 con 0x7f48780cac80 2024-05-27T00:29:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4875400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f487815bb00 0x7f48781523c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55252/0 (socket says 172.21.15.98:55252) 2024-05-27T00:29:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4875400640 1 -- 172.21.15.98:0/1449305075 learned_addr learned my addr 172.21.15.98:0/1449305075 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:29:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4875e00640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48780cac80 0x7f48780c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4876800640 1 --2- 172.21.15.98:0/1449305075 >> v2:172.21.15.204:3300/0 conn(0x7f4878152ad0 0x7f4878162e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3581243022 0 0) 0x7f487815bf90 con 0x7f487815bb00 2024-05-27T00:29:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.906+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f485c003530 con 0x7f487815bb00 2024-05-27T00:29:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2473485070 0 0) 0x7f485c003530 con 0x7f487815bb00 2024-05-27T00:29:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48780cac80 msgr2=0x7f48780c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48780cac80 0x7f48780c9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 >> v2:172.21.15.204:3300/0 conn(0x7f4878152ad0 msgr2=0x7f4878162e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 --2- 172.21.15.98:0/1449305075 >> v2:172.21.15.204:3300/0 conn(0x7f4878152ad0 0x7f4878162e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487815c270 con 0x7f487815bb00 2024-05-27T00:29:01.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f4864001680 con 0x7f487815bb00 2024-05-27T00:29:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 >> v1:172.21.15.98:6789/0 conn(0x7f487815bb00 legacy=0x7f48781523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f485c0039c0 0x7f485c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 0x7f485c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:01.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48780cac80 0x7f485c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48780c8990 con 0x7f485c004810 2024-05-27T00:29:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f485c00b9d0 con 0x7f485c0039c0 2024-05-27T00:29:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f485c00bb50 con 0x7f48780cac80 2024-05-27T00:29:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4875400640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 0x7f485c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4875e00640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f485c0039c0 0x7f485c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4876800640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48780cac80 0x7f485c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4875400640 1 -- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48780cac80 msgr2=0x7f485c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:01.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4875400640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48780cac80 0x7f485c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:01.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4875400640 1 -- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f485c0039c0 msgr2=0x7f485c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:01.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4875400640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f485c0039c0 0x7f485c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:01.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4875400640 1 -- 172.21.15.98:0/1449305075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485c003530 con 0x7f485c004810 2024-05-27T00:29:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4875400640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 0x7f485c004c90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f486400c240 tx=0x7f486401f410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f099614589207785 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.910+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864007d40 con 0x7f485c004810 2024-05-27T00:29:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4864005b00 con 0x7f485c004810 2024-05-27T00:29:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4874a00640 1 -- 172.21.15.98:0/1449305075 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864007070 con 0x7f485c004810 2024-05-27T00:29:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 -- 172.21.15.98:0/1449305075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 msgr2=0x7f485c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 --2- 172.21.15.98:0/1449305075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 0x7f485c004c90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f486400c240 tx=0x7f486401f410 comp rx=0 tx=0).stop 2024-05-27T00:29:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4876800640 1 -- 172.21.15.98:0/1449305075 reap_dead start 2024-05-27T00:29:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 -- 172.21.15.98:0/1449305075 shutdown_connections 2024-05-27T00:29:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 -- 172.21.15.98:0/1449305075 >> 172.21.15.98:0/1449305075 conn(0x7f4878070b30 msgr2=0x7f4878070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 -- 172.21.15.98:0/1449305075 shutdown_connections 2024-05-27T00:29:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 -- 172.21.15.98:0/1449305075 wait complete. 2024-05-27T00:29:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 Processor -- start 2024-05-27T00:29:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 -- start start 2024-05-27T00:29:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 0x7f48781e18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48781e44b0 0x7f487814c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f487806e8f0 0x7f487814cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48780c8be0 con 0x7f485c004810 2024-05-27T00:29:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f487815c830 con 0x7f48781e44b0 2024-05-27T00:29:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4877200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f48780b40f0 con 0x7f487806e8f0 2024-05-27T00:29:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4875e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 0x7f48781e18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4875e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 0x7f48781e18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54080/0 (socket says 172.21.15.98:54080) 2024-05-27T00:29:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4875e00640 1 -- 172.21.15.98:0/2112432766 learned_addr learned my addr 172.21.15.98:0/2112432766 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:29:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4876800640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f487806e8f0 0x7f487814cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4875400640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48781e44b0 0x7f487814c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4875e00640 1 -- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f487806e8f0 msgr2=0x7f487814cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4875e00640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f487806e8f0 0x7f487814cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4875e00640 1 -- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48781e44b0 msgr2=0x7f487814c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:01.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4875e00640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48781e44b0 0x7f487814c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:01.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4875e00640 1 -- 172.21.15.98:0/2112432766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48781e1df0 con 0x7f485c004810 2024-05-27T00:29:01.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4876800640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f487806e8f0 0x7f487814cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:29:01.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.914+0000 7f4875400640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48781e44b0 0x7f487814c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:29:01.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f4875e00640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 0x7f48781e18b0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4868009f90 tx=0x7f486800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:01.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f486e000640 1 -- 172.21.15.98:0/2112432766 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486800fea0 con 0x7f485c004810 2024-05-27T00:29:01.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f486e000640 1 -- 172.21.15.98:0/2112432766 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4868015630 con 0x7f485c004810 2024-05-27T00:29:01.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f486e000640 1 -- 172.21.15.98:0/2112432766 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4868015a20 con 0x7f485c004810 2024-05-27T00:29:01.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f4877200640 1 -- 172.21.15.98:0/2112432766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487815b0d0 con 0x7f485c004810 2024-05-27T00:29:01.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f4877200640 1 -- 172.21.15.98:0/2112432766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f487815b390 con 0x7f485c004810 2024-05-27T00:29:01.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f4877200640 1 -- 172.21.15.98:0/2112432766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4878075c40 con 0x7f485c004810 2024-05-27T00:29:01.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f486e000640 1 -- 172.21.15.98:0/2112432766 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4868015bc0 con 0x7f485c004810 2024-05-27T00:29:01.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f486e000640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48580678f0 0x7f4858069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:01.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f486e000640 1 -- 172.21.15.98:0/2112432766 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f4868087140 con 0x7f485c004810 2024-05-27T00:29:01.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.918+0000 7f4875400640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48580678f0 0x7f4858069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:01.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.922+0000 7f4875400640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48580678f0 0x7f4858069db0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f48640ae000 tx=0x7f487815c160 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:01.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:01.922+0000 7f486e000640 1 -- 172.21.15.98:0/2112432766 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4868019030 con 0x7f485c004810 2024-05-27T00:29:02.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 -- 172.21.15.98:0/2112432766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48780c3e20 con 0x7f485c004810 2024-05-27T00:29:02.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f486e000640 1 -- 172.21.15.98:0/2112432766 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+6010 (secure 0 0 0) 0x7f4868010040 con 0x7f485c004810 2024-05-27T00:29:02.114 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:29:02.114 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:00.122454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:29:02.114 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 59 2024-05-27T00:29:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 -- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48580678f0 msgr2=0x7f4858069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48580678f0 0x7f4858069db0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f48640ae000 tx=0x7f487815c160 comp rx=0 tx=0).stop 2024-05-27T00:29:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 -- 172.21.15.98:0/2112432766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 msgr2=0x7f48781e18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 0x7f48781e18b0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4868009f90 tx=0x7f486800f580 comp rx=0 tx=0).stop 2024-05-27T00:29:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 -- 172.21.15.98:0/2112432766 shutdown_connections 2024-05-27T00:29:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f487806e8f0 0x7f487814cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48580678f0 0x7f4858069db0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48781e44b0 0x7f487814c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 --2- 172.21.15.98:0/2112432766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f485c004810 0x7f48781e18b0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:02.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 -- 172.21.15.98:0/2112432766 >> 172.21.15.98:0/2112432766 conn(0x7f4878070b30 msgr2=0x7f48780c2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:02.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 -- 172.21.15.98:0/2112432766 shutdown_connections 2024-05-27T00:29:02.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:02.114+0000 7f4877200640 1 -- 172.21.15.98:0/2112432766 wait complete. 2024-05-27T00:29:02.135 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:29:04.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:29:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d18a00640 1 Processor -- start 2024-05-27T00:29:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d18a00640 1 -- start start 2024-05-27T00:29:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d18a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d140731f0 0x7f9d140c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d18a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f9d140c79f0 0x7f9d14157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d18a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d140c17d0 con 0x7f9d140735d0 2024-05-27T00:29:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d18a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9d14071990 con 0x7f9d140731f0 2024-05-27T00:29:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d18a00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f9d14070e10 con 0x7f9d140c79f0 2024-05-27T00:29:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d11e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9d140731f0 0x7f9d140c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d11400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9d140735d0 0x7f9d14151430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53072/0 (socket says 172.21.15.98:53072) 2024-05-27T00:29:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d11400640 1 -- 172.21.15.98:0/3733125933 learned_addr learned my addr 172.21.15.98:0/3733125933 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:29:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d12800640 1 --2- 172.21.15.98:0/3733125933 >> v2:172.21.15.204:3300/0 conn(0x7f9d140c79f0 0x7f9d14157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d10a00640 1 -- 172.21.15.98:0/3733125933 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 663867991 0 0) 0x7f9d140c17d0 con 0x7f9d140735d0 2024-05-27T00:29:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d10a00640 1 -- 172.21.15.98:0/3733125933 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cf8003530 con 0x7f9d140735d0 2024-05-27T00:29:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d11e00640 1 -- 172.21.15.98:0/3733125933 >> v2:172.21.15.204:3300/0 conn(0x7f9d140c79f0 msgr2=0x7f9d14157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d11e00640 1 --2- 172.21.15.98:0/3733125933 >> v2:172.21.15.204:3300/0 conn(0x7f9d140c79f0 0x7f9d14157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d11e00640 1 -- 172.21.15.98:0/3733125933 >> v1:172.21.15.98:6789/0 conn(0x7f9d140735d0 legacy=0x7f9d14151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.282+0000 7f9d11e00640 1 -- 172.21.15.98:0/3733125933 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d140710f0 con 0x7f9d140731f0 2024-05-27T00:29:04.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d11e00640 1 --2- 172.21.15.98:0/3733125933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140731f0 0x7f9d140c0eb0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9d1406c530 tx=0x7f9d0401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192d156158fc820 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:04.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d10a00640 1 -- 172.21.15.98:0/3733125933 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d040ab650 con 0x7f9d140731f0 2024-05-27T00:29:04.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d10a00640 1 -- 172.21.15.98:0/3733125933 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d040b0070 con 0x7f9d140731f0 2024-05-27T00:29:04.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d10a00640 1 -- 172.21.15.98:0/3733125933 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d040aba10 con 0x7f9d140731f0 2024-05-27T00:29:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d18a00640 1 -- 172.21.15.98:0/3733125933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140731f0 msgr2=0x7f9d140c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d18a00640 1 --2- 172.21.15.98:0/3733125933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140731f0 0x7f9d140c0eb0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9d1406c530 tx=0x7f9d0401c450 comp rx=0 tx=0).stop 2024-05-27T00:29:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d18a00640 1 -- 172.21.15.98:0/3733125933 shutdown_connections 2024-05-27T00:29:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d18a00640 1 --2- 172.21.15.98:0/3733125933 >> v2:172.21.15.204:3300/0 conn(0x7f9d140c79f0 0x7f9d14157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d18a00640 1 --2- 172.21.15.98:0/3733125933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140731f0 0x7f9d140c0eb0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:04.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d18a00640 1 -- 172.21.15.98:0/3733125933 >> 172.21.15.98:0/3733125933 conn(0x7f9d14067530 msgr2=0x7f9d1406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:04.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d18a00640 1 -- 172.21.15.98:0/3733125933 shutdown_connections 2024-05-27T00:29:04.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.286+0000 7f9d18a00640 1 -- 172.21.15.98:0/3733125933 wait complete. 2024-05-27T00:29:04.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d18a00640 1 Processor -- start 2024-05-27T00:29:04.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d18a00640 1 -- start start 2024-05-27T00:29:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d18a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d140731f0 0x7f9d1413ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d18a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d140735d0 0x7f9d14140990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d18a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140c79f0 0x7f9d14147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d18a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d140c13f0 con 0x7f9d140c79f0 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d18a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d140c8400 con 0x7f9d140735d0 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d18a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9d140c1570 con 0x7f9d140731f0 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d12800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140c79f0 0x7f9d14147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d12800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140c79f0 0x7f9d14147370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40974/0 (socket says 172.21.15.98:40974) 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d12800640 1 -- 172.21.15.98:0/187777006 learned_addr learned my addr 172.21.15.98:0/187777006 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d11400640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d140735d0 0x7f9d14140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d11e00640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d140731f0 0x7f9d1413ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d12800640 1 -- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d140731f0 msgr2=0x7f9d1413ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d12800640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d140731f0 0x7f9d1413ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d12800640 1 -- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d140735d0 msgr2=0x7f9d14140990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d12800640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d140735d0 0x7f9d14140990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d12800640 1 -- 172.21.15.98:0/187777006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d14140ed0 con 0x7f9d140c79f0 2024-05-27T00:29:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d12800640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140c79f0 0x7f9d14147370 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00efc0 tx=0x7f9d0c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d0a000640 1 -- 172.21.15.98:0/187777006 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c018020 con 0x7f9d140c79f0 2024-05-27T00:29:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d0a000640 1 -- 172.21.15.98:0/187777006 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d0c014480 con 0x7f9d140c79f0 2024-05-27T00:29:04.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d0a000640 1 -- 172.21.15.98:0/187777006 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c014870 con 0x7f9d140c79f0 2024-05-27T00:29:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d18a00640 1 -- 172.21.15.98:0/187777006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d141640d0 con 0x7f9d140c79f0 2024-05-27T00:29:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.290+0000 7f9d18a00640 1 -- 172.21.15.98:0/187777006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d14164390 con 0x7f9d140c79f0 2024-05-27T00:29:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.294+0000 7f9d18a00640 1 -- 172.21.15.98:0/187777006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d14075cb0 con 0x7f9d140c79f0 2024-05-27T00:29:04.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.294+0000 7f9d0a000640 1 -- 172.21.15.98:0/187777006 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9d0c01b020 con 0x7f9d140c79f0 2024-05-27T00:29:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.294+0000 7f9d0a000640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9cf40678c0 0x7f9cf4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.294+0000 7f9d0a000640 1 -- 172.21.15.98:0/187777006 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f9d0c086bc0 con 0x7f9d140c79f0 2024-05-27T00:29:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.294+0000 7f9d11e00640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9cf40678c0 0x7f9cf4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.294+0000 7f9d11e00640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9cf40678c0 0x7f9cf4069d80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9d040af000 tx=0x7f9d040a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.298+0000 7f9d0a000640 1 -- 172.21.15.98:0/187777006 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9d0c022070 con 0x7f9d140c79f0 2024-05-27T00:29:04.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.486+0000 7f9d18a00640 1 -- 172.21.15.98:0/187777006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d14158510 con 0x7f9d140c79f0 2024-05-27T00:29:04.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d0a000640 1 -- 172.21.15.98:0/187777006 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+6010 (secure 0 0 0) 0x7f9d0c010040 con 0x7f9d140c79f0 2024-05-27T00:29:04.489 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:29:04.489 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:00.122454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:29:04.490 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 59 2024-05-27T00:29:04.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 -- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9cf40678c0 msgr2=0x7f9cf4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:04.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9cf40678c0 0x7f9cf4069d80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9d040af000 tx=0x7f9d040a7000 comp rx=0 tx=0).stop 2024-05-27T00:29:04.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 -- 172.21.15.98:0/187777006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140c79f0 msgr2=0x7f9d14147370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:04.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140c79f0 0x7f9d14147370 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00efc0 tx=0x7f9d0c00c490 comp rx=0 tx=0).stop 2024-05-27T00:29:04.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 -- 172.21.15.98:0/187777006 shutdown_connections 2024-05-27T00:29:04.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d140c79f0 0x7f9d14147370 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:04.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f9cf40678c0 0x7f9cf4069d80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:04.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d140735d0 0x7f9d14140990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:04.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 --2- 172.21.15.98:0/187777006 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d140731f0 0x7f9d1413ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:04.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 -- 172.21.15.98:0/187777006 >> 172.21.15.98:0/187777006 conn(0x7f9d14067530 msgr2=0x7f9d1406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:04.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 -- 172.21.15.98:0/187777006 shutdown_connections 2024-05-27T00:29:04.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:04.490+0000 7f9d18a00640 1 -- 172.21.15.98:0/187777006 wait complete. 2024-05-27T00:29:04.510 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 59, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 59, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6078": { "addr": "172.21.15.98:6837/1802902856", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 1802902856, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 1802902856, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6078, "incarnation": 58, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:29:00.122454+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6078 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:29:06.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:29:06.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3ee00640 1 Processor -- start 2024-05-27T00:29:06.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3ee00640 1 -- start start 2024-05-27T00:29:06.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3ee00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf38157d80 0x7fbf380c3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:06.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3ee00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbf380c5490 0x7fbf380c3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:06.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3ee00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf3806e7e0 con 0x7fbf38070110 2024-05-27T00:29:06.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3ee00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbf3815b3e0 con 0x7fbf380c5490 2024-05-27T00:29:06.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3ee00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbf3806ea80 con 0x7fbf38157d80 2024-05-27T00:29:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3e400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbf380c5490 0x7fbf380c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3d000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbf38070110 0x7fbf380c4850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53088/0 (socket says 172.21.15.98:53088) 2024-05-27T00:29:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3d000640 1 -- 172.21.15.98:0/3119147205 learned_addr learned my addr 172.21.15.98:0/3119147205 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:29:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3da00640 1 --2- 172.21.15.98:0/3119147205 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf38157d80 0x7fbf380c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3e400640 1 -- 172.21.15.98:0/3119147205 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf38157d80 msgr2=0x7fbf380c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3e400640 1 --2- 172.21.15.98:0/3119147205 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf38157d80 0x7fbf380c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3e400640 1 -- 172.21.15.98:0/3119147205 >> v1:172.21.15.98:6789/0 conn(0x7fbf38070110 legacy=0x7fbf380c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf3e400640 1 -- 172.21.15.98:0/3119147205 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf380c67f0 con 0x7fbf380c5490 2024-05-27T00:29:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.662+0000 7fbf37e00640 1 -- 172.21.15.98:0/3119147205 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1122812815 0 0) 0x7fbf3806e7e0 con 0x7fbf38070110 2024-05-27T00:29:06.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf3e400640 1 --2- 172.21.15.98:0/3119147205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf380c5490 0x7fbf380c3960 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009d90 tx=0x7fbf3001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cef9cbc5a8f275 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:06.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf37e00640 1 -- 172.21.15.98:0/3119147205 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf300b2070 con 0x7fbf380c5490 2024-05-27T00:29:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf37e00640 1 -- 172.21.15.98:0/3119147205 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf300a59c0 con 0x7fbf380c5490 2024-05-27T00:29:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf37e00640 1 -- 172.21.15.98:0/3119147205 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf300a5d90 con 0x7fbf380c5490 2024-05-27T00:29:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3119147205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf380c5490 msgr2=0x7fbf380c3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:06.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf3ee00640 1 --2- 172.21.15.98:0/3119147205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf380c5490 0x7fbf380c3960 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009d90 tx=0x7fbf3001c8e0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3119147205 shutdown_connections 2024-05-27T00:29:06.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf3ee00640 1 --2- 172.21.15.98:0/3119147205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf380c5490 0x7fbf380c3960 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf3ee00640 1 --2- 172.21.15.98:0/3119147205 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf38157d80 0x7fbf380c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3119147205 >> 172.21.15.98:0/3119147205 conn(0x7fbf38070a10 msgr2=0x7fbf38070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:06.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3119147205 shutdown_connections 2024-05-27T00:29:06.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3119147205 wait complete. 2024-05-27T00:29:06.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.666+0000 7fbf3ee00640 1 Processor -- start 2024-05-27T00:29:06.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3ee00640 1 -- start start 2024-05-27T00:29:06.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3ee00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38070110 0x7fbf3814b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:06.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3ee00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf380c5490 0x7fbf381f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:06.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3ee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf38157d80 0x7fbf3814d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:06.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3ee00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf38167a00 con 0x7fbf38070110 2024-05-27T00:29:06.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3ee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbf38167880 con 0x7fbf38157d80 2024-05-27T00:29:06.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3ee00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbf38167b80 con 0x7fbf380c5490 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3da00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38070110 0x7fbf3814b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3da00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38070110 0x7fbf3814b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40990/0 (socket says 172.21.15.98:40990) 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3da00640 1 -- 172.21.15.98:0/3309157186 learned_addr learned my addr 172.21.15.98:0/3309157186 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3e400640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf38157d80 0x7fbf3814d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3d000640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf380c5490 0x7fbf381f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3da00640 1 -- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf380c5490 msgr2=0x7fbf381f7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3da00640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf380c5490 0x7fbf381f7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3da00640 1 -- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf38157d80 msgr2=0x7fbf3814d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3da00640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf38157d80 0x7fbf3814d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3da00640 1 -- 172.21.15.98:0/3309157186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf381f77a0 con 0x7fbf38070110 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3da00640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38070110 0x7fbf3814b690 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c00d620 tx=0x7fbf2c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf35600640 1 -- 172.21.15.98:0/3309157186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c00b620 con 0x7fbf38070110 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf35600640 1 -- 172.21.15.98:0/3309157186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf2c019070 con 0x7fbf38070110 2024-05-27T00:29:06.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf35600640 1 -- 172.21.15.98:0/3309157186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c00b9e0 con 0x7fbf38070110 2024-05-27T00:29:06.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3309157186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf381eda20 con 0x7fbf38070110 2024-05-27T00:29:06.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3309157186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf381ecc60 con 0x7fbf38070110 2024-05-27T00:29:06.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.670+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3309157186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf38075c40 con 0x7fbf38070110 2024-05-27T00:29:06.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.674+0000 7fbf35600640 1 -- 172.21.15.98:0/3309157186 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbf2c00bb80 con 0x7fbf38070110 2024-05-27T00:29:06.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.674+0000 7fbf35600640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf180678c0 0x7fbf18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:06.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.674+0000 7fbf35600640 1 -- 172.21.15.98:0/3309157186 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7fbf2c086520 con 0x7fbf38070110 2024-05-27T00:29:06.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.674+0000 7fbf3d000640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf180678c0 0x7fbf18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:06.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.674+0000 7fbf3d000640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf180678c0 0x7fbf18069d80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbf381568a0 tx=0x7fbf24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:06.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.678+0000 7fbf35600640 1 -- 172.21.15.98:0/3309157186 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fbf2c056bf0 con 0x7fbf38070110 2024-05-27T00:29:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3309157186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf380c4630 con 0x7fbf38070110 2024-05-27T00:29:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf35600640 1 -- 172.21.15.98:0/3309157186 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+6007 (secure 0 0 0) 0x7fbf2c0564f0 con 0x7fbf38070110 2024-05-27T00:29:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:29:06.870 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:05.630363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:29:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 60 2024-05-27T00:29:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf180678c0 msgr2=0x7fbf18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf180678c0 0x7fbf18069d80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbf381568a0 tx=0x7fbf24009040 comp rx=0 tx=0).stop 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3309157186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38070110 msgr2=0x7fbf3814b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38070110 0x7fbf3814b690 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c00d620 tx=0x7fbf2c00dae0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3309157186 shutdown_connections 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf180678c0 0x7fbf18069d80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf38157d80 0x7fbf3814d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf380c5490 0x7fbf381f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 --2- 172.21.15.98:0/3309157186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf38070110 0x7fbf3814b690 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3309157186 >> 172.21.15.98:0/3309157186 conn(0x7fbf38070a10 msgr2=0x7fbf380c4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3309157186 shutdown_connections 2024-05-27T00:29:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:06.870+0000 7fbf3ee00640 1 -- 172.21.15.98:0/3309157186 wait complete. 2024-05-27T00:29:06.891 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:29:08.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:29:09.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.038+0000 7fbc67200640 1 Processor -- start 2024-05-27T00:29:09.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.038+0000 7fbc67200640 1 -- start start 2024-05-27T00:29:09.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc67200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fbc601511d0 0x7fbc60070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:09.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc67200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc600c1000 con 0x7fbc600c8670 2024-05-27T00:29:09.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc67200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc601515b0 con 0x7fbc600cc8f0 2024-05-27T00:29:09.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc67200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fbc600b1dd0 con 0x7fbc601511d0 2024-05-27T00:29:09.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc65200640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbc600c8670 0x7fbc601617d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53102/0 (socket says 172.21.15.98:53102) 2024-05-27T00:29:09.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc65200640 1 -- 172.21.15.98:0/3892933547 learned_addr learned my addr 172.21.15.98:0/3892933547 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:29:09.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5fe00640 1 --2- 172.21.15.98:0/3892933547 >> v2:172.21.15.204:3300/0 conn(0x7fbc601511d0 0x7fbc60070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:09.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3821218735 0 0) 0x7fbc600c1000 con 0x7fbc600c8670 2024-05-27T00:29:09.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc44003450 con 0x7fbc600c8670 2024-05-27T00:29:09.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4034623612 0 0) 0x7fbc44003450 con 0x7fbc600c8670 2024-05-27T00:29:09.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 >> v2:172.21.15.204:3300/0 conn(0x7fbc601511d0 msgr2=0x7fbc60070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:09.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 --2- 172.21.15.98:0/3892933547 >> v2:172.21.15.204:3300/0 conn(0x7fbc601511d0 0x7fbc60070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 >> v1:172.21.15.204:6789/0 conn(0x7fbc600cc8f0 legacy=0x7fbc600c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:09.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc600c83c0 con 0x7fbc600c8670 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fbc58001680 con 0x7fbc600c8670 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 >> v1:172.21.15.98:6789/0 conn(0x7fbc600c8670 legacy=0x7fbc601617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc44003ab0 0x7fbc44003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc44008860 0x7fbc4400acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc4400b570 0x7fbc4400da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc600b1dd0 con 0x7fbc44003ab0 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4400e310 con 0x7fbc44008860 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4400e510 con 0x7fbc4400b570 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5fe00640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc44003ab0 0x7fbc44003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc57400640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc44008860 0x7fbc4400acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc65200640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc4400b570 0x7fbc4400da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5fe00640 1 -- 172.21.15.98:0/3892933547 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc4400b570 msgr2=0x7fbc4400da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5fe00640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc4400b570 0x7fbc4400da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5fe00640 1 -- 172.21.15.98:0/3892933547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc44008860 msgr2=0x7fbc4400acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5fe00640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc44008860 0x7fbc4400acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc5fe00640 1 -- 172.21.15.98:0/3892933547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc44003450 con 0x7fbc44003ab0 2024-05-27T00:29:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.042+0000 7fbc65200640 1 -- 172.21.15.98:0/3892933547 reap_dead start 2024-05-27T00:29:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc5fe00640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc44003ab0 0x7fbc44003ed0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009fc0 tx=0x7fbc5001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443c3206870115f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc500b2070 con 0x7fbc44003ab0 2024-05-27T00:29:09.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc500a5a90 con 0x7fbc44003ab0 2024-05-27T00:29:09.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc5f400640 1 -- 172.21.15.98:0/3892933547 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50009c10 con 0x7fbc44003ab0 2024-05-27T00:29:09.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc67200640 1 -- 172.21.15.98:0/3892933547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc44003ab0 msgr2=0x7fbc44003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:09.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc67200640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc44003ab0 0x7fbc44003ed0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009fc0 tx=0x7fbc5001cca0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc67200640 1 -- 172.21.15.98:0/3892933547 shutdown_connections 2024-05-27T00:29:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc67200640 1 --2- 172.21.15.98:0/3892933547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc44003ab0 0x7fbc44003ed0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc67200640 1 -- 172.21.15.98:0/3892933547 >> 172.21.15.98:0/3892933547 conn(0x7fbc60067530 msgr2=0x7fbc6006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc67200640 1 -- 172.21.15.98:0/3892933547 shutdown_connections 2024-05-27T00:29:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.046+0000 7fbc67200640 1 -- 172.21.15.98:0/3892933547 wait complete. 2024-05-27T00:29:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 Processor -- start 2024-05-27T00:29:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 -- start start 2024-05-27T00:29:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc44003ab0 0x7fbc600cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc6014a230 0x7fbc60155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc60154ce0 0x7fbc601564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc600c8e60 con 0x7fbc6014a230 2024-05-27T00:29:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc60071220 con 0x7fbc60154ce0 2024-05-27T00:29:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbc600c2150 con 0x7fbc44003ab0 2024-05-27T00:29:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc57400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc6014a230 0x7fbc60155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc57400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc6014a230 0x7fbc60155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41006/0 (socket says 172.21.15.98:41006) 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc57400640 1 -- 172.21.15.98:0/4289651558 learned_addr learned my addr 172.21.15.98:0/4289651558 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc5fe00640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc44003ab0 0x7fbc600cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc65200640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc60154ce0 0x7fbc601564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc57400640 1 -- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc44003ab0 msgr2=0x7fbc600cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc57400640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc44003ab0 0x7fbc600cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc57400640 1 -- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc60154ce0 msgr2=0x7fbc601564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc57400640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc60154ce0 0x7fbc601564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc57400640 1 -- 172.21.15.98:0/4289651558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6014e8e0 con 0x7fbc6014a230 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc65200640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc60154ce0 0x7fbc601564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc57400640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc6014a230 0x7fbc60155f60 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbc48001ec0 tx=0x7fbc4800e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc5cc00640 1 -- 172.21.15.98:0/4289651558 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48019070 con 0x7fbc6014a230 2024-05-27T00:29:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc5cc00640 1 -- 172.21.15.98:0/4289651558 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc480052e0 con 0x7fbc6014a230 2024-05-27T00:29:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 -- 172.21.15.98:0/4289651558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc6014eb90 con 0x7fbc6014a230 2024-05-27T00:29:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc5cc00640 1 -- 172.21.15.98:0/4289651558 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48003820 con 0x7fbc6014a230 2024-05-27T00:29:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 -- 172.21.15.98:0/4289651558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc6014f230 con 0x7fbc6014a230 2024-05-27T00:29:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.050+0000 7fbc67200640 1 -- 172.21.15.98:0/4289651558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc60075cb0 con 0x7fbc6014a230 2024-05-27T00:29:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.054+0000 7fbc5cc00640 1 -- 172.21.15.98:0/4289651558 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbc4801c070 con 0x7fbc6014a230 2024-05-27T00:29:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.054+0000 7fbc5cc00640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc3c0678c0 0x7fbc3c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.054+0000 7fbc5cc00640 1 -- 172.21.15.98:0/4289651558 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7fbc48085610 con 0x7fbc6014a230 2024-05-27T00:29:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.054+0000 7fbc5fe00640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc3c0678c0 0x7fbc3c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.054+0000 7fbc5fe00640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc3c0678c0 0x7fbc3c069d80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbc500af000 tx=0x7fbc50004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:09.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.058+0000 7fbc5cc00640 1 -- 172.21.15.98:0/4289651558 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fbc48055720 con 0x7fbc6014a230 2024-05-27T00:29:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.246+0000 7fbc67200640 1 -- 172.21.15.98:0/4289651558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc600713c0 con 0x7fbc6014a230 2024-05-27T00:29:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.246+0000 7fbc5cc00640 1 -- 172.21.15.98:0/4289651558 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+6007 (secure 0 0 0) 0x7fbc48059a20 con 0x7fbc6014a230 2024-05-27T00:29:09.248 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:29:09.248 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:05.630363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:29:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 60 2024-05-27T00:29:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 -- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc3c0678c0 msgr2=0x7fbc3c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc3c0678c0 0x7fbc3c069d80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbc500af000 tx=0x7fbc50004ac0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 -- 172.21.15.98:0/4289651558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc6014a230 msgr2=0x7fbc60155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc6014a230 0x7fbc60155f60 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbc48001ec0 tx=0x7fbc4800e8a0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 -- 172.21.15.98:0/4289651558 shutdown_connections 2024-05-27T00:29:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbc3c0678c0 0x7fbc3c069d80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc60154ce0 0x7fbc601564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc6014a230 0x7fbc60155f60 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 --2- 172.21.15.98:0/4289651558 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc44003ab0 0x7fbc600cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:09.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 -- 172.21.15.98:0/4289651558 >> 172.21.15.98:0/4289651558 conn(0x7fbc60067530 msgr2=0x7fbc60072670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:09.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 -- 172.21.15.98:0/4289651558 shutdown_connections 2024-05-27T00:29:09.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:09.250+0000 7fbc67200640 1 -- 172.21.15.98:0/4289651558 wait complete. 2024-05-27T00:29:09.270 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:29:11.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:29:11.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf41400640 1 Processor -- start 2024-05-27T00:29:11.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf41400640 1 -- start start 2024-05-27T00:29:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf41400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fdf3c152520 0x7fdf3c0c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf41400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf3c073760 0x7fdf3c071e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf41400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf3c06cc70 con 0x7fdf3c0c70e0 2024-05-27T00:29:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf41400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdf3c072fa0 con 0x7fdf3c073760 2024-05-27T00:29:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf41400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fdf3c152900 con 0x7fdf3c152520 2024-05-27T00:29:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf39e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdf3c073760 0x7fdf3c071e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf3b200640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdf3c0c70e0 0x7fdf3c06b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53110/0 (socket says 172.21.15.98:53110) 2024-05-27T00:29:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf3b200640 1 -- 172.21.15.98:0/3218779212 learned_addr learned my addr 172.21.15.98:0/3218779212 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:29:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf3a800640 1 --2- 172.21.15.98:0/3218779212 >> v2:172.21.15.204:3300/0 conn(0x7fdf3c152520 0x7fdf3c0c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:11.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf39e00640 1 -- 172.21.15.98:0/3218779212 >> v2:172.21.15.204:3300/0 conn(0x7fdf3c152520 msgr2=0x7fdf3c0c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:11.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf39e00640 1 --2- 172.21.15.98:0/3218779212 >> v2:172.21.15.204:3300/0 conn(0x7fdf3c152520 0x7fdf3c0c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:11.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf39e00640 1 -- 172.21.15.98:0/3218779212 >> v1:172.21.15.98:6789/0 conn(0x7fdf3c0c70e0 legacy=0x7fdf3c06b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:11.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.417+0000 7fdf39e00640 1 -- 172.21.15.98:0/3218779212 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3c06c6b0 con 0x7fdf3c073760 2024-05-27T00:29:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf39e00640 1 --2- 172.21.15.98:0/3218779212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 0x7fdf3c071e70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0ca960 tx=0x7fdf2801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a2f2ef596e3255 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf39400640 1 -- 172.21.15.98:0/3218779212 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf280b0070 con 0x7fdf3c073760 2024-05-27T00:29:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf39400640 1 -- 172.21.15.98:0/3218779212 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf280ab600 con 0x7fdf3c073760 2024-05-27T00:29:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf39400640 1 -- 172.21.15.98:0/3218779212 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf280ab9d0 con 0x7fdf3c073760 2024-05-27T00:29:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf41400640 1 -- 172.21.15.98:0/3218779212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 msgr2=0x7fdf3c071e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:11.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf41400640 1 --2- 172.21.15.98:0/3218779212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 0x7fdf3c071e70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0ca960 tx=0x7fdf2801c480 comp rx=0 tx=0).stop 2024-05-27T00:29:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf41400640 1 -- 172.21.15.98:0/3218779212 shutdown_connections 2024-05-27T00:29:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf41400640 1 --2- 172.21.15.98:0/3218779212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 0x7fdf3c071e70 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf41400640 1 --2- 172.21.15.98:0/3218779212 >> v2:172.21.15.204:3300/0 conn(0x7fdf3c152520 0x7fdf3c0c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf41400640 1 -- 172.21.15.98:0/3218779212 >> 172.21.15.98:0/3218779212 conn(0x7fdf3c0c9b70 msgr2=0x7fdf3c0c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf41400640 1 -- 172.21.15.98:0/3218779212 shutdown_connections 2024-05-27T00:29:11.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.421+0000 7fdf41400640 1 -- 172.21.15.98:0/3218779212 wait complete. 2024-05-27T00:29:11.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 Processor -- start 2024-05-27T00:29:11.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 -- start start 2024-05-27T00:29:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 0x7fdf3c151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf3c0c70e0 0x7fdf3c14f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf3c152520 0x7fdf3c1ea790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c06bfb0 con 0x7fdf3c073760 2024-05-27T00:29:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c06be30 con 0x7fdf3c152520 2024-05-27T00:29:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdf3c1656a0 con 0x7fdf3c0c70e0 2024-05-27T00:29:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf3a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 0x7fdf3c151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf3a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 0x7fdf3c151160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41034/0 (socket says 172.21.15.98:41034) 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf3a800640 1 -- 172.21.15.98:0/2103375732 learned_addr learned my addr 172.21.15.98:0/2103375732 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf3b200640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf3c152520 0x7fdf3c1ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf39e00640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf3c0c70e0 0x7fdf3c14f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf3a800640 1 -- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf3c0c70e0 msgr2=0x7fdf3c14f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf3a800640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf3c0c70e0 0x7fdf3c14f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf3a800640 1 -- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf3c152520 msgr2=0x7fdf3c1ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf3a800640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf3c152520 0x7fdf3c1ea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf3a800640 1 -- 172.21.15.98:0/2103375732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3c14f300 con 0x7fdf3c073760 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf3a800640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 0x7fdf3c151160 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00e960 tx=0x7fdf2c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf32a00640 1 -- 172.21.15.98:0/2103375732 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c017070 con 0x7fdf3c073760 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf32a00640 1 -- 172.21.15.98:0/2103375732 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf2c00c9a0 con 0x7fdf3c073760 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 -- 172.21.15.98:0/2103375732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf3c1516a0 con 0x7fdf3c073760 2024-05-27T00:29:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf32a00640 1 -- 172.21.15.98:0/2103375732 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c00cd50 con 0x7fdf3c073760 2024-05-27T00:29:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 -- 172.21.15.98:0/2103375732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf3c14d590 con 0x7fdf3c073760 2024-05-27T00:29:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.425+0000 7fdf41400640 1 -- 172.21.15.98:0/2103375732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf3c075cb0 con 0x7fdf3c073760 2024-05-27T00:29:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.429+0000 7fdf32a00640 1 -- 172.21.15.98:0/2103375732 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf2c01a070 con 0x7fdf3c073760 2024-05-27T00:29:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.429+0000 7fdf32a00640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf1c0678c0 0x7fdf1c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.429+0000 7fdf32a00640 1 -- 172.21.15.98:0/2103375732 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7fdf2c0863d0 con 0x7fdf3c073760 2024-05-27T00:29:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.429+0000 7fdf39e00640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf1c0678c0 0x7fdf1c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.429+0000 7fdf39e00640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf1c0678c0 0x7fdf1c069d80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdf280ad000 tx=0x7fdf28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.433+0000 7fdf32a00640 1 -- 172.21.15.98:0/2103375732 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fdf2c056b30 con 0x7fdf3c073760 2024-05-27T00:29:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.629+0000 7fdf41400640 1 -- 172.21.15.98:0/2103375732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf3c0c6cb0 con 0x7fdf3c073760 2024-05-27T00:29:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.633+0000 7fdf32a00640 1 -- 172.21.15.98:0/2103375732 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+6007 (secure 0 0 0) 0x7fdf2c056430 con 0x7fdf3c073760 2024-05-27T00:29:11.634 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:29:11.635 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:05.630363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:29:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 60 2024-05-27T00:29:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 -- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf1c0678c0 msgr2=0x7fdf1c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf1c0678c0 0x7fdf1c069d80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdf280ad000 tx=0x7fdf28008040 comp rx=0 tx=0).stop 2024-05-27T00:29:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 -- 172.21.15.98:0/2103375732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 msgr2=0x7fdf3c151160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:11.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 0x7fdf3c151160 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00e960 tx=0x7fdf2c00ee20 comp rx=0 tx=0).stop 2024-05-27T00:29:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 -- 172.21.15.98:0/2103375732 shutdown_connections 2024-05-27T00:29:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fdf1c0678c0 0x7fdf1c069d80 secure :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fdf280ad000 tx=0x7fdf28008040 comp rx=0 tx=0).stop 2024-05-27T00:29:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf3c152520 0x7fdf3c1ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdf3c0c70e0 0x7fdf3c14f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 --2- 172.21.15.98:0/2103375732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf3c073760 0x7fdf3c151160 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 -- 172.21.15.98:0/2103375732 >> 172.21.15.98:0/2103375732 conn(0x7fdf3c0c9b70 msgr2=0x7fdf3c156780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 -- 172.21.15.98:0/2103375732 shutdown_connections 2024-05-27T00:29:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:11.637+0000 7fdf41400640 1 -- 172.21.15.98:0/2103375732 wait complete. 2024-05-27T00:29:11.686 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:29:13.687 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:29:13.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f9497a00640 1 Processor -- start 2024-05-27T00:29:13.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f9497a00640 1 -- start start 2024-05-27T00:29:13.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f9497a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900731f0 0x7f94900c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:13.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f9497a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f94900735d0 0x7f9490151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f9497a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94900c17d0 con 0x7f94900c79f0 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f9497a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9490071990 con 0x7f94900735d0 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f9497a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9490070e10 con 0x7f94900731f0 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f948fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f94900735d0 0x7f9490151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f948fe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f94900735d0 0x7f9490151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37986/0 (socket says 172.21.15.98:37986) 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f948fe00640 1 -- 172.21.15.98:0/2942135516 learned_addr learned my addr 172.21.15.98:0/2942135516 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f9495000640 1 --2- 172.21.15.98:0/2942135516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900731f0 0x7f94900c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f948fe00640 1 -- 172.21.15.98:0/2942135516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900731f0 msgr2=0x7f94900c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f948fe00640 1 --2- 172.21.15.98:0/2942135516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900731f0 0x7f94900c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f948fe00640 1 -- 172.21.15.98:0/2942135516 >> v1:172.21.15.204:6789/0 conn(0x7f94900c79f0 legacy=0x7f9490157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:13.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.833+0000 7f948fe00640 1 -- 172.21.15.98:0/2942135516 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94900710f0 con 0x7f94900735d0 2024-05-27T00:29:13.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f948fe00640 1 --2- 172.21.15.98:0/2942135516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900735d0 0x7f9490151430 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f947800bb10 tx=0x7f9478021420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c7dda20832bd74 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:13.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f948f400640 1 -- 172.21.15.98:0/2942135516 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94780b6070 con 0x7f94900735d0 2024-05-27T00:29:13.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f948f400640 1 -- 172.21.15.98:0/2942135516 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94780b1600 con 0x7f94900735d0 2024-05-27T00:29:13.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f948f400640 1 -- 172.21.15.98:0/2942135516 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94780b19d0 con 0x7f94900735d0 2024-05-27T00:29:13.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f9497a00640 1 -- 172.21.15.98:0/2942135516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900735d0 msgr2=0x7f9490151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:13.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f9497a00640 1 --2- 172.21.15.98:0/2942135516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900735d0 0x7f9490151430 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f947800bb10 tx=0x7f9478021420 comp rx=0 tx=0).stop 2024-05-27T00:29:13.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f9497a00640 1 -- 172.21.15.98:0/2942135516 shutdown_connections 2024-05-27T00:29:13.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f9497a00640 1 --2- 172.21.15.98:0/2942135516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900735d0 0x7f9490151430 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:13.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f9497a00640 1 --2- 172.21.15.98:0/2942135516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900731f0 0x7f94900c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:13.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f9497a00640 1 -- 172.21.15.98:0/2942135516 >> 172.21.15.98:0/2942135516 conn(0x7f9490067530 msgr2=0x7f949006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:13.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f9497a00640 1 -- 172.21.15.98:0/2942135516 shutdown_connections 2024-05-27T00:29:13.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.837+0000 7f9497a00640 1 -- 172.21.15.98:0/2942135516 wait complete. 2024-05-27T00:29:13.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9497a00640 1 Processor -- start 2024-05-27T00:29:13.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9497a00640 1 -- start start 2024-05-27T00:29:13.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9497a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900731f0 0x7f949015ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:13.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9497a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94900735d0 0x7f94901489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9497a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900c79f0 0x7f9490148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9497a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94900c8400 con 0x7f94900731f0 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9497a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f94900716b0 con 0x7f94900735d0 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9497a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f94900c13f0 con 0x7f94900c79f0 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900731f0 0x7f949015ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900731f0 0x7f949015ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37996/0 (socket says 172.21.15.98:37996) 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495000640 1 -- 172.21.15.98:0/3205775778 learned_addr learned my addr 172.21.15.98:0/3205775778 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f948fe00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94900735d0 0x7f94901489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495a00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900c79f0 0x7f9490148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495000640 1 -- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900c79f0 msgr2=0x7f9490148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:13.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495000640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900c79f0 0x7f9490148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495000640 1 -- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94900735d0 msgr2=0x7f94901489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495000640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94900735d0 0x7f94901489b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495000640 1 -- 172.21.15.98:0/3205775778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9490145680 con 0x7f94900731f0 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495a00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900c79f0 0x7f9490148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9495000640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900731f0 0x7f949015ae80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f948000e960 tx=0x7f948000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f948cc00640 1 -- 172.21.15.98:0/3205775778 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480017070 con 0x7f94900731f0 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f948cc00640 1 -- 172.21.15.98:0/3205775778 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f948000ca00 con 0x7f94900731f0 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f948cc00640 1 -- 172.21.15.98:0/3205775778 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948000cdb0 con 0x7f94900731f0 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9497a00640 1 -- 172.21.15.98:0/3205775778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949006ed50 con 0x7f94900731f0 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.841+0000 7f9497a00640 1 -- 172.21.15.98:0/3205775778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f949014b260 con 0x7f94900731f0 2024-05-27T00:29:13.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.845+0000 7f9497a00640 1 -- 172.21.15.98:0/3205775778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9490075cb0 con 0x7f94900731f0 2024-05-27T00:29:13.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.845+0000 7f948cc00640 1 -- 172.21.15.98:0/3205775778 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f948001a070 con 0x7f94900731f0 2024-05-27T00:29:13.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.845+0000 7f948cc00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f946c0678f0 0x7f946c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:13.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.845+0000 7f948cc00640 1 -- 172.21.15.98:0/3205775778 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f9480085ff0 con 0x7f94900731f0 2024-05-27T00:29:13.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.845+0000 7f948fe00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f946c0678f0 0x7f946c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:13.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.845+0000 7f948fe00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f946c0678f0 0x7f946c069db0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f94780b3000 tx=0x7f94780ac000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:13.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:13.849+0000 7f948cc00640 1 -- 172.21.15.98:0/3205775778 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f94800567f0 con 0x7f94900731f0 2024-05-27T00:29:14.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.037+0000 7f9497a00640 1 -- 172.21.15.98:0/3205775778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94901f0f20 con 0x7f94900731f0 2024-05-27T00:29:14.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.037+0000 7f948cc00640 1 -- 172.21.15.98:0/3205775778 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6014 (secure 0 0 0) 0x7f94800560f0 con 0x7f94900731f0 2024-05-27T00:29:14.040 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:29:14.041 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:12.690345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6078,"qdb_cluster":[6078]},"id":1}]} 2024-05-27T00:29:14.041 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 61 2024-05-27T00:29:14.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 -- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f946c0678f0 msgr2=0x7f946c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:14.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f946c0678f0 0x7f946c069db0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f94780b3000 tx=0x7f94780ac000 comp rx=0 tx=0).stop 2024-05-27T00:29:14.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 -- 172.21.15.98:0/3205775778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900731f0 msgr2=0x7f949015ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:14.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900731f0 0x7f949015ae80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f948000e960 tx=0x7f948000ee20 comp rx=0 tx=0).stop 2024-05-27T00:29:14.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 -- 172.21.15.98:0/3205775778 shutdown_connections 2024-05-27T00:29:14.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94900c79f0 0x7f9490148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:14.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f946c0678f0 0x7f946c069db0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:14.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94900735d0 0x7f94901489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:14.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 --2- 172.21.15.98:0/3205775778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94900731f0 0x7f949015ae80 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:14.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 -- 172.21.15.98:0/3205775778 >> 172.21.15.98:0/3205775778 conn(0x7f9490067530 msgr2=0x7f9490150450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:14.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 -- 172.21.15.98:0/3205775778 shutdown_connections 2024-05-27T00:29:14.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:14.041+0000 7f9497a00640 1 -- 172.21.15.98:0/3205775778 wait complete. 2024-05-27T00:29:14.062 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=5943 2024-05-27T00:29:14.062 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2024-05-27T00:29:20.134 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 50% (123/246) Updating files: 51% (126/246) Updating files: 52% (128/246) Updating files: 53% (131/246) Updating files: 54% (133/246) Updating files: 55% (136/246) Updating files: 56% (138/246) Updating files: 57% (141/246) Updating files: 58% (143/246) Updating files: 59% (146/246) Updating files: 60% (148/246) Updating files: 61% (151/246) Updating files: 62% (153/246) Updating files: 63% (155/246) Updating files: 64% (158/246) Updating files: 65% (160/246) Updating files: 66% (163/246) Updating files: 67% (165/246) Updating files: 68% (168/246) Updating files: 69% (170/246) Updating files: 70% (173/246) Updating files: 71% (175/246) Updating files: 72% (178/246) Updating files: 73% (180/246) Updating files: 74% (183/246) Updating files: 75% (185/246) Updating files: 76% (187/246) Updating files: 77% (190/246) Updating files: 78% (192/246) Updating files: 79% (195/246) Updating files: 80% (197/246) Updating files: 81% (200/246) Updating files: 82% (202/246) Updating files: 83% (205/246) Updating files: 84% (207/246) Updating files: 85% (210/246) Updating files: 86% (212/246) Updating files: 87% (215/246) Updating files: 88% (217/246) Updating files: 89% (219/246) Updating files: 90% (222/246) Updating files: 91% (224/246) Updating files: 92% (227/246) Updating files: 93% (229/246) Updating files: 94% (232/246) Updating files: 95% (234/246) Updating files: 96% (237/246) Updating files: 97% (239/246) Updating files: 98% (242/246) Updating files: 99% (244/246) Updating files: 100% (246/246) Updating files: 100% (246/246), done. 2024-05-27T00:29:20.533 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 1a32f0a0b42 v0.56 2024-05-27T00:29:20.538 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.56 2024-05-27T00:29:20.558 INFO:tasks.workunit.client.0.smithi098.stdout:v0.57 2024-05-27T00:29:20.558 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 57 -eq 48 ] 2024-05-27T00:29:20.558 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.57 2024-05-27T00:29:20.558 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.57 2024-05-27T00:29:20.558 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.57 2024-05-27T00:29:22.063 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-05-27T00:29:22.064 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-27T00:29:22.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-27T00:29:22.068 INFO:tasks.ceph.mds.a:Started 2024-05-27T00:29:22.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:29:22.164 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2024-05-27T00:29:22.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.225+0000 7f5aa0400640 1 Processor -- start 2024-05-27T00:29:22.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5aa0400640 1 -- start start 2024-05-27T00:29:22.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5aa0400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5a98070c00 0x7f5a980c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5aa0400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a98070fe0 0x7f5a98154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5aa0400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f5a980c8f40 0x7f5a98154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5aa0400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5a9815a950 con 0x7f5a98070c00 2024-05-27T00:29:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5aa0400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f5a9815e060 con 0x7f5a980c8f40 2024-05-27T00:29:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5aa0400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5a9815e340 con 0x7f5a98070fe0 2024-05-27T00:29:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5a9da00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5a98070c00 0x7f5a980c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5a9da00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5a98070c00 0x7f5a980c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38020/0 (socket says 172.21.15.98:38020) 2024-05-27T00:29:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5a9da00640 1 -- 172.21.15.98:0/2610923251 learned_addr learned my addr 172.21.15.98:0/2610923251 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:29:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5a9e400640 1 --2- 172.21.15.98:0/2610923251 >> v2:172.21.15.204:3300/0 conn(0x7f5a980c8f40 0x7f5a98154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:22.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5a9d000640 1 --2- 172.21.15.98:0/2610923251 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a98070fe0 0x7f5a98154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:22.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5a9da00640 1 -- 172.21.15.98:0/2610923251 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a98070fe0 msgr2=0x7f5a98154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:22.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5a9da00640 1 --2- 172.21.15.98:0/2610923251 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a98070fe0 0x7f5a98154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5a9da00640 1 -- 172.21.15.98:0/2610923251 >> v2:172.21.15.204:3300/0 conn(0x7f5a980c8f40 msgr2=0x7f5a98154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:29:22.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5a9da00640 1 --2- 172.21.15.98:0/2610923251 >> v2:172.21.15.204:3300/0 conn(0x7f5a980c8f40 0x7f5a98154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.229+0000 7f5a9da00640 1 -- 172.21.15.98:0/2610923251 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9815e620 con 0x7f5a98070c00 2024-05-27T00:29:22.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5a9da00640 1 --2- 172.21.15.98:0/2610923251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070c00 0x7f5a980c8a00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5a98072b80 tx=0x7f5a8c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5671b38b2cbe4d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:22.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5a97e00640 1 -- 172.21.15.98:0/2610923251 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c0b5070 con 0x7f5a98070c00 2024-05-27T00:29:22.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5a97e00640 1 -- 172.21.15.98:0/2610923251 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a8c0a89d0 con 0x7f5a98070c00 2024-05-27T00:29:22.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5a97e00640 1 -- 172.21.15.98:0/2610923251 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c0a8da0 con 0x7f5a98070c00 2024-05-27T00:29:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5aa0400640 1 -- 172.21.15.98:0/2610923251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070c00 msgr2=0x7f5a980c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5aa0400640 1 --2- 172.21.15.98:0/2610923251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070c00 0x7f5a980c8a00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5a98072b80 tx=0x7f5a8c01ed10 comp rx=0 tx=0).stop 2024-05-27T00:29:22.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5aa0400640 1 -- 172.21.15.98:0/2610923251 shutdown_connections 2024-05-27T00:29:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5aa0400640 1 --2- 172.21.15.98:0/2610923251 >> v2:172.21.15.204:3300/0 conn(0x7f5a980c8f40 0x7f5a98154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5aa0400640 1 --2- 172.21.15.98:0/2610923251 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a98070fe0 0x7f5a98154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5aa0400640 1 --2- 172.21.15.98:0/2610923251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070c00 0x7f5a980c8a00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5aa0400640 1 -- 172.21.15.98:0/2610923251 >> 172.21.15.98:0/2610923251 conn(0x7f5a98072f90 msgr2=0x7f5a980733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5aa0400640 1 -- 172.21.15.98:0/2610923251 shutdown_connections 2024-05-27T00:29:22.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.233+0000 7f5aa0400640 1 -- 172.21.15.98:0/2610923251 wait complete. 2024-05-27T00:29:22.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5aa0400640 1 Processor -- start 2024-05-27T00:29:22.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5aa0400640 1 -- start start 2024-05-27T00:29:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5aa0400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a98070c00 0x7f5a981f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5aa0400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070fe0 0x7f5a980ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5aa0400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a980c8f40 0x7f5a981528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:22.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5aa0400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5a980cb570 con 0x7f5a98070fe0 2024-05-27T00:29:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5aa0400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9815ebe0 con 0x7f5a980c8f40 2024-05-27T00:29:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5aa0400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5a980cb6f0 con 0x7f5a98070c00 2024-05-27T00:29:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5a9d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070fe0 0x7f5a980ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5a9d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070fe0 0x7f5a980ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38028/0 (socket says 172.21.15.98:38028) 2024-05-27T00:29:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5a9d000640 1 -- 172.21.15.98:0/998440779 learned_addr learned my addr 172.21.15.98:0/998440779 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:29:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5a9e400640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a980c8f40 0x7f5a981528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:22.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.237+0000 7f5a9da00640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a98070c00 0x7f5a981f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:22.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a9d000640 1 -- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a98070c00 msgr2=0x7f5a981f0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:22.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a9d000640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a98070c00 0x7f5a981f0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a9d000640 1 -- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a980c8f40 msgr2=0x7f5a981528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:22.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a9d000640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a980c8f40 0x7f5a981528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a9d000640 1 -- 172.21.15.98:0/998440779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a98152240 con 0x7f5a98070fe0 2024-05-27T00:29:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a9d000640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070fe0 0x7f5a980ce6e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400d800 tx=0x7f5a8400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a95600640 1 -- 172.21.15.98:0/998440779 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8400b6c0 con 0x7f5a98070fe0 2024-05-27T00:29:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a95600640 1 -- 172.21.15.98:0/998440779 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a84019070 con 0x7f5a98070fe0 2024-05-27T00:29:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a95600640 1 -- 172.21.15.98:0/998440779 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8400baa0 con 0x7f5a98070fe0 2024-05-27T00:29:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5aa0400640 1 -- 172.21.15.98:0/998440779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a981f1270 con 0x7f5a98070fe0 2024-05-27T00:29:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5aa0400640 1 -- 172.21.15.98:0/998440779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a980c7ee0 con 0x7f5a98070fe0 2024-05-27T00:29:22.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5aa0400640 1 -- 172.21.15.98:0/998440779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a98075cb0 con 0x7f5a98070fe0 2024-05-27T00:29:22.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a95600640 1 -- 172.21.15.98:0/998440779 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a8400bc40 con 0x7f5a98070fe0 2024-05-27T00:29:22.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a95600640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a780678c0 0x7f5a78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:29:22.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.241+0000 7f5a95600640 1 -- 172.21.15.98:0/998440779 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f5a84086400 con 0x7f5a98070fe0 2024-05-27T00:29:22.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.245+0000 7f5a9da00640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a780678c0 0x7f5a78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:29:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.245+0000 7f5a9da00640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a780678c0 0x7f5a78069d80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0b2000 tx=0x7f5a8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:29:22.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.249+0000 7f5a95600640 1 -- 172.21.15.98:0/998440779 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f5a84056480 con 0x7f5a98070fe0 2024-05-27T00:29:22.457 INFO:tasks.ceph.mds.c.smithi098.stderr:2024-05-27T00:29:22.457+0000 7fd531e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:29:22.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.577+0000 7f5aa0400640 1 -- 172.21.15.98:0/998440779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a980c9720 con 0x7f5a98070fe0 2024-05-27T00:29:22.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.577+0000 7f5a95600640 1 -- 172.21.15.98:0/998440779 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6839 (secure 0 0 0) 0x7f5a8405a780 con 0x7f5a98070fe0 2024-05-27T00:29:22.578 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:29:22.579 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:12.690345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6078,"qdb_cluster":[6078]},"id":1}]} 2024-05-27T00:29:22.579 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2024-05-27T00:29:22.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 -- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a780678c0 msgr2=0x7f5a78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:22.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a780678c0 0x7f5a78069d80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c0b2000 tx=0x7f5a8c009040 comp rx=0 tx=0).stop 2024-05-27T00:29:22.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 -- 172.21.15.98:0/998440779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070fe0 msgr2=0x7f5a980ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:29:22.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070fe0 0x7f5a980ce6e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400d800 tx=0x7f5a8400dcc0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 -- 172.21.15.98:0/998440779 shutdown_connections 2024-05-27T00:29:22.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5a780678c0 0x7f5a78069d80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a980c8f40 0x7f5a981528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5a98070fe0 0x7f5a980ce6e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 --2- 172.21.15.98:0/998440779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a98070c00 0x7f5a981f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:29:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 -- 172.21.15.98:0/998440779 >> 172.21.15.98:0/998440779 conn(0x7f5a98072f90 msgr2=0x7f5a9806d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:29:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 -- 172.21.15.98:0/998440779 shutdown_connections 2024-05-27T00:29:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:29:22.581+0000 7f5aa0400640 1 -- 172.21.15.98:0/998440779 wait complete. 2024-05-27T00:29:22.611 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-05-27T00:29:22.611 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 61 secs before thrashing 2024-05-27T00:29:22.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:29:22.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:29:23.042 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 61% (186/304) Updating files: 62% (189/304) Updating files: 63% (192/304) Updating files: 64% (195/304) Updating files: 65% (198/304) Updating files: 66% (201/304) Updating files: 67% (204/304) Updating files: 68% (207/304) Updating files: 69% (210/304) Updating files: 70% (213/304) Updating files: 71% (216/304) Updating files: 72% (219/304) Updating files: 73% (222/304) Updating files: 74% (225/304) Updating files: 75% (228/304) Updating files: 76% (232/304) Updating files: 77% (235/304) Updating files: 78% (238/304) Updating files: 79% (241/304) Updating files: 80% (244/304) Updating files: 81% (247/304) Updating files: 82% (250/304) Updating files: 83% (253/304) Updating files: 84% (256/304) Updating files: 85% (259/304) Updating files: 86% (262/304) Updating files: 87% (265/304) Updating files: 88% (268/304) Updating files: 89% (271/304) Updating files: 90% (274/304) Updating files: 91% (277/304) Updating files: 92% (280/304) Updating files: 93% (283/304) Updating files: 94% (286/304) Updating files: 95% (289/304) Updating files: 96% (292/304) Updating files: 97% (295/304) Updating files: 98% (298/304) Updating files: 99% (301/304) Updating files: 100% (304/304) Updating files: 100% (304/304), done. 2024-05-27T00:29:23.409 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 9a7a9d06c06 v0.57 2024-05-27T00:29:23.422 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.57 2024-05-27T00:29:23.467 INFO:tasks.workunit.client.0.smithi098.stdout:v0.58 2024-05-27T00:29:23.467 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 58 -eq 48 ] 2024-05-27T00:29:23.468 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.58 2024-05-27T00:29:23.468 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.58 2024-05-27T00:29:23.468 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.58 2024-05-27T00:29:25.856 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 48% (130/268) Updating files: 49% (132/268) Updating files: 50% (134/268) Updating files: 51% (137/268) Updating files: 52% (140/268) Updating files: 53% (143/268) Updating files: 54% (145/268) Updating files: 55% (148/268) Updating files: 56% (151/268) Updating files: 57% (153/268) Updating files: 58% (156/268) Updating files: 59% (159/268) Updating files: 60% (161/268) Updating files: 61% (164/268) Updating files: 62% (167/268) Updating files: 63% (169/268) Updating files: 64% (172/268) Updating files: 65% (175/268) Updating files: 66% (177/268) Updating files: 67% (180/268) Updating files: 68% (183/268) Updating files: 69% (185/268) Updating files: 70% (188/268) Updating files: 71% (191/268) Updating files: 72% (193/268) Updating files: 73% (196/268) Updating files: 74% (199/268) Updating files: 75% (201/268) Updating files: 76% (204/268) Updating files: 77% (207/268) Updating files: 78% (210/268) Updating files: 79% (212/268) Updating files: 80% (215/268) Updating files: 81% (218/268) Updating files: 82% (220/268) Updating files: 83% (223/268) Updating files: 84% (226/268) Updating files: 85% (228/268) Updating files: 86% (231/268) Updating files: 87% (234/268) Updating files: 88% (236/268) Updating files: 89% (239/268) Updating files: 90% (242/268) Updating files: 91% (244/268) Updating files: 92% (247/268) Updating files: 93% (250/268) Updating files: 94% (252/268) Updating files: 95% (255/268) Updating files: 96% (258/268) Updating files: 97% (260/268) Updating files: 97% (262/268) Updating files: 98% (263/268) Updating files: 99% (266/268) Updating files: 100% (268/268) Updating files: 100% (268/268), done. 2024-05-27T00:29:25.882 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at ba3f91e7504 v0.58 2024-05-27T00:29:25.889 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.58 2024-05-27T00:29:26.003 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 59 -eq 48 ] 2024-05-27T00:29:26.004 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.59 2024-05-27T00:29:26.004 INFO:tasks.workunit.client.0.smithi098.stdout:v0.59 2024-05-27T00:29:26.004 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.59 2024-05-27T00:29:26.004 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.59 2024-05-27T00:29:27.087 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at cbae6a435c6 v0.59 2024-05-27T00:29:27.094 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.59 2024-05-27T00:29:27.183 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 60 -eq 48 ] 2024-05-27T00:29:27.183 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.60 2024-05-27T00:29:27.183 INFO:tasks.workunit.client.0.smithi098.stdout:v0.60 2024-05-27T00:29:27.184 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.60 2024-05-27T00:29:27.184 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.60 2024-05-27T00:29:43.887 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 58% (151/256) Updating files: 59% (152/256) Updating files: 60% (154/256) Updating files: 61% (157/256) Updating files: 62% (159/256) Updating files: 63% (162/256) Updating files: 64% (164/256) Updating files: 65% (167/256) Updating files: 66% (169/256) Updating files: 67% (172/256) Updating files: 68% (175/256) Updating files: 69% (177/256) Updating files: 70% (180/256) Updating files: 71% (182/256) Updating files: 72% (185/256) Updating files: 73% (187/256) Updating files: 74% (190/256) Updating files: 75% (192/256) Updating files: 76% (195/256) Updating files: 77% (198/256) Updating files: 78% (200/256) Updating files: 79% (203/256) Updating files: 80% (205/256) Updating files: 81% (208/256) Updating files: 82% (210/256) Updating files: 83% (213/256) Updating files: 84% (216/256) Updating files: 85% (218/256) Updating files: 86% (221/256) Updating files: 87% (223/256) Updating files: 88% (226/256) Updating files: 89% (228/256) Updating files: 90% (231/256) Updating files: 91% (233/256) Updating files: 92% (236/256) Updating files: 93% (239/256) Updating files: 94% (241/256) Updating files: 95% (244/256) Updating files: 96% (246/256) Updating files: 97% (249/256) Updating files: 98% (251/256) Updating files: 99% (254/256) Updating files: 100% (256/256) Updating files: 100% (256/256), done. 2024-05-27T00:29:43.926 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at f26f7a39021 v0.60 2024-05-27T00:29:43.934 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.60 2024-05-27T00:29:44.005 INFO:tasks.workunit.client.0.smithi098.stdout:v0.61 2024-05-27T00:29:44.005 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 61 -eq 48 ] 2024-05-27T00:29:44.006 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.61 2024-05-27T00:29:44.006 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.61 2024-05-27T00:29:44.006 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.61 2024-05-27T00:29:45.885 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 70% (265/376) Updating files: 71% (267/376) Updating files: 72% (271/376) Updating files: 73% (275/376) Updating files: 74% (279/376) Updating files: 75% (282/376) Updating files: 76% (286/376) Updating files: 77% (290/376) Updating files: 78% (294/376) Updating files: 79% (298/376) Updating files: 80% (301/376) Updating files: 81% (305/376) Updating files: 82% (309/376) Updating files: 83% (313/376) Updating files: 84% (316/376) Updating files: 85% (320/376) Updating files: 86% (324/376) Updating files: 87% (328/376) Updating files: 88% (331/376) Updating files: 89% (335/376) Updating files: 90% (339/376) Updating files: 91% (343/376) Updating files: 92% (346/376) Updating files: 93% (350/376) Updating files: 94% (354/376) Updating files: 95% (358/376) Updating files: 96% (361/376) Updating files: 97% (365/376) Updating files: 98% (369/376) Updating files: 99% (373/376) Updating files: 100% (376/376) Updating files: 100% (376/376), done. 2024-05-27T00:29:45.927 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 237f3f1e8d8 v0.61 2024-05-27T00:29:45.935 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.61 2024-05-27T00:29:46.104 INFO:tasks.workunit.client.0.smithi098.stdout:v0.62 2024-05-27T00:29:46.104 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 62 -eq 48 ] 2024-05-27T00:29:46.104 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.62 2024-05-27T00:29:46.104 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.62 2024-05-27T00:29:46.104 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.62 2024-05-27T00:29:46.778 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 9382379cc6b v0.62 2024-05-27T00:29:46.781 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.62 2024-05-27T00:29:46.920 INFO:tasks.workunit.client.0.smithi098.stdout:v0.63 2024-05-27T00:29:46.921 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 63 -eq 48 ] 2024-05-27T00:29:46.921 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.63 2024-05-27T00:29:46.921 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.63 2024-05-27T00:29:46.921 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.63 2024-05-27T00:29:48.146 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 054e96cf79e v0.63 2024-05-27T00:29:48.152 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.63 2024-05-27T00:29:48.280 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 64 -eq 48 ] 2024-05-27T00:29:48.280 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.64 2024-05-27T00:29:48.280 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.64 2024-05-27T00:29:48.280 INFO:tasks.workunit.client.0.smithi098.stdout:v0.64 2024-05-27T00:29:48.280 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.64 2024-05-27T00:29:49.422 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 42e06c12db6 v0.64 2024-05-27T00:29:49.429 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.64 2024-05-27T00:29:49.556 INFO:tasks.workunit.client.0.smithi098.stdout:v0.65 2024-05-27T00:29:49.556 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 65 -eq 48 ] 2024-05-27T00:29:49.556 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.65 2024-05-27T00:29:49.556 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.65 2024-05-27T00:29:49.556 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.65 2024-05-27T00:29:53.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:29:53.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:30:03.948 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 52% (106/203) Updating files: 53% (108/203) Updating files: 54% (110/203) Updating files: 55% (112/203) Updating files: 56% (114/203) Updating files: 57% (116/203) Updating files: 58% (118/203) Updating files: 59% (120/203) Updating files: 60% (122/203) Updating files: 61% (124/203) Updating files: 62% (126/203) Updating files: 63% (128/203) Updating files: 64% (130/203) Updating files: 65% (132/203) Updating files: 66% (134/203) Updating files: 67% (137/203) Updating files: 68% (139/203) Updating files: 69% (141/203) Updating files: 70% (143/203) Updating files: 71% (145/203) Updating files: 72% (147/203) Updating files: 73% (149/203) Updating files: 74% (151/203) Updating files: 75% (153/203) Updating files: 76% (155/203) Updating files: 77% (157/203) Updating files: 78% (159/203) Updating files: 79% (161/203) Updating files: 80% (163/203) Updating files: 81% (165/203) Updating files: 82% (167/203) Updating files: 83% (169/203) Updating files: 84% (171/203) Updating files: 85% (173/203) Updating files: 86% (175/203) Updating files: 87% (177/203) Updating files: 88% (179/203) Updating files: 89% (181/203) Updating files: 90% (183/203) Updating files: 91% (185/203) Updating files: 92% (187/203) Updating files: 93% (189/203) Updating files: 94% (191/203) Updating files: 95% (193/203) Updating files: 96% (195/203) Updating files: 97% (197/203) Updating files: 98% (199/203) Updating files: 99% (201/203) Updating files: 99% (202/203) Updating files: 100% (203/203) Updating files: 100% (203/203), done. 2024-05-27T00:30:04.047 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at c2d517ef966 v0.65 2024-05-27T00:30:04.053 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.65 2024-05-27T00:30:04.076 INFO:tasks.workunit.client.0.smithi098.stdout:v0.66 2024-05-27T00:30:04.077 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 66 -eq 48 ] 2024-05-27T00:30:04.077 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.66 2024-05-27T00:30:04.077 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.66 2024-05-27T00:30:04.077 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.66 2024-05-27T00:30:05.311 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at b6b48dbefad v0.66 2024-05-27T00:30:05.316 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.66 2024-05-27T00:30:05.407 INFO:tasks.workunit.client.0.smithi098.stdout:v0.67 2024-05-27T00:30:05.407 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 67 -eq 48 ] 2024-05-27T00:30:05.407 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.67 2024-05-27T00:30:05.407 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.67 2024-05-27T00:30:05.407 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.67 2024-05-27T00:30:07.480 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 63% (227/355) Updating files: 64% (228/355) Updating files: 65% (231/355) Updating files: 66% (235/355) Updating files: 67% (238/355) Updating files: 68% (242/355) Updating files: 69% (245/355) Updating files: 70% (249/355) Updating files: 71% (253/355) Updating files: 72% (256/355) Updating files: 73% (260/355) Updating files: 74% (263/355) Updating files: 75% (267/355) Updating files: 76% (270/355) Updating files: 77% (274/355) Updating files: 78% (277/355) Updating files: 79% (281/355) Updating files: 80% (284/355) Updating files: 81% (288/355) Updating files: 82% (292/355) Updating files: 83% (295/355) Updating files: 84% (299/355) Updating files: 85% (302/355) Updating files: 86% (306/355) Updating files: 87% (309/355) Updating files: 88% (313/355) Updating files: 89% (316/355) Updating files: 90% (320/355) Updating files: 91% (324/355) Updating files: 92% (327/355) Updating files: 93% (331/355) Updating files: 94% (334/355) Updating files: 95% (338/355) Updating files: 96% (341/355) Updating files: 97% (345/355) Updating files: 98% (348/355) Updating files: 99% (352/355) Updating files: 100% (355/355) Updating files: 100% (355/355), done. 2024-05-27T00:30:07.520 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at e3b7bc5bce8 v0.67 2024-05-27T00:30:07.537 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.67 2024-05-27T00:30:07.693 INFO:tasks.workunit.client.0.smithi098.stdout:v0.68 2024-05-27T00:30:07.693 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 68 -eq 48 ] 2024-05-27T00:30:07.693 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.68 2024-05-27T00:30:07.693 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.68 2024-05-27T00:30:07.694 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.68 2024-05-27T00:30:09.621 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 43% (106/241) Updating files: 44% (107/241) Updating files: 45% (109/241) Updating files: 46% (111/241) Updating files: 47% (114/241) Updating files: 48% (116/241) Updating files: 49% (119/241) Updating files: 50% (121/241) Updating files: 51% (123/241) Updating files: 52% (126/241) Updating files: 53% (128/241) Updating files: 54% (131/241) Updating files: 55% (133/241) Updating files: 56% (135/241) Updating files: 57% (138/241) Updating files: 58% (140/241) Updating files: 59% (143/241) Updating files: 60% (145/241) Updating files: 61% (148/241) Updating files: 62% (150/241) Updating files: 63% (152/241) Updating files: 64% (155/241) Updating files: 65% (157/241) Updating files: 66% (160/241) Updating files: 67% (162/241) Updating files: 68% (164/241) Updating files: 69% (167/241) Updating files: 70% (169/241) Updating files: 71% (172/241) Updating files: 72% (174/241) Updating files: 73% (176/241) Updating files: 74% (179/241) Updating files: 75% (181/241) Updating files: 76% (184/241) Updating files: 77% (186/241) Updating files: 78% (188/241) Updating files: 79% (191/241) Updating files: 80% (193/241) Updating files: 81% (196/241) Updating files: 82% (198/241) Updating files: 83% (201/241) Updating files: 84% (203/241) Updating files: 85% (205/241) Updating files: 86% (208/241) Updating files: 87% (210/241) Updating files: 88% (213/241) Updating files: 89% (215/241) Updating files: 90% (217/241) Updating files: 91% (220/241) Updating files: 92% (222/241) Updating files: 93% (225/241) Updating files: 94% (227/241) Updating files: 95% (229/241) Updating files: 96% (232/241) Updating files: 97% (234/241) Updating files: 98% (237/241) Updating files: 99% (239/241) Updating files: 100% (241/241) Updating files: 100% (241/241), done. 2024-05-27T00:30:09.699 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at b4cf0f2574b v0.68 2024-05-27T00:30:09.713 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.68 2024-05-27T00:30:09.764 INFO:tasks.workunit.client.0.smithi098.stdout:v0.69 2024-05-27T00:30:09.764 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 69 -eq 48 ] 2024-05-27T00:30:09.764 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.69 2024-05-27T00:30:09.764 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.69 2024-05-27T00:30:09.764 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.69 2024-05-27T00:30:23.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:30:23.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:23.616 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:30:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.792+0000 7f1a29a00640 1 Processor -- start 2024-05-27T00:30:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.792+0000 7f1a29a00640 1 -- start start 2024-05-27T00:30:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a29a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24070c00 0x7f1a240c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a29a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1a24070fe0 0x7f1a24154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a29a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2415a950 con 0x7f1a240c8f40 2024-05-27T00:30:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a29a00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f1a2415e060 con 0x7f1a24070fe0 2024-05-27T00:30:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a29a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1a2415e340 con 0x7f1a24070c00 2024-05-27T00:30:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a23400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1a240c8f40 0x7f1a2415ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60776/0 (socket says 172.21.15.98:60776) 2024-05-27T00:30:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a23400640 1 -- 172.21.15.98:0/312486730 learned_addr learned my addr 172.21.15.98:0/312486730 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:30:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a1be00640 1 --2- 172.21.15.98:0/312486730 >> v2:172.21.15.204:3300/0 conn(0x7f1a24070fe0 0x7f1a24154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a22a00640 1 --2- 172.21.15.98:0/312486730 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24070c00 0x7f1a240c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a1be00640 1 -- 172.21.15.98:0/312486730 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24070c00 msgr2=0x7f1a240c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a1be00640 1 --2- 172.21.15.98:0/312486730 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24070c00 0x7f1a240c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a1be00640 1 -- 172.21.15.98:0/312486730 >> v1:172.21.15.98:6789/0 conn(0x7f1a240c8f40 legacy=0x7f1a2415ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.796+0000 7f1a1be00640 1 -- 172.21.15.98:0/312486730 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2415e620 con 0x7f1a24070fe0 2024-05-27T00:30:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a1be00640 1 --2- 172.21.15.98:0/312486730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a24070fe0 0x7f1a24154780 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0097c0 tx=0x7f1a0c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52fdbe4755b8f2e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a22000640 1 -- 172.21.15.98:0/312486730 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c0b5070 con 0x7f1a24070fe0 2024-05-27T00:30:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a22000640 1 -- 172.21.15.98:0/312486730 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c0a8990 con 0x7f1a24070fe0 2024-05-27T00:30:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a22000640 1 -- 172.21.15.98:0/312486730 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c0a8d60 con 0x7f1a24070fe0 2024-05-27T00:30:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a29a00640 1 -- 172.21.15.98:0/312486730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a24070fe0 msgr2=0x7f1a24154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a29a00640 1 --2- 172.21.15.98:0/312486730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a24070fe0 0x7f1a24154780 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0097c0 tx=0x7f1a0c01ec90 comp rx=0 tx=0).stop 2024-05-27T00:30:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a29a00640 1 -- 172.21.15.98:0/312486730 shutdown_connections 2024-05-27T00:30:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a29a00640 1 --2- 172.21.15.98:0/312486730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a24070fe0 0x7f1a24154780 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a29a00640 1 --2- 172.21.15.98:0/312486730 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24070c00 0x7f1a240c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a29a00640 1 -- 172.21.15.98:0/312486730 >> 172.21.15.98:0/312486730 conn(0x7f1a24072f90 msgr2=0x7f1a240733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a29a00640 1 -- 172.21.15.98:0/312486730 shutdown_connections 2024-05-27T00:30:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.800+0000 7f1a29a00640 1 -- 172.21.15.98:0/312486730 wait complete. 2024-05-27T00:30:23.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a29a00640 1 Processor -- start 2024-05-27T00:30:23.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a29a00640 1 -- start start 2024-05-27T00:30:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a29a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a24070c00 0x7f1a241e5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a29a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24070fe0 0x7f1a241e4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:23.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a29a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a240c8f40 0x7f1a241e4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a29a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2415b560 con 0x7f1a24070fe0 2024-05-27T00:30:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a29a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2415ebe0 con 0x7f1a24070c00 2024-05-27T00:30:23.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a29a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1a2415b6e0 con 0x7f1a240c8f40 2024-05-27T00:30:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24070fe0 0x7f1a241e4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24070fe0 0x7f1a241e4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57948/0 (socket says 172.21.15.98:57948) 2024-05-27T00:30:23.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1be00640 1 -- 172.21.15.98:0/1176213378 learned_addr learned my addr 172.21.15.98:0/1176213378 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a22a00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a24070c00 0x7f1a241e5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a23400640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a240c8f40 0x7f1a241e4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1be00640 1 -- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a240c8f40 msgr2=0x7f1a241e4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1be00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a240c8f40 0x7f1a241e4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1be00640 1 -- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a24070c00 msgr2=0x7f1a241e5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1be00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a24070c00 0x7f1a241e5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1be00640 1 -- 172.21.15.98:0/1176213378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a241576e0 con 0x7f1a24070fe0 2024-05-27T00:30:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a23400640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a240c8f40 0x7f1a241e4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:30:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a22a00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a24070c00 0x7f1a241e5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:30:23.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1be00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24070fe0 0x7f1a241e4990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0b3040 tx=0x7f1a0c01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1aa00640 1 -- 172.21.15.98:0/1176213378 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c0bf070 con 0x7f1a24070fe0 2024-05-27T00:30:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1aa00640 1 -- 172.21.15.98:0/1176213378 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c0b07a0 con 0x7f1a24070fe0 2024-05-27T00:30:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a29a00640 1 -- 172.21.15.98:0/1176213378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a241579c0 con 0x7f1a24070fe0 2024-05-27T00:30:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.804+0000 7f1a1aa00640 1 -- 172.21.15.98:0/1176213378 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c0b5040 con 0x7f1a24070fe0 2024-05-27T00:30:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.808+0000 7f1a29a00640 1 -- 172.21.15.98:0/1176213378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a24157f80 con 0x7f1a24070fe0 2024-05-27T00:30:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.808+0000 7f1a13e00640 1 -- 172.21.15.98:0/1176213378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19f40052a0 con 0x7f1a24070fe0 2024-05-27T00:30:23.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.808+0000 7f1a1aa00640 1 -- 172.21.15.98:0/1176213378 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1a0c0c2070 con 0x7f1a24070fe0 2024-05-27T00:30:23.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.808+0000 7f1a1aa00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 0x7f1a00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:23.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.808+0000 7f1a1aa00640 1 -- 172.21.15.98:0/1176213378 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f1a0c1227b0 con 0x7f1a24070fe0 2024-05-27T00:30:23.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.808+0000 7f1a22a00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 0x7f1a00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:23.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.812+0000 7f1a22a00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 0x7f1a00069d80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009f50 tx=0x7f1a14009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:23.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:23.812+0000 7f1a1aa00640 1 -- 172.21.15.98:0/1176213378 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1a0c0187f0 con 0x7f1a24070fe0 2024-05-27T00:30:24.019 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 46% (72/155) Updating files: 47% (73/155) Updating files: 48% (75/155) Updating files: 49% (76/155) Updating files: 50% (78/155) Updating files: 51% (80/155) Updating files: 52% (81/155) Updating files: 53% (83/155) Updating files: 54% (84/155) Updating files: 55% (86/155) Updating files: 56% (87/155) Updating files: 57% (89/155) Updating files: 58% (90/155) Updating files: 59% (92/155) Updating files: 60% (93/155) Updating files: 61% (95/155) Updating files: 62% (97/155) Updating files: 63% (98/155) Updating files: 64% (100/155) Updating files: 65% (101/155) Updating files: 66% (103/155) Updating files: 67% (104/155) Updating files: 68% (106/155) Updating files: 69% (107/155) Updating files: 70% (109/155) Updating files: 71% (111/155) Updating files: 72% (112/155) Updating files: 73% (114/155) Updating files: 74% (115/155) Updating files: 75% (117/155) Updating files: 76% (118/155) Updating files: 77% (120/155) Updating files: 78% (121/155) Updating files: 79% (123/155) Updating files: 80% (124/155) Updating files: 81% (126/155) Updating files: 82% (128/155) Updating files: 83% (129/155) Updating files: 84% (131/155) Updating files: 85% (132/155) Updating files: 86% (134/155) Updating files: 87% (135/155) Updating files: 88% (137/155) Updating files: 89% (138/155) Updating files: 90% (140/155) Updating files: 91% (142/155) Updating files: 92% (143/155) Updating files: 93% (145/155) Updating files: 94% (146/155) Updating files: 95% (148/155) Updating files: 96% (149/155) Updating files: 97% (151/155) Updating files: 98% (152/155) Updating files: 99% (154/155) Updating files: 100% (155/155) Updating files: 100% (155/155), done. 2024-05-27T00:30:24.072 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 6ca6f2f9f75 v0.69 2024-05-27T00:30:24.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.068+0000 7f1a13e00640 1 -- 172.21.15.98:0/1176213378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19f4005590 con 0x7f1a24070fe0 2024-05-27T00:30:24.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a1aa00640 1 -- 172.21.15.98:0/1176213378 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6839 (secure 0 0 0) 0x7f1a0c0f28c0 con 0x7f1a24070fe0 2024-05-27T00:30:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:24.074 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:12.690345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6078,"qdb_cluster":[6078]},"id":1}]} 2024-05-27T00:30:24.074 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2024-05-27T00:30:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a13e00640 1 -- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 msgr2=0x7f1a00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a13e00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 0x7f1a00069d80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009f50 tx=0x7f1a14009190 comp rx=0 tx=0).stop 2024-05-27T00:30:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a13e00640 1 -- 172.21.15.98:0/1176213378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24070fe0 msgr2=0x7f1a241e4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a13e00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24070fe0 0x7f1a241e4990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0b3040 tx=0x7f1a0c01ba30 comp rx=0 tx=0).stop 2024-05-27T00:30:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a13e00640 1 -- 172.21.15.98:0/1176213378 shutdown_connections 2024-05-27T00:30:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a13e00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a240c8f40 0x7f1a241e4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a13e00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a24070fe0 0x7f1a241e4990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a13e00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 0x7f1a00069d80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a13e00640 1 --2- 172.21.15.98:0/1176213378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a24070c00 0x7f1a241e5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.072+0000 7f1a13e00640 1 -- 172.21.15.98:0/1176213378 >> 172.21.15.98:0/1176213378 conn(0x7f1a24072f90 msgr2=0x7f1a2415ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.076+0000 7f1a13e00640 1 -- 172.21.15.98:0/1176213378 shutdown_connections 2024-05-27T00:30:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.076+0000 7f1a13e00640 1 -- 172.21.15.98:0/1176213378 wait complete. 2024-05-27T00:30:24.080 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.69 2024-05-27T00:30:24.098 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-05-27T00:30:24.099 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-27T00:30:24.099 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:30:24.181 INFO:tasks.ceph.mds.c:Stopped 2024-05-27T00:30:24.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:24.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.360+0000 7efdb9800640 1 Processor -- start 2024-05-27T00:30:24.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.360+0000 7efdb9800640 1 -- start start 2024-05-27T00:30:24.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.360+0000 7efdb9800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb40cc1f0 0x7efdb41592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.360+0000 7efdb9800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdb40c1f60 con 0x7efdb40cbe10 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.360+0000 7efdb9800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdb40c6dd0 con 0x7efdb4073570 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.360+0000 7efdb9800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efdb4159810 con 0x7efdb40cc1f0 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.360+0000 7efdb2a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efdb40cbe10 0x7efdb40c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60788/0 (socket says 172.21.15.98:60788) 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.360+0000 7efdb2a00640 1 -- 172.21.15.98:0/1077905340 learned_addr learned my addr 172.21.15.98:0/1077905340 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb2000640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb40cc1f0 0x7efdb41592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346463924 0 0) 0x7efdb40c1f60 con 0x7efdb40cbe10 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd98003530 con 0x7efdb40cbe10 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3289851601 0 0) 0x7efd98003530 con 0x7efdb40cbe10 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb40cc1f0 msgr2=0x7efdb41592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb40cc1f0 0x7efdb41592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 >> v1:172.21.15.204:6789/0 conn(0x7efdb4073570 legacy=0x7efdb41559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb40c70b0 con 0x7efdb40cbe10 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7efda4001a00 con 0x7efdb40cbe10 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 >> v1:172.21.15.98:6789/0 conn(0x7efdb40cbe10 legacy=0x7efdb40c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd980039f0 0x7efd98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb40cc1f0 0x7efd98004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd98004d50 0x7efd980051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdb4159810 con 0x7efd98004d50 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd9800ba00 con 0x7efd980039f0 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efd9800bc00 con 0x7efdb40cc1f0 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb3400640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd98004d50 0x7efd980051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb2a00640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd980039f0 0x7efd98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb2000640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb40cc1f0 0x7efd98004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:24.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb3400640 1 -- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb40cc1f0 msgr2=0x7efd98004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:24.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb3400640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb40cc1f0 0x7efd98004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb3400640 1 -- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd980039f0 msgr2=0x7efd98003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:24.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb3400640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd980039f0 0x7efd98003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.364+0000 7efdb3400640 1 -- 172.21.15.98:0/1077905340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd98003530 con 0x7efd98004d50 2024-05-27T00:30:24.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb3400640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd98004d50 0x7efd980051d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7efdb40b1b60 tx=0x7efdac01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc5ac010e2ca37 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:24.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac001460 con 0x7efd98004d50 2024-05-27T00:30:24.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdac0b0070 con 0x7efd98004d50 2024-05-27T00:30:24.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb1600640 1 -- 172.21.15.98:0/1077905340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac0ab840 con 0x7efd98004d50 2024-05-27T00:30:24.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 -- 172.21.15.98:0/1077905340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd98004d50 msgr2=0x7efd980051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:24.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 --2- 172.21.15.98:0/1077905340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd98004d50 0x7efd980051d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7efdb40b1b60 tx=0x7efdac01c4c0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb3400640 1 -- 172.21.15.98:0/1077905340 reap_dead start 2024-05-27T00:30:24.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 -- 172.21.15.98:0/1077905340 shutdown_connections 2024-05-27T00:30:24.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 -- 172.21.15.98:0/1077905340 >> 172.21.15.98:0/1077905340 conn(0x7efdb4067530 msgr2=0x7efdb406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:24.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 -- 172.21.15.98:0/1077905340 shutdown_connections 2024-05-27T00:30:24.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 -- 172.21.15.98:0/1077905340 wait complete. 2024-05-27T00:30:24.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 Processor -- start 2024-05-27T00:30:24.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 -- start start 2024-05-27T00:30:24.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd98004d50 0x7efdb41f00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:24.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb41f1530 0x7efdb41ded40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:24.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb4157b20 0x7efdb40720c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:24.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdb40c1330 con 0x7efdb41f1530 2024-05-27T00:30:24.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efdb40c1050 con 0x7efd98004d50 2024-05-27T00:30:24.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb9800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efdb40c79d0 con 0x7efdb4157b20 2024-05-27T00:30:24.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.368+0000 7efdb2000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb41f1530 0x7efdb41ded40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:24.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb2000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb41f1530 0x7efdb41ded40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57972/0 (socket says 172.21.15.98:57972) 2024-05-27T00:30:24.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb2000640 1 -- 172.21.15.98:0/3122987385 learned_addr learned my addr 172.21.15.98:0/3122987385 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:24.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb2a00640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd98004d50 0x7efdb41f00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:24.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb3400640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb4157b20 0x7efdb40720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:24.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb2000640 1 -- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb4157b20 msgr2=0x7efdb40720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:24.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb2000640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb4157b20 0x7efdb40720c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb2000640 1 -- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd98004d50 msgr2=0x7efdb41f00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:24.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb2000640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd98004d50 0x7efdb41f00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb2000640 1 -- 172.21.15.98:0/3122987385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb4156820 con 0x7efdb41f1530 2024-05-27T00:30:24.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb2000640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb41f1530 0x7efdb41ded40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7efda0011a00 tx=0x7efda0011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdaaa00640 1 -- 172.21.15.98:0/3122987385 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda000fa90 con 0x7efdb41f1530 2024-05-27T00:30:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdaaa00640 1 -- 172.21.15.98:0/3122987385 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efda000fc30 con 0x7efdb41f1530 2024-05-27T00:30:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdaaa00640 1 -- 172.21.15.98:0/3122987385 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda0014440 con 0x7efdb41f1530 2024-05-27T00:30:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb9800640 1 -- 172.21.15.98:0/3122987385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb4156ac0 con 0x7efdb41f1530 2024-05-27T00:30:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb9800640 1 -- 172.21.15.98:0/3122987385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb4157040 con 0x7efdb41f1530 2024-05-27T00:30:24.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdb9800640 1 -- 172.21.15.98:0/3122987385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdb4075cb0 con 0x7efdb41f1530 2024-05-27T00:30:24.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.372+0000 7efdaaa00640 1 -- 172.21.15.98:0/3122987385 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efda0012040 con 0x7efdb41f1530 2024-05-27T00:30:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.376+0000 7efdaaa00640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd940678f0 0x7efd94069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.376+0000 7efdaaa00640 1 -- 172.21.15.98:0/3122987385 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7efda00865f0 con 0x7efdb41f1530 2024-05-27T00:30:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.376+0000 7efdb2a00640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd940678f0 0x7efd94069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:24.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.376+0000 7efdb2a00640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd940678f0 0x7efd94069db0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efda40013e0 tx=0x7efda4006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:24.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.380+0000 7efdaaa00640 1 -- 172.21.15.98:0/3122987385 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7efda0019030 con 0x7efdb41f1530 2024-05-27T00:30:24.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.568+0000 7efdb9800640 1 -- 172.21.15.98:0/3122987385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efdb40cb970 con 0x7efdb41f1530 2024-05-27T00:30:24.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdaaa00640 1 -- 172.21.15.98:0/3122987385 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6839 (secure 0 0 0) 0x7efda0056670 con 0x7efdb41f1530 2024-05-27T00:30:24.574 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:24.574 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:12.690345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6078,"qdb_cluster":[6078]},"id":1}]} 2024-05-27T00:30:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2024-05-27T00:30:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 -- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd940678f0 msgr2=0x7efd94069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd940678f0 0x7efd94069db0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efda40013e0 tx=0x7efda4006040 comp rx=0 tx=0).stop 2024-05-27T00:30:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 -- 172.21.15.98:0/3122987385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb41f1530 msgr2=0x7efdb41ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:24.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb41f1530 0x7efdb41ded40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7efda0011a00 tx=0x7efda0011ec0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 -- 172.21.15.98:0/3122987385 shutdown_connections 2024-05-27T00:30:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdb4157b20 0x7efdb40720c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdb41f1530 0x7efdb41ded40 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7efd940678f0 0x7efd94069db0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 --2- 172.21.15.98:0/3122987385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd98004d50 0x7efdb41f00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 -- 172.21.15.98:0/3122987385 >> 172.21.15.98:0/3122987385 conn(0x7efdb4067530 msgr2=0x7efdb40b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 -- 172.21.15.98:0/3122987385 shutdown_connections 2024-05-27T00:30:24.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:24.572+0000 7efdb9800640 1 -- 172.21.15.98:0/3122987385 wait complete. 2024-05-27T00:30:24.595 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:30:26.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:26.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445a00640 1 Processor -- start 2024-05-27T00:30:26.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445a00640 1 -- start start 2024-05-27T00:30:26.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f844015bb00 0x7f84401523c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:26.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84400cb060 0x7f8440152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:26.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f844015bf90 con 0x7f84400cac80 2024-05-27T00:30:26.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f84400c84f0 con 0x7f84400cb060 2024-05-27T00:30:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f84400c8990 con 0x7f844015bb00 2024-05-27T00:30:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84400cb060 0x7f8440152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84400cb060 0x7f8440152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57986/0 (socket says 172.21.15.98:57986) 2024-05-27T00:30:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445000640 1 -- 172.21.15.98:0/3975467328 learned_addr learned my addr 172.21.15.98:0/3975467328 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8437400640 1 --2- 172.21.15.98:0/3975467328 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f844015bb00 0x7f84401523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445000640 1 -- 172.21.15.98:0/3975467328 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f844015bb00 msgr2=0x7f84401523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445000640 1 --2- 172.21.15.98:0/3975467328 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f844015bb00 0x7f84401523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445000640 1 -- 172.21.15.98:0/3975467328 >> v1:172.21.15.204:6789/0 conn(0x7f84400cac80 legacy=0x7f84400c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.740+0000 7f8445000640 1 -- 172.21.15.98:0/3975467328 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844015c270 con 0x7f84400cb060 2024-05-27T00:30:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f8445000640 1 --2- 172.21.15.98:0/3975467328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cb060 0x7f8440152900 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8438009870 tx=0x7f843801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cea84a25daae461 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:26.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f843f400640 1 -- 172.21.15.98:0/3975467328 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84380b1070 con 0x7f84400cb060 2024-05-27T00:30:26.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f843f400640 1 -- 172.21.15.98:0/3975467328 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84380ac630 con 0x7f84400cb060 2024-05-27T00:30:26.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f843f400640 1 -- 172.21.15.98:0/3975467328 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84380a8050 con 0x7f84400cb060 2024-05-27T00:30:26.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f8445a00640 1 -- 172.21.15.98:0/3975467328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cb060 msgr2=0x7f8440152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:26.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f8445a00640 1 --2- 172.21.15.98:0/3975467328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cb060 0x7f8440152900 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8438009870 tx=0x7f843801c520 comp rx=0 tx=0).stop 2024-05-27T00:30:26.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f8445a00640 1 -- 172.21.15.98:0/3975467328 shutdown_connections 2024-05-27T00:30:26.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f8445a00640 1 --2- 172.21.15.98:0/3975467328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cb060 0x7f8440152900 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:26.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f8445a00640 1 --2- 172.21.15.98:0/3975467328 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f844015bb00 0x7f84401523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:26.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f8445a00640 1 -- 172.21.15.98:0/3975467328 >> 172.21.15.98:0/3975467328 conn(0x7f8440070b30 msgr2=0x7f8440070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:26.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f8445a00640 1 -- 172.21.15.98:0/3975467328 shutdown_connections 2024-05-27T00:30:26.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.744+0000 7f8445a00640 1 -- 172.21.15.98:0/3975467328 wait complete. 2024-05-27T00:30:26.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445a00640 1 Processor -- start 2024-05-27T00:30:26.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445a00640 1 -- start start 2024-05-27T00:30:26.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cac80 0x7f84401533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:26.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84400cb060 0x7f84401f5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:26.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84401f2760 0x7f844014da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:26.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84400c9810 con 0x7f84400cac80 2024-05-27T00:30:26.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f84400b40f0 con 0x7f84401f2760 2024-05-27T00:30:26.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8440158590 con 0x7f84400cb060 2024-05-27T00:30:26.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cac80 0x7f84401533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:26.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cac80 0x7f84401533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57990/0 (socket says 172.21.15.98:57990) 2024-05-27T00:30:26.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843fe00640 1 -- 172.21.15.98:0/4160984281 learned_addr learned my addr 172.21.15.98:0/4160984281 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:26.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8437400640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84400cb060 0x7f84401f5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:26.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445000640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84401f2760 0x7f844014da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:26.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843fe00640 1 -- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84400cb060 msgr2=0x7f84401f5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:26.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843fe00640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84400cb060 0x7f84401f5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:26.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843fe00640 1 -- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84401f2760 msgr2=0x7f844014da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:26.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843fe00640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84401f2760 0x7f844014da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:26.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843fe00640 1 -- 172.21.15.98:0/4160984281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844015b0a0 con 0x7f84400cac80 2024-05-27T00:30:26.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843fe00640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cac80 0x7f84401533f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f843000c7f0 tx=0x7f843000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:26.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843cc00640 1 -- 172.21.15.98:0/4160984281 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8430016020 con 0x7f84400cac80 2024-05-27T00:30:26.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843cc00640 1 -- 172.21.15.98:0/4160984281 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f843000aa50 con 0x7f84400cac80 2024-05-27T00:30:26.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f843cc00640 1 -- 172.21.15.98:0/4160984281 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843000ae40 con 0x7f84400cac80 2024-05-27T00:30:26.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445a00640 1 -- 172.21.15.98:0/4160984281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84401f1cc0 con 0x7f84400cac80 2024-05-27T00:30:26.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.748+0000 7f8445a00640 1 -- 172.21.15.98:0/4160984281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f844015aae0 con 0x7f84400cac80 2024-05-27T00:30:26.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.752+0000 7f8445a00640 1 -- 172.21.15.98:0/4160984281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8440075c40 con 0x7f84400cac80 2024-05-27T00:30:26.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.752+0000 7f843cc00640 1 -- 172.21.15.98:0/4160984281 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8430019020 con 0x7f84400cac80 2024-05-27T00:30:26.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.752+0000 7f843cc00640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f841c0678f0 0x7f841c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:26.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.752+0000 7f843cc00640 1 -- 172.21.15.98:0/4160984281 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f8430086150 con 0x7f84400cac80 2024-05-27T00:30:26.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.752+0000 7f8437400640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f841c0678f0 0x7f841c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:26.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.752+0000 7f8437400640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f841c0678f0 0x7f841c069db0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8440152930 tx=0x7f8428008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:26.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.756+0000 7f843cc00640 1 -- 172.21.15.98:0/4160984281 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8430020030 con 0x7f84400cac80 2024-05-27T00:30:26.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.944+0000 7f8445a00640 1 -- 172.21.15.98:0/4160984281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8440152f50 con 0x7f84400cac80 2024-05-27T00:30:26.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.944+0000 7f843cc00640 1 -- 172.21.15.98:0/4160984281 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6839 (secure 0 0 0) 0x7f84300568b0 con 0x7f84400cac80 2024-05-27T00:30:26.948 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:26.948 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:12.690345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6078,"qdb_cluster":[6078]},"id":1}]} 2024-05-27T00:30:26.948 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2024-05-27T00:30:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 -- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f841c0678f0 msgr2=0x7f841c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f841c0678f0 0x7f841c069db0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8440152930 tx=0x7f8428008040 comp rx=0 tx=0).stop 2024-05-27T00:30:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 -- 172.21.15.98:0/4160984281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cac80 msgr2=0x7f84401533f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cac80 0x7f84401533f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f843000c7f0 tx=0x7f843000ccb0 comp rx=0 tx=0).stop 2024-05-27T00:30:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 -- 172.21.15.98:0/4160984281 shutdown_connections 2024-05-27T00:30:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f841c0678f0 0x7f841c069db0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84401f2760 0x7f844014da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84400cb060 0x7f84401f5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 --2- 172.21.15.98:0/4160984281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84400cac80 0x7f84401533f0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 -- 172.21.15.98:0/4160984281 >> 172.21.15.98:0/4160984281 conn(0x7f8440070b30 msgr2=0x7f84401577d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 -- 172.21.15.98:0/4160984281 shutdown_connections 2024-05-27T00:30:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:26.948+0000 7f8445a00640 1 -- 172.21.15.98:0/4160984281 wait complete. 2024-05-27T00:30:26.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:30:28.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:29.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78fd000640 1 Processor -- start 2024-05-27T00:30:29.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78fd000640 1 -- start start 2024-05-27T00:30:29.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78fd000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f78f80c79f0 0x7f78f815bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78fd000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78f815c430 0x7f78f8152380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78fd000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f815c880 con 0x7f78f8072f30 2024-05-27T00:30:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78fd000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f78f806e4e0 con 0x7f78f80c79f0 2024-05-27T00:30:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78fd000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f78f8073a20 con 0x7f78f815c430 2024-05-27T00:30:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78f5400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f78f80c79f0 0x7f78f815bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78f5400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f78f80c79f0 0x7f78f815bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57996/0 (socket says 172.21.15.98:57996) 2024-05-27T00:30:29.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78f5400640 1 -- 172.21.15.98:0/746638043 learned_addr learned my addr 172.21.15.98:0/746638043 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78f6800640 1 --2- 172.21.15.98:0/746638043 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78f815c430 0x7f78f8152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78f5400640 1 -- 172.21.15.98:0/746638043 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78f815c430 msgr2=0x7f78f8152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78f5400640 1 --2- 172.21.15.98:0/746638043 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78f815c430 0x7f78f8152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78f5400640 1 -- 172.21.15.98:0/746638043 >> v1:172.21.15.98:6789/0 conn(0x7f78f8072f30 legacy=0x7f78f8073310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:29.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.116+0000 7f78f5400640 1 -- 172.21.15.98:0/746638043 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f8159270 con 0x7f78f80c79f0 2024-05-27T00:30:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78f5400640 1 --2- 172.21.15.98:0/746638043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f80c79f0 0x7f78f815bef0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f78e40098a0 tx=0x7f78e401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12febf45d984715f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78f4a00640 1 -- 172.21.15.98:0/746638043 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e40b5070 con 0x7f78f80c79f0 2024-05-27T00:30:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78f4a00640 1 -- 172.21.15.98:0/746638043 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e40a8a30 con 0x7f78f80c79f0 2024-05-27T00:30:29.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78f4a00640 1 -- 172.21.15.98:0/746638043 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e40a8e00 con 0x7f78f80c79f0 2024-05-27T00:30:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78fd000640 1 -- 172.21.15.98:0/746638043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f80c79f0 msgr2=0x7f78f815bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78fd000640 1 --2- 172.21.15.98:0/746638043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f80c79f0 0x7f78f815bef0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f78e40098a0 tx=0x7f78e401ed30 comp rx=0 tx=0).stop 2024-05-27T00:30:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78fd000640 1 -- 172.21.15.98:0/746638043 shutdown_connections 2024-05-27T00:30:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78fd000640 1 --2- 172.21.15.98:0/746638043 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78f815c430 0x7f78f8152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78fd000640 1 --2- 172.21.15.98:0/746638043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f80c79f0 0x7f78f815bef0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:29.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78fd000640 1 -- 172.21.15.98:0/746638043 >> 172.21.15.98:0/746638043 conn(0x7f78f80c9af0 msgr2=0x7f78f80c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78fd000640 1 -- 172.21.15.98:0/746638043 shutdown_connections 2024-05-27T00:30:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78fd000640 1 -- 172.21.15.98:0/746638043 wait complete. 2024-05-27T00:30:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78fd000640 1 Processor -- start 2024-05-27T00:30:29.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.120+0000 7f78fd000640 1 -- start start 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78fd000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f8072f30 0x7f78f80af5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78fd000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f80c79f0 0x7f78f81563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78fd000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78f815c430 0x7f78f815af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78fd000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78f80c9250 con 0x7f78f8072f30 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78fd000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78f806e690 con 0x7f78f80c79f0 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78fd000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f78f80c94a0 con 0x7f78f815c430 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f5e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f8072f30 0x7f78f80af5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f5e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f8072f30 0x7f78f80af5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58004/0 (socket says 172.21.15.98:58004) 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f5e00640 1 -- 172.21.15.98:0/2670688348 learned_addr learned my addr 172.21.15.98:0/2670688348 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f6800640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78f815c430 0x7f78f815af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:29.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f5400640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f80c79f0 0x7f78f81563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f5e00640 1 -- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78f815c430 msgr2=0x7f78f815af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f5e00640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78f815c430 0x7f78f815af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f5e00640 1 -- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f80c79f0 msgr2=0x7f78f81563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f5e00640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f80c79f0 0x7f78f81563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f5e00640 1 -- 172.21.15.98:0/2670688348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f81568f0 con 0x7f78f8072f30 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78f5e00640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f8072f30 0x7f78f80af5f0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f78e800c420 tx=0x7f78e800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78ee000640 1 -- 172.21.15.98:0/2670688348 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e80052d0 con 0x7f78f8072f30 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78ee000640 1 -- 172.21.15.98:0/2670688348 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e8005470 con 0x7f78f8072f30 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78ee000640 1 -- 172.21.15.98:0/2670688348 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8007260 con 0x7f78f8072f30 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78fd000640 1 -- 172.21.15.98:0/2670688348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f8159b60 con 0x7f78f8072f30 2024-05-27T00:30:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78fd000640 1 -- 172.21.15.98:0/2670688348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f80aedf0 con 0x7f78f8072f30 2024-05-27T00:30:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78fd000640 1 -- 172.21.15.98:0/2670688348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f8075cb0 con 0x7f78f8072f30 2024-05-27T00:30:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.124+0000 7f78ee000640 1 -- 172.21.15.98:0/2670688348 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f78e8007400 con 0x7f78f8072f30 2024-05-27T00:30:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.128+0000 7f78ee000640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78d80678c0 0x7f78d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.128+0000 7f78ee000640 1 -- 172.21.15.98:0/2670688348 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f78e8085570 con 0x7f78f8072f30 2024-05-27T00:30:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.128+0000 7f78f5400640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78d80678c0 0x7f78d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:29.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.128+0000 7f78f5400640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78d80678c0 0x7f78d8069d80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f78e40b2000 tx=0x7f78e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:29.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.132+0000 7f78ee000640 1 -- 172.21.15.98:0/2670688348 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f78e8055cd0 con 0x7f78f8072f30 2024-05-27T00:30:29.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.320+0000 7f78fd000640 1 -- 172.21.15.98:0/2670688348 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78f80725f0 con 0x7f78f8072f30 2024-05-27T00:30:29.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.320+0000 7f78ee000640 1 -- 172.21.15.98:0/2670688348 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6839 (secure 0 0 0) 0x7f78e80555d0 con 0x7f78f8072f30 2024-05-27T00:30:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:12.690345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6078,"qdb_cluster":[6078]},"id":1}]} 2024-05-27T00:30:29.325 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 -- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78d80678c0 msgr2=0x7f78d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78d80678c0 0x7f78d8069d80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f78e40b2000 tx=0x7f78e4008040 comp rx=0 tx=0).stop 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 -- 172.21.15.98:0/2670688348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f8072f30 msgr2=0x7f78f80af5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f8072f30 0x7f78f80af5f0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f78e800c420 tx=0x7f78e800c8e0 comp rx=0 tx=0).stop 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 -- 172.21.15.98:0/2670688348 shutdown_connections 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78f815c430 0x7f78f815af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f78d80678c0 0x7f78d8069d80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f80c79f0 0x7f78f81563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 --2- 172.21.15.98:0/2670688348 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78f8072f30 0x7f78f80af5f0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 -- 172.21.15.98:0/2670688348 >> 172.21.15.98:0/2670688348 conn(0x7f78f80c9af0 msgr2=0x7f78f8073310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:29.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 -- 172.21.15.98:0/2670688348 shutdown_connections 2024-05-27T00:30:29.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:29.324+0000 7f78fd000640 1 -- 172.21.15.98:0/2670688348 wait complete. 2024-05-27T00:30:29.346 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:30:31.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5af2e00640 1 Processor -- start 2024-05-27T00:30:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5af2e00640 1 -- start start 2024-05-27T00:30:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5af2e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5aec0cc1f0 0x7f5aec1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5af2e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5aec073570 0x7f5aec1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5af2e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aec0c1f60 con 0x7f5aec0cbe10 2024-05-27T00:30:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5af2e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5aec159810 con 0x7f5aec073570 2024-05-27T00:30:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5af2e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec0c6dd0 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5af0e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5aec073570 0x7f5aec1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5af0e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5aec073570 0x7f5aec1559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58014/0 (socket says 172.21.15.98:58014) 2024-05-27T00:30:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5aebe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5aec0cbe10 0x7f5aec0c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60800/0 (socket says 172.21.15.98:60800) 2024-05-27T00:30:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.492+0000 7f5af0e00640 1 -- 172.21.15.98:0/958899432 learned_addr learned my addr 172.21.15.98:0/958899432 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5aeb400640 1 --2- 172.21.15.98:0/958899432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5aec0cc1f0 0x7f5aec1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5af0e00640 1 -- 172.21.15.98:0/958899432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5aec0cc1f0 msgr2=0x7f5aec1592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5af0e00640 1 --2- 172.21.15.98:0/958899432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5aec0cc1f0 0x7f5aec1592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5af0e00640 1 -- 172.21.15.98:0/958899432 >> v1:172.21.15.98:6789/0 conn(0x7f5aec0cbe10 legacy=0x7f5aec0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5af0e00640 1 -- 172.21.15.98:0/958899432 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec0c70b0 con 0x7f5aec073570 2024-05-27T00:30:31.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5af0e00640 1 --2- 172.21.15.98:0/958899432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec073570 0x7f5aec1559e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009870 tx=0x7f5ae401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150899eafb401628 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:31.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5aeaa00640 1 -- 172.21.15.98:0/958899432 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40b5070 con 0x7f5aec073570 2024-05-27T00:30:31.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5aeaa00640 1 -- 172.21.15.98:0/958899432 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae40a8a50 con 0x7f5aec073570 2024-05-27T00:30:31.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5aeaa00640 1 -- 172.21.15.98:0/958899432 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40a8e20 con 0x7f5aec073570 2024-05-27T00:30:31.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5af2e00640 1 -- 172.21.15.98:0/958899432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec073570 msgr2=0x7f5aec1559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:31.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.496+0000 7f5af2e00640 1 --2- 172.21.15.98:0/958899432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec073570 0x7f5aec1559e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009870 tx=0x7f5ae401ed50 comp rx=0 tx=0).stop 2024-05-27T00:30:31.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 -- 172.21.15.98:0/958899432 shutdown_connections 2024-05-27T00:30:31.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 --2- 172.21.15.98:0/958899432 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec073570 0x7f5aec1559e0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:31.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 --2- 172.21.15.98:0/958899432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5aec0cc1f0 0x7f5aec1592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:31.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 -- 172.21.15.98:0/958899432 >> 172.21.15.98:0/958899432 conn(0x7f5aec067530 msgr2=0x7f5aec06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:31.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 -- 172.21.15.98:0/958899432 shutdown_connections 2024-05-27T00:30:31.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 -- 172.21.15.98:0/958899432 wait complete. 2024-05-27T00:30:31.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 Processor -- start 2024-05-27T00:30:31.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 -- start start 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5aec073570 0x7f5aec158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5aec0cbe10 0x7f5aec148070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec0cc1f0 0x7f5aec151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec0c1330 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec0c1050 con 0x7f5aec073570 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec0c79d0 con 0x7f5aec0cbe10 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af0e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec0cc1f0 0x7f5aec151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af0e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec0cc1f0 0x7f5aec151880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58030/0 (socket says 172.21.15.98:58030) 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af0e00640 1 -- 172.21.15.98:0/1579178785 learned_addr learned my addr 172.21.15.98:0/1579178785 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5aeb400640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5aec0cbe10 0x7f5aec148070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:31.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5aebe00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5aec073570 0x7f5aec158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af0e00640 1 -- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5aec0cbe10 msgr2=0x7f5aec148070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af0e00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5aec0cbe10 0x7f5aec148070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af0e00640 1 -- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5aec073570 msgr2=0x7f5aec158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af0e00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5aec073570 0x7f5aec158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af0e00640 1 -- 172.21.15.98:0/1579178785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec0c4c00 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af0e00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec0cc1f0 0x7f5aec151880 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40b3040 tx=0x7f5ae401baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5ae3e00640 1 -- 172.21.15.98:0/1579178785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40c0070 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5ae3e00640 1 -- 172.21.15.98:0/1579178785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae40ae550 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.500+0000 7f5af2e00640 1 -- 172.21.15.98:0/1579178785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aec0c4500 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.504+0000 7f5ae3e00640 1 -- 172.21.15.98:0/1579178785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40b5070 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.504+0000 7f5af2e00640 1 -- 172.21.15.98:0/1579178785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aec1f2370 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.504+0000 7f5af2e00640 1 -- 172.21.15.98:0/1579178785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aec1566a0 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.504+0000 7f5ae3e00640 1 -- 172.21.15.98:0/1579178785 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5ae40c3070 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.504+0000 7f5ae3e00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5acc0678c0 0x7f5acc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:31.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.504+0000 7f5ae3e00640 1 -- 172.21.15.98:0/1579178785 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f5ae41238c0 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.504+0000 7f5aebe00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5acc0678c0 0x7f5acc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:31.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.508+0000 7f5aebe00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5acc0678c0 0x7f5acc069d80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5adc007fd0 tx=0x7f5adc005b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:31.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.508+0000 7f5ae3e00640 1 -- 172.21.15.98:0/1579178785 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f5ae40188b0 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.712+0000 7f5af2e00640 1 -- 172.21.15.98:0/1579178785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5aec075cb0 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.712+0000 7f5ae3e00640 1 -- 172.21.15.98:0/1579178785 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6839 (secure 0 0 0) 0x7f5ae40f4030 con 0x7f5aec0cc1f0 2024-05-27T00:30:31.717 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:31.717 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:12.690345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6078,"qdb_cluster":[6078]},"id":1}]} 2024-05-27T00:30:31.717 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2024-05-27T00:30:31.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 -- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5acc0678c0 msgr2=0x7f5acc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:31.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5acc0678c0 0x7f5acc069d80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5adc007fd0 tx=0x7f5adc005b80 comp rx=0 tx=0).stop 2024-05-27T00:30:31.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 -- 172.21.15.98:0/1579178785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec0cc1f0 msgr2=0x7f5aec151880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:31.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec0cc1f0 0x7f5aec151880 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40b3040 tx=0x7f5ae401baf0 comp rx=0 tx=0).stop 2024-05-27T00:30:31.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 -- 172.21.15.98:0/1579178785 shutdown_connections 2024-05-27T00:30:31.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5aec0cc1f0 0x7f5aec151880 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:31.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5aec0cbe10 0x7f5aec148070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:31.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f5acc0678c0 0x7f5acc069d80 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:31.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 --2- 172.21.15.98:0/1579178785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5aec073570 0x7f5aec158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:31.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 -- 172.21.15.98:0/1579178785 >> 172.21.15.98:0/1579178785 conn(0x7f5aec067530 msgr2=0x7f5aec0c8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:31.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 -- 172.21.15.98:0/1579178785 shutdown_connections 2024-05-27T00:30:31.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:31.716+0000 7f5af2e00640 1 -- 172.21.15.98:0/1579178785 wait complete. 2024-05-27T00:30:31.738 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:30:33.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb2da00640 1 Processor -- start 2024-05-27T00:30:33.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb2da00640 1 -- start start 2024-05-27T00:30:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb2da00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070c00 0x7ffb280c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb2da00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7ffb28070fe0 0x7ffb28154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb2da00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb2815a950 con 0x7ffb280c8f40 2024-05-27T00:30:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb2da00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7ffb2815e060 con 0x7ffb28070fe0 2024-05-27T00:30:33.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb2da00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffb2815e340 con 0x7ffb28070c00 2024-05-27T00:30:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb26000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7ffb28070fe0 0x7ffb28154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb26a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070c00 0x7ffb280c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb26000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7ffb28070fe0 0x7ffb28154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:52884/0 (socket says 172.21.15.98:52884) 2024-05-27T00:30:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb26000640 1 -- 172.21.15.98:0/4132999569 learned_addr learned my addr 172.21.15.98:0/4132999569 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb27400640 1 --1- 172.21.15.98:0/4132999569 >> v1:172.21.15.204:6789/0 conn(0x7ffb280c8f40 0x7ffb2815ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:57272/0 (socket says 172.21.15.98:57272) 2024-05-27T00:30:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb26a00640 1 -- 172.21.15.98:0/4132999569 >> v2:172.21.15.204:3300/0 conn(0x7ffb28070fe0 msgr2=0x7ffb28154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb26a00640 1 --2- 172.21.15.98:0/4132999569 >> v2:172.21.15.204:3300/0 conn(0x7ffb28070fe0 0x7ffb28154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb26a00640 1 -- 172.21.15.98:0/4132999569 >> v1:172.21.15.204:6789/0 conn(0x7ffb280c8f40 legacy=0x7ffb2815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:33.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.884+0000 7ffb26a00640 1 -- 172.21.15.98:0/4132999569 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2815e620 con 0x7ffb28070c00 2024-05-27T00:30:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb26a00640 1 --2- 172.21.15.98:0/4132999569 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070c00 0x7ffb280c8a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb180097c0 tx=0x7ffb1801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c3caff85797b802 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb25600640 1 -- 172.21.15.98:0/4132999569 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb180b5070 con 0x7ffb28070c00 2024-05-27T00:30:33.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb25600640 1 -- 172.21.15.98:0/4132999569 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb180a8990 con 0x7ffb28070c00 2024-05-27T00:30:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb25600640 1 -- 172.21.15.98:0/4132999569 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb180a8d60 con 0x7ffb28070c00 2024-05-27T00:30:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb2da00640 1 -- 172.21.15.98:0/4132999569 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070c00 msgr2=0x7ffb280c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb2da00640 1 --2- 172.21.15.98:0/4132999569 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070c00 0x7ffb280c8a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb180097c0 tx=0x7ffb1801ec90 comp rx=0 tx=0).stop 2024-05-27T00:30:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb2da00640 1 -- 172.21.15.98:0/4132999569 shutdown_connections 2024-05-27T00:30:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb2da00640 1 --2- 172.21.15.98:0/4132999569 >> v2:172.21.15.204:3300/0 conn(0x7ffb28070fe0 0x7ffb28154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:33.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb2da00640 1 --2- 172.21.15.98:0/4132999569 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070c00 0x7ffb280c8a00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:33.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb2da00640 1 -- 172.21.15.98:0/4132999569 >> 172.21.15.98:0/4132999569 conn(0x7ffb28072f90 msgr2=0x7ffb280733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:33.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb2da00640 1 -- 172.21.15.98:0/4132999569 shutdown_connections 2024-05-27T00:30:33.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.888+0000 7ffb2da00640 1 -- 172.21.15.98:0/4132999569 wait complete. 2024-05-27T00:30:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb2da00640 1 Processor -- start 2024-05-27T00:30:33.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb2da00640 1 -- start start 2024-05-27T00:30:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb2da00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb28070c00 0x7ffb281f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb2da00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070fe0 0x7ffb28151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb2da00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb280c8f40 0x7ffb28151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:33.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb2da00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2815b560 con 0x7ffb28070c00 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb2da00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2815ebe0 con 0x7ffb280c8f40 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb2da00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffb2815b6e0 con 0x7ffb28070fe0 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb26a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb28070c00 0x7ffb281f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb26a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb28070c00 0x7ffb281f3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41590/0 (socket says 172.21.15.98:41590) 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb26a00640 1 -- 172.21.15.98:0/1405570835 learned_addr learned my addr 172.21.15.98:0/1405570835 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb26000640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070fe0 0x7ffb28151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb27400640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb280c8f40 0x7ffb28151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb26a00640 1 -- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070fe0 msgr2=0x7ffb28151510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb26a00640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070fe0 0x7ffb28151510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb26a00640 1 -- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb280c8f40 msgr2=0x7ffb28151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb26a00640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb280c8f40 0x7ffb28151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:33.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb26a00640 1 -- 172.21.15.98:0/1405570835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2814c030 con 0x7ffb28070c00 2024-05-27T00:30:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb26a00640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb28070c00 0x7ffb281f3d30 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ffb180b3040 tx=0x7ffb1801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb1ea00640 1 -- 172.21.15.98:0/1405570835 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb180bf070 con 0x7ffb28070c00 2024-05-27T00:30:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb1ea00640 1 -- 172.21.15.98:0/1405570835 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb180b07a0 con 0x7ffb28070c00 2024-05-27T00:30:33.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb2da00640 1 -- 172.21.15.98:0/1405570835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb2814f760 con 0x7ffb28070c00 2024-05-27T00:30:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb1ea00640 1 -- 172.21.15.98:0/1405570835 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb180b5040 con 0x7ffb28070c00 2024-05-27T00:30:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.892+0000 7ffb2da00640 1 -- 172.21.15.98:0/1405570835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb2814ef00 con 0x7ffb28070c00 2024-05-27T00:30:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.896+0000 7ffb2da00640 1 -- 172.21.15.98:0/1405570835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb28075cb0 con 0x7ffb28070c00 2024-05-27T00:30:33.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.896+0000 7ffb1ea00640 1 -- 172.21.15.98:0/1405570835 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffb180c2070 con 0x7ffb28070c00 2024-05-27T00:30:33.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.896+0000 7ffb1ea00640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffb080678c0 0x7ffb08069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:33.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.896+0000 7ffb1ea00640 1 -- 172.21.15.98:0/1405570835 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7ffb18122880 con 0x7ffb28070c00 2024-05-27T00:30:33.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.896+0000 7ffb26000640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffb080678c0 0x7ffb08069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:33.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.896+0000 7ffb26000640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffb080678c0 0x7ffb08069d80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ffb2814ec40 tx=0x7ffb14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:33.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:33.900+0000 7ffb1ea00640 1 -- 172.21.15.98:0/1405570835 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7ffb180187f0 con 0x7ffb28070c00 2024-05-27T00:30:34.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.088+0000 7ffb2da00640 1 -- 172.21.15.98:0/1405570835 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb280c9720 con 0x7ffb28070c00 2024-05-27T00:30:34.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.088+0000 7ffb1ea00640 1 -- 172.21.15.98:0/1405570835 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6839 (secure 0 0 0) 0x7ffb180f2f50 con 0x7ffb28070c00 2024-05-27T00:30:34.091 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:34.092 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:12.690345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6078,"qdb_cluster":[6078]},"id":1}]} 2024-05-27T00:30:34.092 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2024-05-27T00:30:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 -- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffb080678c0 msgr2=0x7ffb08069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:34.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffb080678c0 0x7ffb08069d80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ffb2814ec40 tx=0x7ffb14008040 comp rx=0 tx=0).stop 2024-05-27T00:30:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 -- 172.21.15.98:0/1405570835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb28070c00 msgr2=0x7ffb281f3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb28070c00 0x7ffb281f3d30 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ffb180b3040 tx=0x7ffb1801ba30 comp rx=0 tx=0).stop 2024-05-27T00:30:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 -- 172.21.15.98:0/1405570835 shutdown_connections 2024-05-27T00:30:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7ffb080678c0 0x7ffb08069d80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb280c8f40 0x7ffb28151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb28070fe0 0x7ffb28151510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 --2- 172.21.15.98:0/1405570835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb28070c00 0x7ffb281f3d30 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 -- 172.21.15.98:0/1405570835 >> 172.21.15.98:0/1405570835 conn(0x7ffb28072f90 msgr2=0x7ffb280c86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 -- 172.21.15.98:0/1405570835 shutdown_connections 2024-05-27T00:30:34.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:34.092+0000 7ffb2da00640 1 -- 172.21.15.98:0/1405570835 wait complete. 2024-05-27T00:30:34.113 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:30:36.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f872f400640 1 Processor -- start 2024-05-27T00:30:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f872f400640 1 -- start start 2024-05-27T00:30:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f872f400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f87280cc1f0 0x7f87281592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f872f400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 0x7f87281559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f872f400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87280c1f60 con 0x7f87280cbe10 2024-05-27T00:30:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f872f400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f8728159810 con 0x7f87280cc1f0 2024-05-27T00:30:36.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f872f400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f87280c6dd0 con 0x7f8728073570 2024-05-27T00:30:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f8727e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f87280cc1f0 0x7f87281592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f872d400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 0x7f87281559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f8727e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f87280cc1f0 0x7f87281592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:52906/0 (socket says 172.21.15.98:52906) 2024-05-27T00:30:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f872d400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 0x7f87281559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:52146/0 (socket says 172.21.15.98:52146) 2024-05-27T00:30:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f8727e00640 1 -- 172.21.15.98:0/2309696082 learned_addr learned my addr 172.21.15.98:0/2309696082 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f8727e00640 1 -- 172.21.15.98:0/2309696082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 msgr2=0x7f87281559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f8727e00640 1 --2- 172.21.15.98:0/2309696082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 0x7f87281559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f8727e00640 1 -- 172.21.15.98:0/2309696082 >> v1:172.21.15.204:6789/0 conn(0x7f87280cbe10 legacy=0x7f87280c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:36.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.260+0000 7f8727e00640 1 -- 172.21.15.98:0/2309696082 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87280c70b0 con 0x7f87280cc1f0 2024-05-27T00:30:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f8727e00640 1 --2- 172.21.15.98:0/2309696082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87280cc1f0 0x7f87281592d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8714009870 tx=0x7f871401ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed687bab978047bb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f8727400640 1 -- 172.21.15.98:0/2309696082 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87140b5070 con 0x7f87280cc1f0 2024-05-27T00:30:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f8727400640 1 -- 172.21.15.98:0/2309696082 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87140a8a50 con 0x7f87280cc1f0 2024-05-27T00:30:36.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f8727400640 1 -- 172.21.15.98:0/2309696082 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87140a8e20 con 0x7f87280cc1f0 2024-05-27T00:30:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f872f400640 1 -- 172.21.15.98:0/2309696082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87280cc1f0 msgr2=0x7f87281592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f872f400640 1 --2- 172.21.15.98:0/2309696082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87280cc1f0 0x7f87281592d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8714009870 tx=0x7f871401ed50 comp rx=0 tx=0).stop 2024-05-27T00:30:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f872f400640 1 -- 172.21.15.98:0/2309696082 shutdown_connections 2024-05-27T00:30:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f872f400640 1 --2- 172.21.15.98:0/2309696082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 0x7f87281559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f872f400640 1 --2- 172.21.15.98:0/2309696082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87280cc1f0 0x7f87281592d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f872f400640 1 -- 172.21.15.98:0/2309696082 >> 172.21.15.98:0/2309696082 conn(0x7f8728067530 msgr2=0x7f872806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f872f400640 1 -- 172.21.15.98:0/2309696082 shutdown_connections 2024-05-27T00:30:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.264+0000 7f872f400640 1 -- 172.21.15.98:0/2309696082 wait complete. 2024-05-27T00:30:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872f400640 1 Processor -- start 2024-05-27T00:30:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872f400640 1 -- start start 2024-05-27T00:30:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872f400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 0x7f8728151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87280cbe10 0x7f87281f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87280cc1f0 0x7f87281f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872f400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87280c1330 con 0x7f87280cc1f0 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872f400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f87280c1050 con 0x7f87280cbe10 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872f400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f87280c79d0 con 0x7f8728073570 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87280cc1f0 0x7f87281f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87280cc1f0 0x7f87281f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41598/0 (socket says 172.21.15.98:41598) 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872d400640 1 -- 172.21.15.98:0/3508432886 learned_addr learned my addr 172.21.15.98:0/3508432886 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872ca00640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 0x7f8728151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f8727e00640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87280cbe10 0x7f87281f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872d400640 1 -- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 msgr2=0x7f8728151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872d400640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 0x7f8728151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872d400640 1 -- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87280cbe10 msgr2=0x7f87281f37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872d400640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87280cbe10 0x7f87281f37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872d400640 1 -- 172.21.15.98:0/3508432886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87281f0360 con 0x7f87280cc1f0 2024-05-27T00:30:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872ca00640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 0x7f8728151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:30:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f8727e00640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87280cbe10 0x7f87281f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:30:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872d400640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87280cc1f0 0x7f87281f3d00 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f872000efc0 tx=0x7f872000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f8724c00640 1 -- 172.21.15.98:0/3508432886 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720018020 con 0x7f87280cc1f0 2024-05-27T00:30:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f8724c00640 1 -- 172.21.15.98:0/3508432886 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8720014480 con 0x7f87280cc1f0 2024-05-27T00:30:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872f400640 1 -- 172.21.15.98:0/3508432886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87280c3d90 con 0x7f87280cc1f0 2024-05-27T00:30:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f8724c00640 1 -- 172.21.15.98:0/3508432886 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720014870 con 0x7f87280cc1f0 2024-05-27T00:30:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.268+0000 7f872f400640 1 -- 172.21.15.98:0/3508432886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87281f08e0 con 0x7f87280cc1f0 2024-05-27T00:30:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.272+0000 7f872f400640 1 -- 172.21.15.98:0/3508432886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8728075cb0 con 0x7f87280cc1f0 2024-05-27T00:30:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.272+0000 7f8724c00640 1 -- 172.21.15.98:0/3508432886 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f872001b020 con 0x7f87280cc1f0 2024-05-27T00:30:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.272+0000 7f8724c00640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87080678f0 0x7f8708069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.272+0000 7f8724c00640 1 -- 172.21.15.98:0/3508432886 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f8720086bb0 con 0x7f87280cc1f0 2024-05-27T00:30:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.272+0000 7f872ca00640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87080678f0 0x7f8708069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.272+0000 7f872ca00640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87080678f0 0x7f8708069db0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f871c001710 tx=0x7f871c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.276+0000 7f8724c00640 1 -- 172.21.15.98:0/3508432886 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8720022070 con 0x7f87280cc1f0 2024-05-27T00:30:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.464+0000 7f872f400640 1 -- 172.21.15.98:0/3508432886 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8728158bd0 con 0x7f87280cc1f0 2024-05-27T00:30:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.464+0000 7f8724c00640 1 -- 172.21.15.98:0/3508432886 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6839 (secure 0 0 0) 0x7f8720010040 con 0x7f87280cc1f0 2024-05-27T00:30:36.469 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:36.469 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:12.690345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6078,"qdb_cluster":[6078]},"id":1}]} 2024-05-27T00:30:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 -- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87080678f0 msgr2=0x7f8708069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87080678f0 0x7f8708069db0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f871c001710 tx=0x7f871c009040 comp rx=0 tx=0).stop 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 -- 172.21.15.98:0/3508432886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87280cc1f0 msgr2=0x7f87281f3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87280cc1f0 0x7f87281f3d00 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f872000efc0 tx=0x7f872000c490 comp rx=0 tx=0).stop 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 -- 172.21.15.98:0/3508432886 shutdown_connections 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87280cc1f0 0x7f87281f3d00 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f87080678f0 0x7f8708069db0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f87280cbe10 0x7f87281f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 --2- 172.21.15.98:0/3508432886 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8728073570 0x7f8728151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 -- 172.21.15.98:0/3508432886 >> 172.21.15.98:0/3508432886 conn(0x7f8728067530 msgr2=0x7f8728154940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 -- 172.21.15.98:0/3508432886 shutdown_connections 2024-05-27T00:30:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:36.468+0000 7f872f400640 1 -- 172.21.15.98:0/3508432886 wait complete. 2024-05-27T00:30:36.491 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:30:38.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:38.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.636+0000 7f8b0aa00640 1 Processor -- start 2024-05-27T00:30:38.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.636+0000 7f8b0aa00640 1 -- start start 2024-05-27T00:30:38.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8b0aa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8b040735d0 0x7f8b04151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:38.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8b0aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040c79f0 0x7f8b04151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:38.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8b0aa00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b040c17d0 con 0x7f8b040731f0 2024-05-27T00:30:38.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8b0aa00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8b04071990 con 0x7f8b040735d0 2024-05-27T00:30:38.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8b0aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8b04070e10 con 0x7f8b040c79f0 2024-05-27T00:30:38.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8afb400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8b040735d0 0x7f8b04151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:38.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8b03e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8b040731f0 0x7f8b040c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33296/0 (socket says 172.21.15.98:33296) 2024-05-27T00:30:38.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8b03e00640 1 -- 172.21.15.98:0/1777739969 learned_addr learned my addr 172.21.15.98:0/1777739969 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:30:38.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8b08a00640 1 --2- 172.21.15.98:0/1777739969 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040c79f0 0x7f8b04151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:38.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8afb400640 1 -- 172.21.15.98:0/1777739969 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040c79f0 msgr2=0x7f8b04151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:38.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8afb400640 1 --2- 172.21.15.98:0/1777739969 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040c79f0 0x7f8b04151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:38.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8afb400640 1 -- 172.21.15.98:0/1777739969 >> v1:172.21.15.98:6789/0 conn(0x7f8b040731f0 legacy=0x7f8b040c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:38.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8afb400640 1 -- 172.21.15.98:0/1777739969 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b040710f0 con 0x7f8b040735d0 2024-05-27T00:30:38.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.640+0000 7f8b03400640 1 -- 172.21.15.98:0/1777739969 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2894681586 0 0) 0x7f8b040c17d0 con 0x7f8b040731f0 2024-05-27T00:30:38.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8afb400640 1 --2- 172.21.15.98:0/1777739969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 0x7f8b04151430 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8aec0097c0 tx=0x7f8aec01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e990443259e776 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:38.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b03400640 1 -- 172.21.15.98:0/1777739969 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0b5070 con 0x7f8b040735d0 2024-05-27T00:30:38.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b03400640 1 -- 172.21.15.98:0/1777739969 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aec0a8990 con 0x7f8b040735d0 2024-05-27T00:30:38.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b03400640 1 -- 172.21.15.98:0/1777739969 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0a8d60 con 0x7f8b040735d0 2024-05-27T00:30:38.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1777739969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 msgr2=0x7f8b04151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:38.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 --2- 172.21.15.98:0/1777739969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 0x7f8b04151430 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8aec0097c0 tx=0x7f8aec01ec90 comp rx=0 tx=0).stop 2024-05-27T00:30:38.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1777739969 shutdown_connections 2024-05-27T00:30:38.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 --2- 172.21.15.98:0/1777739969 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040c79f0 0x7f8b04151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:38.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 --2- 172.21.15.98:0/1777739969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 0x7f8b04151430 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:38.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1777739969 >> 172.21.15.98:0/1777739969 conn(0x7f8b04067530 msgr2=0x7f8b0406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:38.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1777739969 shutdown_connections 2024-05-27T00:30:38.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1777739969 wait complete. 2024-05-27T00:30:38.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 Processor -- start 2024-05-27T00:30:38.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 -- start start 2024-05-27T00:30:38.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040731f0 0x7f8b041eea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:38.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 0x7f8b04153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b040c79f0 0x7f8b041ef2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b040c8400 con 0x7f8b040735d0 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8b040716b0 con 0x7f8b040c79f0 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8b0aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8b04157970 con 0x7f8b040731f0 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.644+0000 7f8afb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 0x7f8b04153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8afb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 0x7f8b04153980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41610/0 (socket says 172.21.15.98:41610) 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8afb400640 1 -- 172.21.15.98:0/1644964169 learned_addr learned my addr 172.21.15.98:0/1644964169 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b03e00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040731f0 0x7f8b041eea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b08a00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b040c79f0 0x7f8b041ef2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8afb400640 1 -- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040731f0 msgr2=0x7f8b041eea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8afb400640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040731f0 0x7f8b041eea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:38.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8afb400640 1 -- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b040c79f0 msgr2=0x7f8b041ef2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:38.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8afb400640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b040c79f0 0x7f8b041ef2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:38.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8afb400640 1 -- 172.21.15.98:0/1644964169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b041edf90 con 0x7f8b040735d0 2024-05-27T00:30:38.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8afb400640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 0x7f8b04153980 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8aec01c8c0 tx=0x7f8aec01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b00c00640 1 -- 172.21.15.98:0/1644964169 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0bf070 con 0x7f8b040735d0 2024-05-27T00:30:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b00c00640 1 -- 172.21.15.98:0/1644964169 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aec0b0820 con 0x7f8b040735d0 2024-05-27T00:30:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b00c00640 1 -- 172.21.15.98:0/1644964169 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0b5040 con 0x7f8b040735d0 2024-05-27T00:30:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1644964169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b041f1fa0 con 0x7f8b040735d0 2024-05-27T00:30:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1644964169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b041f2280 con 0x7f8b040735d0 2024-05-27T00:30:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1644964169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b04075cb0 con 0x7f8b040735d0 2024-05-27T00:30:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b00c00640 1 -- 172.21.15.98:0/1644964169 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8aec0b0c10 con 0x7f8b040735d0 2024-05-27T00:30:38.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b00c00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ae00678c0 0x7f8ae0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:38.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.648+0000 7f8b00c00640 1 -- 172.21.15.98:0/1644964169 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6685+0+0 (secure 0 0 0) 0x7f8aec122ad0 con 0x7f8b040735d0 2024-05-27T00:30:38.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.652+0000 7f8b03e00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ae00678c0 0x7f8ae0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:38.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.652+0000 7f8b03e00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ae00678c0 0x7f8ae0069d80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8af4000f70 tx=0x7f8af4007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:38.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.652+0000 7f8b00c00640 1 -- 172.21.15.98:0/1644964169 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8aec0187f0 con 0x7f8b040735d0 2024-05-27T00:30:38.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.840+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1644964169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b0414e570 con 0x7f8b040735d0 2024-05-27T00:30:38.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b00c00640 1 -- 172.21.15.98:0/1644964169 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6839 (secure 0 0 0) 0x7f8aec0f3120 con 0x7f8b040735d0 2024-05-27T00:30:38.846 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:38.846 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:29:12.690345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6078":{"gid":6078,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":15,"addr":"172.21.15.98:6837/1802902856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1802902856},{"type":"v1","addr":"172.21.15.98:6837","nonce":1802902856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6078,"qdb_cluster":[6078]},"id":1}]} 2024-05-27T00:30:38.846 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 62 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ae00678c0 msgr2=0x7f8ae0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ae00678c0 0x7f8ae0069d80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8af4000f70 tx=0x7f8af4007b60 comp rx=0 tx=0).stop 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1644964169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 msgr2=0x7f8b04153980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 0x7f8b04153980 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8aec01c8c0 tx=0x7f8aec01ba30 comp rx=0 tx=0).stop 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1644964169 shutdown_connections 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8ae00678c0 0x7f8ae0069d80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b040c79f0 0x7f8b041ef2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b040735d0 0x7f8b04153980 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 --2- 172.21.15.98:0/1644964169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040731f0 0x7f8b041eea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1644964169 >> 172.21.15.98:0/1644964169 conn(0x7f8b04067530 msgr2=0x7f8b040c6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1644964169 shutdown_connections 2024-05-27T00:30:38.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:38.844+0000 7f8b0aa00640 1 -- 172.21.15.98:0/1644964169 wait complete. 2024-05-27T00:30:38.869 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T00:30:40.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:41.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.016+0000 7f2d32800640 1 Processor -- start 2024-05-27T00:30:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.016+0000 7f2d32800640 1 -- start start 2024-05-27T00:30:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.016+0000 7f2d32800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c154a80 0x7f2d2c072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.016+0000 7f2d32800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d2c062720 0x7f2d2c15e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.016+0000 7f2d32800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f2d2c0c6f60 0x7f2d2c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.016+0000 7f2d32800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2d2c0726a0 con 0x7f2d2c062720 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.016+0000 7f2d32800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f2d2c0700d0 con 0x7f2d2c0c6f60 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.016+0000 7f2d32800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2d2c071a90 con 0x7f2d2c154a80 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.016+0000 7f2d2aa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d2c062720 0x7f2d2c15e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2aa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2d2c062720 0x7f2d2c15e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41614/0 (socket says 172.21.15.98:41614) 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2aa00640 1 -- 172.21.15.98:0/1048172694 learned_addr learned my addr 172.21.15.98:0/1048172694 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2be00640 1 --2- 172.21.15.98:0/1048172694 >> v2:172.21.15.204:3300/0 conn(0x7f2d2c0c6f60 0x7f2d2c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2b400640 1 --2- 172.21.15.98:0/1048172694 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c154a80 0x7f2d2c072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2aa00640 1 -- 172.21.15.98:0/1048172694 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c154a80 msgr2=0x7f2d2c072b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2aa00640 1 --2- 172.21.15.98:0/1048172694 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c154a80 0x7f2d2c072b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2aa00640 1 -- 172.21.15.98:0/1048172694 >> v2:172.21.15.204:3300/0 conn(0x7f2d2c0c6f60 msgr2=0x7f2d2c06ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2aa00640 1 --2- 172.21.15.98:0/1048172694 >> v2:172.21.15.204:3300/0 conn(0x7f2d2c0c6f60 0x7f2d2c06ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2aa00640 1 -- 172.21.15.98:0/1048172694 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2c071d70 con 0x7f2d2c062720 2024-05-27T00:30:41.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2aa00640 1 --2- 172.21.15.98:0/1048172694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c062720 0x7f2d2c15e6b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c072610 tx=0x7f2d1801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd07d696a7172255 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:41.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2a000640 1 -- 172.21.15.98:0/1048172694 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d180b0070 con 0x7f2d2c062720 2024-05-27T00:30:41.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2a000640 1 -- 172.21.15.98:0/1048172694 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d180ab600 con 0x7f2d2c062720 2024-05-27T00:30:41.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.020+0000 7f2d2a000640 1 -- 172.21.15.98:0/1048172694 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d180ab9d0 con 0x7f2d2c062720 2024-05-27T00:30:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 -- 172.21.15.98:0/1048172694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c062720 msgr2=0x7f2d2c15e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 --2- 172.21.15.98:0/1048172694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c062720 0x7f2d2c15e6b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c072610 tx=0x7f2d1801c490 comp rx=0 tx=0).stop 2024-05-27T00:30:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 -- 172.21.15.98:0/1048172694 shutdown_connections 2024-05-27T00:30:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 --2- 172.21.15.98:0/1048172694 >> v2:172.21.15.204:3300/0 conn(0x7f2d2c0c6f60 0x7f2d2c06ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 --2- 172.21.15.98:0/1048172694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c062720 0x7f2d2c15e6b0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 --2- 172.21.15.98:0/1048172694 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c154a80 0x7f2d2c072b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 -- 172.21.15.98:0/1048172694 >> 172.21.15.98:0/1048172694 conn(0x7f2d2c0cc560 msgr2=0x7f2d2c0cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:41.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 -- 172.21.15.98:0/1048172694 shutdown_connections 2024-05-27T00:30:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 -- 172.21.15.98:0/1048172694 wait complete. 2024-05-27T00:30:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 Processor -- start 2024-05-27T00:30:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 -- start start 2024-05-27T00:30:41.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c062720 0x7f2d2c14a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d2c0c6f60 0x7f2d2c15d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c154a80 0x7f2d2c1f71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c0c74f0 con 0x7f2d2c154a80 2024-05-27T00:30:41.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c0c7370 con 0x7f2d2c0c6f60 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d32800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2d2c15a740 con 0x7f2d2c062720 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c154a80 0x7f2d2c1f71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c154a80 0x7f2d2c1f71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41622/0 (socket says 172.21.15.98:41622) 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2be00640 1 -- 172.21.15.98:0/1882631733 learned_addr learned my addr 172.21.15.98:0/1882631733 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2aa00640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d2c0c6f60 0x7f2d2c15d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2b400640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c062720 0x7f2d2c14a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2be00640 1 -- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c062720 msgr2=0x7f2d2c14a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2be00640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c062720 0x7f2d2c14a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2be00640 1 -- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d2c0c6f60 msgr2=0x7f2d2c15d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2be00640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d2c0c6f60 0x7f2d2c15d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2be00640 1 -- 172.21.15.98:0/1882631733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2c0c4e10 con 0x7f2d2c154a80 2024-05-27T00:30:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2b400640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c062720 0x7f2d2c14a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:30:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d2be00640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c154a80 0x7f2d2c1f71f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2d24007e40 tx=0x7f2d2400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d23400640 1 -- 172.21.15.98:0/1882631733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24014620 con 0x7f2d2c154a80 2024-05-27T00:30:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d23400640 1 -- 172.21.15.98:0/1882631733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d24019070 con 0x7f2d2c154a80 2024-05-27T00:30:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.024+0000 7f2d23400640 1 -- 172.21.15.98:0/1882631733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24014a20 con 0x7f2d2c154a80 2024-05-27T00:30:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.028+0000 7f2d32800640 1 -- 172.21.15.98:0/1882631733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d2c062d80 con 0x7f2d2c154a80 2024-05-27T00:30:41.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.028+0000 7f2d32800640 1 -- 172.21.15.98:0/1882631733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d2c15be90 con 0x7f2d2c154a80 2024-05-27T00:30:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.028+0000 7f2d32800640 1 -- 172.21.15.98:0/1882631733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d2c075cb0 con 0x7f2d2c154a80 2024-05-27T00:30:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.028+0000 7f2d23400640 1 -- 172.21.15.98:0/1882631733 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2d24014bc0 con 0x7f2d2c154a80 2024-05-27T00:30:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.028+0000 7f2d23400640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2d0c0678f0 0x7f2d0c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.028+0000 7f2d23400640 1 -- 172.21.15.98:0/1882631733 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f2d24086990 con 0x7f2d2c154a80 2024-05-27T00:30:41.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.028+0000 7f2d2b400640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2d0c0678f0 0x7f2d0c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.032+0000 7f2d2b400640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2d0c0678f0 0x7f2d0c069db0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00b970 tx=0x7f2d1c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:41.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.032+0000 7f2d23400640 1 -- 172.21.15.98:0/1882631733 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f2d2c0c4e10 con 0x7f2d2c154a80 2024-05-27T00:30:41.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 -- 172.21.15.98:0/1882631733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d2c072b30 con 0x7f2d2c154a80 2024-05-27T00:30:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d23400640 1 -- 172.21.15.98:0/1882631733 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6006 (secure 0 0 0) 0x7f2d24057090 con 0x7f2d2c154a80 2024-05-27T00:30:41.223 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:41.223 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:30:40.296221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:30:41.223 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 63 2024-05-27T00:30:41.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 -- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2d0c0678f0 msgr2=0x7f2d0c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:41.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2d0c0678f0 0x7f2d0c069db0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00b970 tx=0x7f2d1c008040 comp rx=0 tx=0).stop 2024-05-27T00:30:41.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 -- 172.21.15.98:0/1882631733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c154a80 msgr2=0x7f2d2c1f71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:41.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c154a80 0x7f2d2c1f71f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2d24007e40 tx=0x7f2d2400e4b0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 -- 172.21.15.98:0/1882631733 shutdown_connections 2024-05-27T00:30:41.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d2c154a80 0x7f2d2c1f71f0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2d0c0678f0 0x7f2d0c069db0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d2c0c6f60 0x7f2d2c15d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 --2- 172.21.15.98:0/1882631733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d2c062720 0x7f2d2c14a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.220+0000 7f2d32800640 1 -- 172.21.15.98:0/1882631733 >> 172.21.15.98:0/1882631733 conn(0x7f2d2c0cc560 msgr2=0x7f2d2c1585b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:41.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.224+0000 7f2d32800640 1 -- 172.21.15.98:0/1882631733 shutdown_connections 2024-05-27T00:30:41.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.224+0000 7f2d32800640 1 -- 172.21.15.98:0/1882631733 wait complete. 2024-05-27T00:30:41.245 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-05-27T00:30:41.245 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:30:41.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:41.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3dc000640 1 Processor -- start 2024-05-27T00:30:41.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3dc000640 1 -- start start 2024-05-27T00:30:41.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3dc000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3d40cdf90 0x7fb3d406f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3dc000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb3d40ce370 0x7fb3d4165140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3dc000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3d406cfb0 con 0x7fb3d40c4fd0 2024-05-27T00:30:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3dc000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb3d406d500 con 0x7fb3d40cdf90 2024-05-27T00:30:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3dc000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb3d406ef50 con 0x7fb3d40ce370 2024-05-27T00:30:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3d9600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3d40cdf90 0x7fb3d406f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3d9600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb3d40cdf90 0x7fb3d406f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41638/0 (socket says 172.21.15.98:41638) 2024-05-27T00:30:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3d9600640 1 -- 172.21.15.98:0/429603341 learned_addr learned my addr 172.21.15.98:0/429603341 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:41.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3da000640 1 --2- 172.21.15.98:0/429603341 >> v2:172.21.15.204:3300/0 conn(0x7fb3d40ce370 0x7fb3d4165140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3d9600640 1 -- 172.21.15.98:0/429603341 >> v2:172.21.15.204:3300/0 conn(0x7fb3d40ce370 msgr2=0x7fb3d4165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:41.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3d9600640 1 --2- 172.21.15.98:0/429603341 >> v2:172.21.15.204:3300/0 conn(0x7fb3d40ce370 0x7fb3d4165140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3d9600640 1 -- 172.21.15.98:0/429603341 >> v1:172.21.15.204:6789/0 conn(0x7fb3d40c4fd0 legacy=0x7fb3d40c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:41.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.440+0000 7fb3d9600640 1 -- 172.21.15.98:0/429603341 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d40ca840 con 0x7fb3d40cdf90 2024-05-27T00:30:41.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3d9600640 1 --2- 172.21.15.98:0/429603341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d40cdf90 0x7fb3d406f760 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4070b50 tx=0x7fb3c801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd7644f2c97c1d5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:41.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3d3e00640 1 -- 172.21.15.98:0/429603341 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80b5070 con 0x7fb3d40cdf90 2024-05-27T00:30:41.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3d3e00640 1 -- 172.21.15.98:0/429603341 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c80a89d0 con 0x7fb3d40cdf90 2024-05-27T00:30:41.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3d3e00640 1 -- 172.21.15.98:0/429603341 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80a8da0 con 0x7fb3d40cdf90 2024-05-27T00:30:41.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3dc000640 1 -- 172.21.15.98:0/429603341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d40cdf90 msgr2=0x7fb3d406f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:41.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3dc000640 1 --2- 172.21.15.98:0/429603341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d40cdf90 0x7fb3d406f760 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4070b50 tx=0x7fb3c801ecd0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3dc000640 1 -- 172.21.15.98:0/429603341 shutdown_connections 2024-05-27T00:30:41.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3dc000640 1 --2- 172.21.15.98:0/429603341 >> v2:172.21.15.204:3300/0 conn(0x7fb3d40ce370 0x7fb3d4165140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3dc000640 1 --2- 172.21.15.98:0/429603341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d40cdf90 0x7fb3d406f760 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3dc000640 1 -- 172.21.15.98:0/429603341 >> 172.21.15.98:0/429603341 conn(0x7fb3d406d680 msgr2=0x7fb3d406da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:41.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3dc000640 1 -- 172.21.15.98:0/429603341 shutdown_connections 2024-05-27T00:30:41.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.444+0000 7fb3dc000640 1 -- 172.21.15.98:0/429603341 wait complete. 2024-05-27T00:30:41.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 Processor -- start 2024-05-27T00:30:41.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 -- start start 2024-05-27T00:30:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d40c4fd0 0x7fb3d4154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3d40cdf90 0x7fb3d41f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d40ce370 0x7fb3d415ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d40ce800 con 0x7fb3d40ce370 2024-05-27T00:30:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d406b610 con 0x7fb3d40c4fd0 2024-05-27T00:30:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb3d406d220 con 0x7fb3d40cdf90 2024-05-27T00:30:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3d9600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d40c4fd0 0x7fb3d4154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3da000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d40ce370 0x7fb3d415ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3d9600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d40c4fd0 0x7fb3d4154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:52958/0 (socket says 172.21.15.98:52958) 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3d9600640 1 -- 172.21.15.98:0/2344879990 learned_addr learned my addr 172.21.15.98:0/2344879990 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3d8c00640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3d40cdf90 0x7fb3d41f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3da000640 1 -- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3d40cdf90 msgr2=0x7fb3d41f16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3da000640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3d40cdf90 0x7fb3d41f16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3da000640 1 -- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d40c4fd0 msgr2=0x7fb3d4154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3da000640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d40c4fd0 0x7fb3d4154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3da000640 1 -- 172.21.15.98:0/2344879990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d414e4e0 con 0x7fb3d40ce370 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3da000640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d40ce370 0x7fb3d415ca50 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc007dc0 tx=0x7fb3cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3d1600640 1 -- 172.21.15.98:0/2344879990 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc00edd0 con 0x7fb3d40ce370 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3d1600640 1 -- 172.21.15.98:0/2344879990 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3cc0039a0 con 0x7fb3d40ce370 2024-05-27T00:30:41.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 -- 172.21.15.98:0/2344879990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d414e780 con 0x7fb3d40ce370 2024-05-27T00:30:41.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3d1600640 1 -- 172.21.15.98:0/2344879990 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc003d90 con 0x7fb3d40ce370 2024-05-27T00:30:41.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 -- 172.21.15.98:0/2344879990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d414ed40 con 0x7fb3d40ce370 2024-05-27T00:30:41.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.448+0000 7fb3dc000640 1 -- 172.21.15.98:0/2344879990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d4075cb0 con 0x7fb3d40ce370 2024-05-27T00:30:41.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.452+0000 7fb3d1600640 1 -- 172.21.15.98:0/2344879990 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb3cc01b020 con 0x7fb3d40ce370 2024-05-27T00:30:41.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.452+0000 7fb3d1600640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3b40678c0 0x7fb3b4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:41.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.452+0000 7fb3d1600640 1 -- 172.21.15.98:0/2344879990 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7fb3cc086150 con 0x7fb3d40ce370 2024-05-27T00:30:41.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.452+0000 7fb3d9600640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3b40678c0 0x7fb3b4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:41.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.452+0000 7fb3d9600640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3b40678c0 0x7fb3b4069d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb3c801ed10 tx=0x7fb3c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:41.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.456+0000 7fb3d1600640 1 -- 172.21.15.98:0/2344879990 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb3cc056200 con 0x7fb3d40ce370 2024-05-27T00:30:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.644+0000 7fb3dc000640 1 -- 172.21.15.98:0/2344879990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3d40c4790 con 0x7fb3d40ce370 2024-05-27T00:30:41.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.644+0000 7fb3d1600640 1 -- 172.21.15.98:0/2344879990 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6006 (secure 0 0 0) 0x7fb3cc05a500 con 0x7fb3d40ce370 2024-05-27T00:30:41.649 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:41.649 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:30:40.296221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:30:41.649 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 63 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 -- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3b40678c0 msgr2=0x7fb3b4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3b40678c0 0x7fb3b4069d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb3c801ed10 tx=0x7fb3c8009040 comp rx=0 tx=0).stop 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 -- 172.21.15.98:0/2344879990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d40ce370 msgr2=0x7fb3d415ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d40ce370 0x7fb3d415ca50 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc007dc0 tx=0x7fb3cc00e4b0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 -- 172.21.15.98:0/2344879990 shutdown_connections 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d40ce370 0x7fb3d415ca50 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3d40cdf90 0x7fb3d41f16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb3b40678c0 0x7fb3b4069d80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 --2- 172.21.15.98:0/2344879990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3d40c4fd0 0x7fb3d4154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 -- 172.21.15.98:0/2344879990 >> 172.21.15.98:0/2344879990 conn(0x7fb3d406d680 msgr2=0x7fb3d40c3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:41.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 -- 172.21.15.98:0/2344879990 shutdown_connections 2024-05-27T00:30:41.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:41.648+0000 7fb3dc000640 1 -- 172.21.15.98:0/2344879990 wait complete. 2024-05-27T00:30:41.671 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6240": { "addr": "172.21.15.98:6834/3070334683", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 3070334683, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 3070334683, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6240, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:30:40.296221+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6240 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:30:43.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.816+0000 7f3b9f000640 1 Processor -- start 2024-05-27T00:30:43.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.816+0000 7f3b9f000640 1 -- start start 2024-05-27T00:30:43.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b9f000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b980cbe10 0x7f3b981592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b9f000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f3b980cc1f0 0x7f3b980c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b9f000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b980c6dd0 con 0x7f3b98073570 2024-05-27T00:30:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b9f000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f3b98159810 con 0x7f3b980cc1f0 2024-05-27T00:30:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b9f000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3b980c70b0 con 0x7f3b980cbe10 2024-05-27T00:30:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b9d000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3b98073570 0x7f3b981559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59116/0 (socket says 172.21.15.98:59116) 2024-05-27T00:30:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b9d000640 1 -- 172.21.15.98:0/1108242346 learned_addr learned my addr 172.21.15.98:0/1108242346 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:30:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b8f400640 1 --2- 172.21.15.98:0/1108242346 >> v2:172.21.15.204:3300/0 conn(0x7f3b980cc1f0 0x7f3b980c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:43.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97e00640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b980cbe10 0x7f3b981592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:43.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 310207897 0 0) 0x7f3b980c6dd0 con 0x7f3b98073570 2024-05-27T00:30:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b7c003530 con 0x7f3b98073570 2024-05-27T00:30:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 299711694 0 0) 0x7f3b7c003530 con 0x7f3b98073570 2024-05-27T00:30:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b980cbe10 msgr2=0x7f3b981592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b980cbe10 0x7f3b981592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 >> v2:172.21.15.204:3300/0 conn(0x7f3b980cc1f0 msgr2=0x7f3b980c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 --2- 172.21.15.98:0/1108242346 >> v2:172.21.15.204:3300/0 conn(0x7f3b980cc1f0 0x7f3b980c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b980c7390 con 0x7f3b98073570 2024-05-27T00:30:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f3b90001a00 con 0x7f3b98073570 2024-05-27T00:30:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 >> v1:172.21.15.98:6789/0 conn(0x7f3b98073570 legacy=0x7f3b981559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:43.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b980cbe10 0x7f3b7c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 0x7f3b7c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b7c004d50 0x7f3b7c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b98159810 con 0x7f3b7c003f40 2024-05-27T00:30:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3b980c70b0 con 0x7f3b7c004d50 2024-05-27T00:30:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3b7c00ba00 con 0x7f3b980cbe10 2024-05-27T00:30:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b8f400640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 0x7f3b7c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b97e00640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b980cbe10 0x7f3b7c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b9d000640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b7c004d50 0x7f3b7c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b8f400640 1 -- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b980cbe10 msgr2=0x7f3b7c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:43.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b8f400640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b980cbe10 0x7f3b7c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:43.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b8f400640 1 -- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b7c004d50 msgr2=0x7f3b7c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:43.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b8f400640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b7c004d50 0x7f3b7c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:43.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.820+0000 7f3b8f400640 1 -- 172.21.15.98:0/1108242346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7c003530 con 0x7f3b7c003f40 2024-05-27T00:30:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b8f400640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 0x7f3b7c004810 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3b80009ab0 tx=0x7f3b8001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127cc87b32a9392b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b800b2070 con 0x7f3b7c003f40 2024-05-27T00:30:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b800a5680 con 0x7f3b7c003f40 2024-05-27T00:30:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b97400640 1 -- 172.21.15.98:0/1108242346 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b800096b0 con 0x7f3b7c003f40 2024-05-27T00:30:43.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b9f000640 1 -- 172.21.15.98:0/1108242346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 msgr2=0x7f3b7c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b9f000640 1 --2- 172.21.15.98:0/1108242346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 0x7f3b7c004810 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3b80009ab0 tx=0x7f3b8001ca90 comp rx=0 tx=0).stop 2024-05-27T00:30:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b9d000640 1 -- 172.21.15.98:0/1108242346 reap_dead start 2024-05-27T00:30:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b9f000640 1 -- 172.21.15.98:0/1108242346 shutdown_connections 2024-05-27T00:30:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b9f000640 1 -- 172.21.15.98:0/1108242346 >> 172.21.15.98:0/1108242346 conn(0x7f3b98067530 msgr2=0x7f3b9806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b9f000640 1 -- 172.21.15.98:0/1108242346 shutdown_connections 2024-05-27T00:30:43.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b9f000640 1 -- 172.21.15.98:0/1108242346 wait complete. 2024-05-27T00:30:43.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b9f000640 1 Processor -- start 2024-05-27T00:30:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.824+0000 7f3b9f000640 1 -- start start 2024-05-27T00:30:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b9f000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 0x7f3b981566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b9f000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b98157980 0x7f3b980b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b9f000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b98158c20 0x7f3b98070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b9f000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b980c79d0 con 0x7f3b7c003f40 2024-05-27T00:30:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b9f000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3b980c1330 con 0x7f3b98157980 2024-05-27T00:30:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b9f000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3b980c6990 con 0x7f3b98158c20 2024-05-27T00:30:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b97e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 0x7f3b981566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:43.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b97e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 0x7f3b981566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53212/0 (socket says 172.21.15.98:53212) 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b97e00640 1 -- 172.21.15.98:0/828665756 learned_addr learned my addr 172.21.15.98:0/828665756 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b9d000640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b98158c20 0x7f3b98070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b8f400640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b98157980 0x7f3b980b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b97e00640 1 -- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b98158c20 msgr2=0x7f3b98070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b97e00640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b98158c20 0x7f3b98070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b97e00640 1 -- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b98157980 msgr2=0x7f3b980b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b97e00640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b98157980 0x7f3b980b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b97e00640 1 -- 172.21.15.98:0/828665756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b98156ce0 con 0x7f3b7c003f40 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b97e00640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 0x7f3b981566a0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3b880119d0 tx=0x7f3b88011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b94c00640 1 -- 172.21.15.98:0/828665756 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8800fb50 con 0x7f3b7c003f40 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b94c00640 1 -- 172.21.15.98:0/828665756 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b8800fcf0 con 0x7f3b7c003f40 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b9f000640 1 -- 172.21.15.98:0/828665756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b98156fc0 con 0x7f3b7c003f40 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b94c00640 1 -- 172.21.15.98:0/828665756 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8800fea0 con 0x7f3b7c003f40 2024-05-27T00:30:43.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b9f000640 1 -- 172.21.15.98:0/828665756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b98158420 con 0x7f3b7c003f40 2024-05-27T00:30:43.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b9f000640 1 -- 172.21.15.98:0/828665756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b98075cb0 con 0x7f3b7c003f40 2024-05-27T00:30:43.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.828+0000 7f3b94c00640 1 -- 172.21.15.98:0/828665756 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3b88012040 con 0x7f3b7c003f40 2024-05-27T00:30:43.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.832+0000 7f3b94c00640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b740678f0 0x7f3b74069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:43.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.832+0000 7f3b94c00640 1 -- 172.21.15.98:0/828665756 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f3b880867a0 con 0x7f3b7c003f40 2024-05-27T00:30:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.832+0000 7f3b8f400640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b740678f0 0x7f3b74069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:43.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.832+0000 7f3b8f400640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b740678f0 0x7f3b74069db0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3b800af000 tx=0x7f3b800046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:43.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:43.836+0000 7f3b94c00640 1 -- 172.21.15.98:0/828665756 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f3b88019030 con 0x7f3b7c003f40 2024-05-27T00:30:44.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.024+0000 7f3b9f000640 1 -- 172.21.15.98:0/828665756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b980cb870 con 0x7f3b7c003f40 2024-05-27T00:30:44.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.024+0000 7f3b94c00640 1 -- 172.21.15.98:0/828665756 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6006 (secure 0 0 0) 0x7f3b88056ed0 con 0x7f3b7c003f40 2024-05-27T00:30:44.028 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:44.028 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:30:40.296221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:30:44.028 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 63 2024-05-27T00:30:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 -- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b740678f0 msgr2=0x7f3b74069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b740678f0 0x7f3b74069db0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3b800af000 tx=0x7f3b800046d0 comp rx=0 tx=0).stop 2024-05-27T00:30:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 -- 172.21.15.98:0/828665756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 msgr2=0x7f3b981566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 0x7f3b981566a0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3b880119d0 tx=0x7f3b88011e90 comp rx=0 tx=0).stop 2024-05-27T00:30:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 -- 172.21.15.98:0/828665756 shutdown_connections 2024-05-27T00:30:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b98158c20 0x7f3b98070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b740678f0 0x7f3b74069db0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b98157980 0x7f3b980b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 --2- 172.21.15.98:0/828665756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b7c003f40 0x7f3b981566a0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 -- 172.21.15.98:0/828665756 >> 172.21.15.98:0/828665756 conn(0x7f3b98067530 msgr2=0x7f3b980c9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:44.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 -- 172.21.15.98:0/828665756 shutdown_connections 2024-05-27T00:30:44.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:44.028+0000 7f3b9f000640 1 -- 172.21.15.98:0/828665756 wait complete. 2024-05-27T00:30:44.049 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:30:46.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:46.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.192+0000 7f7f86e00640 1 Processor -- start 2024-05-27T00:30:46.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.192+0000 7f7f86e00640 1 -- start start 2024-05-27T00:30:46.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.192+0000 7f7f86e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7f800731f0 0x7f7f800c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f86e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f800735d0 0x7f7f80151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f86e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f800c17d0 con 0x7f7f800c79f0 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f86e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7f80071990 con 0x7f7f800735d0 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f86e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7f80070e10 con 0x7f7f800731f0 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f77e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f800735d0 0x7f7f80151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f77e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f800735d0 0x7f7f80151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53222/0 (socket says 172.21.15.98:53222) 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f77e00640 1 -- 172.21.15.98:0/3566873482 learned_addr learned my addr 172.21.15.98:0/3566873482 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f7fe00640 1 --2- 172.21.15.98:0/3566873482 >> v2:172.21.15.204:3300/0 conn(0x7f7f800731f0 0x7f7f800c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f77e00640 1 -- 172.21.15.98:0/3566873482 >> v2:172.21.15.204:3300/0 conn(0x7f7f800731f0 msgr2=0x7f7f800c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f77e00640 1 --2- 172.21.15.98:0/3566873482 >> v2:172.21.15.204:3300/0 conn(0x7f7f800731f0 0x7f7f800c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f77e00640 1 -- 172.21.15.98:0/3566873482 >> v1:172.21.15.204:6789/0 conn(0x7f7f800c79f0 legacy=0x7f7f80157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f77e00640 1 -- 172.21.15.98:0/3566873482 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f800710f0 con 0x7f7f800735d0 2024-05-27T00:30:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f77e00640 1 --2- 172.21.15.98:0/3566873482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800735d0 0x7f7f80151430 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7f800cca90 tx=0x7f7f6801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60d537b9195afd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.196+0000 7f7f7f400640 1 -- 172.21.15.98:0/3566873482 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f680aa610 con 0x7f7f800735d0 2024-05-27T00:30:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f7f400640 1 -- 172.21.15.98:0/3566873482 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f680af070 con 0x7f7f800735d0 2024-05-27T00:30:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f7f400640 1 -- 172.21.15.98:0/3566873482 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f680aa9d0 con 0x7f7f800735d0 2024-05-27T00:30:46.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f86e00640 1 -- 172.21.15.98:0/3566873482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800735d0 msgr2=0x7f7f80151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:46.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f86e00640 1 --2- 172.21.15.98:0/3566873482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800735d0 0x7f7f80151430 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7f800cca90 tx=0x7f7f6801c4b0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f86e00640 1 -- 172.21.15.98:0/3566873482 shutdown_connections 2024-05-27T00:30:46.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f86e00640 1 --2- 172.21.15.98:0/3566873482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800735d0 0x7f7f80151430 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f86e00640 1 --2- 172.21.15.98:0/3566873482 >> v2:172.21.15.204:3300/0 conn(0x7f7f800731f0 0x7f7f800c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f86e00640 1 -- 172.21.15.98:0/3566873482 >> 172.21.15.98:0/3566873482 conn(0x7f7f80067530 msgr2=0x7f7f8006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:46.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f86e00640 1 -- 172.21.15.98:0/3566873482 shutdown_connections 2024-05-27T00:30:46.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f86e00640 1 -- 172.21.15.98:0/3566873482 wait complete. 2024-05-27T00:30:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f86e00640 1 Processor -- start 2024-05-27T00:30:46.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.200+0000 7f7f86e00640 1 -- start start 2024-05-27T00:30:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f86e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800731f0 0x7f7f8014d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f86e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f800735d0 0x7f7f8014da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f86e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f800c79f0 0x7f7f801f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f86e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f800c13f0 con 0x7f7f800731f0 2024-05-27T00:30:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f86e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7f80151d80 con 0x7f7f800c79f0 2024-05-27T00:30:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f86e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7f800c1570 con 0x7f7f800735d0 2024-05-27T00:30:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800731f0 0x7f7f8014d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800731f0 0x7f7f8014d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53236/0 (socket says 172.21.15.98:53236) 2024-05-27T00:30:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7fe00640 1 -- 172.21.15.98:0/3182103463 learned_addr learned my addr 172.21.15.98:0/3182103463 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f77e00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f800735d0 0x7f7f8014da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f84e00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f800c79f0 0x7f7f801f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7fe00640 1 -- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f800735d0 msgr2=0x7f7f8014da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7fe00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f800735d0 0x7f7f8014da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7fe00640 1 -- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f800c79f0 msgr2=0x7f7f801f7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7fe00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f800c79f0 0x7f7f801f7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7fe00640 1 -- 172.21.15.98:0/3182103463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f801403a0 con 0x7f7f800731f0 2024-05-27T00:30:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7fe00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800731f0 0x7f7f8014d4f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000d800 tx=0x7f7f7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7cc00640 1 -- 172.21.15.98:0/3182103463 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7000b6c0 con 0x7f7f800731f0 2024-05-27T00:30:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7cc00640 1 -- 172.21.15.98:0/3182103463 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f70019070 con 0x7f7f800731f0 2024-05-27T00:30:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f7cc00640 1 -- 172.21.15.98:0/3182103463 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7000ba60 con 0x7f7f800731f0 2024-05-27T00:30:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f86e00640 1 -- 172.21.15.98:0/3182103463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f80155d40 con 0x7f7f800731f0 2024-05-27T00:30:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f86e00640 1 -- 172.21.15.98:0/3182103463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f801557d0 con 0x7f7f800731f0 2024-05-27T00:30:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.204+0000 7f7f86e00640 1 -- 172.21.15.98:0/3182103463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f80075cb0 con 0x7f7f800731f0 2024-05-27T00:30:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.208+0000 7f7f7cc00640 1 -- 172.21.15.98:0/3182103463 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7f7000bc00 con 0x7f7f800731f0 2024-05-27T00:30:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.208+0000 7f7f7cc00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 0x7f7f5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.208+0000 7f7f7cc00640 1 -- 172.21.15.98:0/3182103463 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f7f70086450 con 0x7f7f800731f0 2024-05-27T00:30:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.208+0000 7f7f77e00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 0x7f7f5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.208+0000 7f7f77e00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 0x7f7f5c069d80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7f680ae000 tx=0x7f7f680a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.212+0000 7f7f7cc00640 1 -- 172.21.15.98:0/3182103463 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7f70056500 con 0x7f7f800731f0 2024-05-27T00:30:46.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.412+0000 7f7f86e00640 1 -- 172.21.15.98:0/3182103463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f8014e950 con 0x7f7f800731f0 2024-05-27T00:30:46.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.412+0000 7f7f7cc00640 1 -- 172.21.15.98:0/3182103463 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6006 (secure 0 0 0) 0x7f7f7005a800 con 0x7f7f800731f0 2024-05-27T00:30:46.416 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:46.416 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:30:40.296221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:30:46.417 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 63 2024-05-27T00:30:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 -- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 msgr2=0x7f7f5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 0x7f7f5c069d80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7f680ae000 tx=0x7f7f680a7000 comp rx=0 tx=0).stop 2024-05-27T00:30:46.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 -- 172.21.15.98:0/3182103463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800731f0 msgr2=0x7f7f8014d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800731f0 0x7f7f8014d4f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000d800 tx=0x7f7f7000dcc0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 -- 172.21.15.98:0/3182103463 shutdown_connections 2024-05-27T00:30:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7f5c0678c0 0x7f7f5c069d80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f800c79f0 0x7f7f801f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f800735d0 0x7f7f8014da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 --2- 172.21.15.98:0/3182103463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f800731f0 0x7f7f8014d4f0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 -- 172.21.15.98:0/3182103463 >> 172.21.15.98:0/3182103463 conn(0x7f7f80067530 msgr2=0x7f7f800c6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 -- 172.21.15.98:0/3182103463 shutdown_connections 2024-05-27T00:30:46.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:46.416+0000 7f7f86e00640 1 -- 172.21.15.98:0/3182103463 wait complete. 2024-05-27T00:30:46.439 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:30:48.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:48.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd966200640 1 Processor -- start 2024-05-27T00:30:48.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd966200640 1 -- start start 2024-05-27T00:30:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd966200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9600c4fd0 0x7fd96006f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd966200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd9600ce370 0x7fd960165140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:48.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd966200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96006cfb0 con 0x7fd9600cdf90 2024-05-27T00:30:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd966200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd96006d500 con 0x7fd9600ce370 2024-05-27T00:30:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd966200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd96006ef50 con 0x7fd9600c4fd0 2024-05-27T00:30:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95f400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd9600cdf90 0x7fd9600c8fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59118/0 (socket says 172.21.15.98:59118) 2024-05-27T00:30:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95f400640 1 -- 172.21.15.98:0/2035566247 learned_addr learned my addr 172.21.15.98:0/2035566247 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:30:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95fe00640 1 --2- 172.21.15.98:0/2035566247 >> v2:172.21.15.204:3300/0 conn(0x7fd9600ce370 0x7fd960165140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95ea00640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9600c4fd0 0x7fd96006f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 267318978 0 0) 0x7fd96006cfb0 con 0x7fd9600cdf90 2024-05-27T00:30:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd944003530 con 0x7fd9600cdf90 2024-05-27T00:30:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 917761359 0 0) 0x7fd944003530 con 0x7fd9600cdf90 2024-05-27T00:30:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9600c4fd0 msgr2=0x7fd96006f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9600c4fd0 0x7fd96006f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 >> v2:172.21.15.204:3300/0 conn(0x7fd9600ce370 msgr2=0x7fd960165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:48.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 --2- 172.21.15.98:0/2035566247 >> v2:172.21.15.204:3300/0 conn(0x7fd9600ce370 0x7fd960165140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9600ca840 con 0x7fd9600cdf90 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fd950001680 con 0x7fd9600cdf90 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 >> v1:172.21.15.98:6789/0 conn(0x7fd9600cdf90 legacy=0x7fd9600c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9440039c0 0x7fd944003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 0x7fd944004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9600c4fd0 0x7fd9440051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd96006d500 con 0x7fd944004810 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd96006ef50 con 0x7fd9440039c0 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd94400b9d0 con 0x7fd9600c4fd0 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95ea00640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 0x7fd944004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:48.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95f400640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9440039c0 0x7fd944003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95fe00640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9600c4fd0 0x7fd9440051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95ea00640 1 -- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9600c4fd0 msgr2=0x7fd9440051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95ea00640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9600c4fd0 0x7fd9440051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95ea00640 1 -- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9440039c0 msgr2=0x7fd944003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95ea00640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9440039c0 0x7fd944003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.584+0000 7fd95ea00640 1 -- 172.21.15.98:0/2035566247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd944003530 con 0x7fd944004810 2024-05-27T00:30:48.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd95ea00640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 0x7fd944004c90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd948009ab0 tx=0x7fd94801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27b29fa7e1805c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9480b2070 con 0x7fd944004810 2024-05-27T00:30:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9480a5560 con 0x7fd944004810 2024-05-27T00:30:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd95e000640 1 -- 172.21.15.98:0/2035566247 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9480096b0 con 0x7fd944004810 2024-05-27T00:30:48.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd966200640 1 -- 172.21.15.98:0/2035566247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 msgr2=0x7fd944004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd966200640 1 --2- 172.21.15.98:0/2035566247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 0x7fd944004c90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd948009ab0 tx=0x7fd94801ca50 comp rx=0 tx=0).stop 2024-05-27T00:30:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd95fe00640 1 -- 172.21.15.98:0/2035566247 reap_dead start 2024-05-27T00:30:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd966200640 1 -- 172.21.15.98:0/2035566247 shutdown_connections 2024-05-27T00:30:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd966200640 1 -- 172.21.15.98:0/2035566247 >> 172.21.15.98:0/2035566247 conn(0x7fd96006d680 msgr2=0x7fd96006da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd966200640 1 -- 172.21.15.98:0/2035566247 shutdown_connections 2024-05-27T00:30:48.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.588+0000 7fd966200640 1 -- 172.21.15.98:0/2035566247 wait complete. 2024-05-27T00:30:48.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd966200640 1 Processor -- start 2024-05-27T00:30:48.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd966200640 1 -- start start 2024-05-27T00:30:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd966200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 0x7fd9601f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:48.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd966200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd960153590 0x7fd9601f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd966200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd960153da0 0x7fd960154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd966200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9600ce7f0 con 0x7fd944004810 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd966200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd96006b610 con 0x7fd960153da0 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd966200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd96006d220 con 0x7fd960153590 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 0x7fd9601f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95f400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 0x7fd9601f4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53250/0 (socket says 172.21.15.98:53250) 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95f400640 1 -- 172.21.15.98:0/3440507276 learned_addr learned my addr 172.21.15.98:0/3440507276 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95ea00640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd960153590 0x7fd9601f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95fe00640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd960153da0 0x7fd960154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95f400640 1 -- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd960153590 msgr2=0x7fd9601f0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95f400640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd960153590 0x7fd9601f0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95f400640 1 -- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd960153da0 msgr2=0x7fd960154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:48.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95f400640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd960153da0 0x7fd960154220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95f400640 1 -- 172.21.15.98:0/3440507276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9600c5a70 con 0x7fd944004810 2024-05-27T00:30:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd95f400640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 0x7fd9601f4100 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd950007370 tx=0x7fd950007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd957400640 1 -- 172.21.15.98:0/3440507276 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95001a070 con 0x7fd944004810 2024-05-27T00:30:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd957400640 1 -- 172.21.15.98:0/3440507276 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95000f9c0 con 0x7fd944004810 2024-05-27T00:30:48.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd966200640 1 -- 172.21.15.98:0/3440507276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9601f10d0 con 0x7fd944004810 2024-05-27T00:30:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.592+0000 7fd957400640 1 -- 172.21.15.98:0/3440507276 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95000fd90 con 0x7fd944004810 2024-05-27T00:30:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.596+0000 7fd966200640 1 -- 172.21.15.98:0/3440507276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9601512c0 con 0x7fd944004810 2024-05-27T00:30:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.596+0000 7fd966200640 1 -- 172.21.15.98:0/3440507276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd960075cb0 con 0x7fd944004810 2024-05-27T00:30:48.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.596+0000 7fd957400640 1 -- 172.21.15.98:0/3440507276 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd95001d070 con 0x7fd944004810 2024-05-27T00:30:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.596+0000 7fd957400640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd93c0678c0 0x7fd93c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.596+0000 7fd957400640 1 -- 172.21.15.98:0/3440507276 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7fd950085bf0 con 0x7fd944004810 2024-05-27T00:30:48.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.596+0000 7fd95ea00640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd93c0678c0 0x7fd93c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:48.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.600+0000 7fd95ea00640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd93c0678c0 0x7fd93c069d80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd9480af000 tx=0x7fd9480046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:48.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.600+0000 7fd957400640 1 -- 172.21.15.98:0/3440507276 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd950056260 con 0x7fd944004810 2024-05-27T00:30:48.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.800+0000 7fd966200640 1 -- 172.21.15.98:0/3440507276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd96006f540 con 0x7fd944004810 2024-05-27T00:30:48.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd957400640 1 -- 172.21.15.98:0/3440507276 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6006 (secure 0 0 0) 0x7fd950055b60 con 0x7fd944004810 2024-05-27T00:30:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:48.805 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:30:40.296221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:30:48.806 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 63 2024-05-27T00:30:48.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 -- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd93c0678c0 msgr2=0x7fd93c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:48.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd93c0678c0 0x7fd93c069d80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd9480af000 tx=0x7fd9480046d0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 -- 172.21.15.98:0/3440507276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 msgr2=0x7fd9601f4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:48.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 0x7fd9601f4100 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd950007370 tx=0x7fd950007830 comp rx=0 tx=0).stop 2024-05-27T00:30:48.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 -- 172.21.15.98:0/3440507276 shutdown_connections 2024-05-27T00:30:48.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd93c0678c0 0x7fd93c069d80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd960153da0 0x7fd960154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd960153590 0x7fd9601f0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 --2- 172.21.15.98:0/3440507276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd944004810 0x7fd9601f4100 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:48.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 -- 172.21.15.98:0/3440507276 >> 172.21.15.98:0/3440507276 conn(0x7fd96006d680 msgr2=0x7fd9600c3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:48.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 -- 172.21.15.98:0/3440507276 shutdown_connections 2024-05-27T00:30:48.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:48.804+0000 7fd966200640 1 -- 172.21.15.98:0/3440507276 wait complete. 2024-05-27T00:30:48.828 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:30:50.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:50.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad4da00640 1 Processor -- start 2024-05-27T00:30:50.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad4da00640 1 -- start start 2024-05-27T00:30:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad4da00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad480cac80 0x7fad480c9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad4da00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad4815bb00 0x7fad481523c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad4da00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fad480c9640 0x7fad48152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad4da00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fad480c84f0 con 0x7fad480cac80 2024-05-27T00:30:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad4da00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fad480c8990 con 0x7fad480c9640 2024-05-27T00:30:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad4da00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fad4815bf90 con 0x7fad4815bb00 2024-05-27T00:30:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad47e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad480cac80 0x7fad480c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad47e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad480cac80 0x7fad480c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53260/0 (socket says 172.21.15.98:53260) 2024-05-27T00:30:50.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad47e00640 1 -- 172.21.15.98:0/917059104 learned_addr learned my addr 172.21.15.98:0/917059104 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad47400640 1 --2- 172.21.15.98:0/917059104 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad4815bb00 0x7fad481523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad4d000640 1 --2- 172.21.15.98:0/917059104 >> v2:172.21.15.204:3300/0 conn(0x7fad480c9640 0x7fad48152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad47e00640 1 -- 172.21.15.98:0/917059104 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad4815bb00 msgr2=0x7fad481523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad47e00640 1 --2- 172.21.15.98:0/917059104 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad4815bb00 0x7fad481523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad47e00640 1 -- 172.21.15.98:0/917059104 >> v2:172.21.15.204:3300/0 conn(0x7fad480c9640 msgr2=0x7fad48152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:50.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad47e00640 1 --2- 172.21.15.98:0/917059104 >> v2:172.21.15.204:3300/0 conn(0x7fad480c9640 0x7fad48152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:50.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad47e00640 1 -- 172.21.15.98:0/917059104 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4815c270 con 0x7fad480cac80 2024-05-27T00:30:50.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad4d000640 1 --2- 172.21.15.98:0/917059104 >> v2:172.21.15.204:3300/0 conn(0x7fad480c9640 0x7fad48152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:30:50.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.964+0000 7fad47400640 1 --2- 172.21.15.98:0/917059104 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad4815bb00 0x7fad481523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:30:50.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad47e00640 1 --2- 172.21.15.98:0/917059104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480cac80 0x7fad480c9100 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fad480c5670 tx=0x7fad3801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8cabc430925703 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:50.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad46a00640 1 -- 172.21.15.98:0/917059104 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad380b1070 con 0x7fad480cac80 2024-05-27T00:30:50.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad46a00640 1 -- 172.21.15.98:0/917059104 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad380ac630 con 0x7fad480cac80 2024-05-27T00:30:50.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad46a00640 1 -- 172.21.15.98:0/917059104 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad380a8050 con 0x7fad480cac80 2024-05-27T00:30:50.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 -- 172.21.15.98:0/917059104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480cac80 msgr2=0x7fad480c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:50.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 --2- 172.21.15.98:0/917059104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480cac80 0x7fad480c9100 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fad480c5670 tx=0x7fad3801c520 comp rx=0 tx=0).stop 2024-05-27T00:30:50.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 -- 172.21.15.98:0/917059104 shutdown_connections 2024-05-27T00:30:50.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 --2- 172.21.15.98:0/917059104 >> v2:172.21.15.204:3300/0 conn(0x7fad480c9640 0x7fad48152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:50.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 --2- 172.21.15.98:0/917059104 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad4815bb00 0x7fad481523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:50.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 --2- 172.21.15.98:0/917059104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480cac80 0x7fad480c9100 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 -- 172.21.15.98:0/917059104 >> 172.21.15.98:0/917059104 conn(0x7fad48070b30 msgr2=0x7fad48070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 -- 172.21.15.98:0/917059104 shutdown_connections 2024-05-27T00:30:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 -- 172.21.15.98:0/917059104 wait complete. 2024-05-27T00:30:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 Processor -- start 2024-05-27T00:30:50.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.968+0000 7fad4da00640 1 -- start start 2024-05-27T00:30:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad4da00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480c9640 0x7fad48147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad4da00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad480cac80 0x7fad48147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad4da00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad481492b0 0x7fad480c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad4da00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad48158590 con 0x7fad480c9640 2024-05-27T00:30:50.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad4da00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fad480b40f0 con 0x7fad480cac80 2024-05-27T00:30:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad4da00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fad4806e990 con 0x7fad481492b0 2024-05-27T00:30:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad47e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480c9640 0x7fad48147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad47e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480c9640 0x7fad48147860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53262/0 (socket says 172.21.15.98:53262) 2024-05-27T00:30:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad47e00640 1 -- 172.21.15.98:0/3267008916 learned_addr learned my addr 172.21.15.98:0/3267008916 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad4d000640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad481492b0 0x7fad480c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:50.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad47400640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad480cac80 0x7fad48147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad47e00640 1 -- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad481492b0 msgr2=0x7fad480c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad47e00640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad481492b0 0x7fad480c6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad47e00640 1 -- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad480cac80 msgr2=0x7fad48147da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad47e00640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad480cac80 0x7fad48147da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad47e00640 1 -- 172.21.15.98:0/3267008916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad481f42f0 con 0x7fad480c9640 2024-05-27T00:30:50.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad47e00640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480c9640 0x7fad48147860 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fad3801abd0 tx=0x7fad380192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad3fe00640 1 -- 172.21.15.98:0/3267008916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad380bc070 con 0x7fad480c9640 2024-05-27T00:30:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad3fe00640 1 -- 172.21.15.98:0/3267008916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad380ab800 con 0x7fad480c9640 2024-05-27T00:30:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad3fe00640 1 -- 172.21.15.98:0/3267008916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad380b1070 con 0x7fad480c9640 2024-05-27T00:30:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad4da00640 1 -- 172.21.15.98:0/3267008916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad480c7260 con 0x7fad480c9640 2024-05-27T00:30:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad4da00640 1 -- 172.21.15.98:0/3267008916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad481f45b0 con 0x7fad480c9640 2024-05-27T00:30:50.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad4da00640 1 -- 172.21.15.98:0/3267008916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad480c5ba0 con 0x7fad480c9640 2024-05-27T00:30:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.972+0000 7fad3fe00640 1 -- 172.21.15.98:0/3267008916 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fad380ac4f0 con 0x7fad480c9640 2024-05-27T00:30:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.976+0000 7fad3fe00640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad280678f0 0x7fad28069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.976+0000 7fad3fe00640 1 -- 172.21.15.98:0/3267008916 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7fad3811fcd0 con 0x7fad480c9640 2024-05-27T00:30:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.976+0000 7fad47400640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad280678f0 0x7fad28069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.976+0000 7fad47400640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad280678f0 0x7fad28069db0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fad34006fd0 tx=0x7fad34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:50.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:50.980+0000 7fad3fe00640 1 -- 172.21.15.98:0/3267008916 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fad38016110 con 0x7fad480c9640 2024-05-27T00:30:51.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.168+0000 7fad4da00640 1 -- 172.21.15.98:0/3267008916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad48075c40 con 0x7fad480c9640 2024-05-27T00:30:51.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.168+0000 7fad3fe00640 1 -- 172.21.15.98:0/3267008916 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+6010 (secure 0 0 0) 0x7fad380f0350 con 0x7fad480c9640 2024-05-27T00:30:51.173 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:30:50.372999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:30:51.174 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 64 2024-05-27T00:30:51.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 -- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad280678f0 msgr2=0x7fad28069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:51.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad280678f0 0x7fad28069db0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fad34006fd0 tx=0x7fad34008040 comp rx=0 tx=0).stop 2024-05-27T00:30:51.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 -- 172.21.15.98:0/3267008916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480c9640 msgr2=0x7fad48147860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:51.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480c9640 0x7fad48147860 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fad3801abd0 tx=0x7fad380192c0 comp rx=0 tx=0).stop 2024-05-27T00:30:51.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 -- 172.21.15.98:0/3267008916 shutdown_connections 2024-05-27T00:30:51.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad481492b0 0x7fad480c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:51.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fad280678f0 0x7fad28069db0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:51.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad480cac80 0x7fad48147da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:51.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 --2- 172.21.15.98:0/3267008916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad480c9640 0x7fad48147860 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:51.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 -- 172.21.15.98:0/3267008916 >> 172.21.15.98:0/3267008916 conn(0x7fad48070b30 msgr2=0x7fad48149b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:51.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 -- 172.21.15.98:0/3267008916 shutdown_connections 2024-05-27T00:30:51.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:51.172+0000 7fad4da00640 1 -- 172.21.15.98:0/3267008916 wait complete. 2024-05-27T00:30:51.194 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:30:53.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:53.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.340+0000 7f1093800640 1 Processor -- start 2024-05-27T00:30:53.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.340+0000 7f1093800640 1 -- start start 2024-05-27T00:30:53.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.340+0000 7f1093800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c070e10 0x7f108c1627e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:53.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.340+0000 7f1093800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f108c0c1000 con 0x7f108c1511d0 2024-05-27T00:30:53.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.340+0000 7f1093800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f108c1515b0 con 0x7f108c0cc8f0 2024-05-27T00:30:53.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.340+0000 7f1093800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f108c0b1dd0 con 0x7f108c070e10 2024-05-27T00:30:53.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.340+0000 7f1090e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f108c1511d0 0x7f108c0c1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45796/0 (socket says 172.21.15.98:45796) 2024-05-27T00:30:53.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.340+0000 7f1090e00640 1 -- 172.21.15.98:0/1137546312 learned_addr learned my addr 172.21.15.98:0/1137546312 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:30:53.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.340+0000 7f1091800640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c070e10 0x7f108c1627e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.340+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1030068768 0 0) 0x7f108c0c1000 con 0x7f108c1511d0 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1070004640 con 0x7f108c1511d0 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 214468498 0 0) 0x7f1070004640 con 0x7f108c1511d0 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c070e10 msgr2=0x7f108c1627e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c070e10 0x7f108c1627e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 >> v1:172.21.15.204:6789/0 conn(0x7f108c0cc8f0 legacy=0x7f108c0c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108c0c83c0 con 0x7f108c1511d0 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f107c0016a0 con 0x7f108c1511d0 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 >> v1:172.21.15.98:6789/0 conn(0x7f108c1511d0 legacy=0x7f108c0c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1070004b00 0x7f1070004f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c070e10 0x7f1070005920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1070005e60 0x7f10700062e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:53.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f108c0b1dd0 con 0x7f1070005e60 2024-05-27T00:30:53.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f107000cc10 con 0x7f1070004b00 2024-05-27T00:30:53.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f10700034d0 con 0x7f108c070e10 2024-05-27T00:30:53.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f1091800640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1070005e60 0x7f10700062e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:53.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f1090e00640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1070004b00 0x7f1070004f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:53.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f108be00640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c070e10 0x7f1070005920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:53.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f1091800640 1 -- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c070e10 msgr2=0x7f1070005920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:53.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f1091800640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c070e10 0x7f1070005920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:53.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f1091800640 1 -- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1070004b00 msgr2=0x7f1070004f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:53.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f1091800640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1070004b00 0x7f1070004f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:53.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f1091800640 1 -- 172.21.15.98:0/1137546312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1070004640 con 0x7f1070005e60 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.344+0000 7f1091800640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1070005e60 0x7f10700062e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f1084009fc0 tx=0x7f108401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658d6202b4074ca2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10840b2070 con 0x7f1070005e60 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10840a5b80 con 0x7f1070005e60 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f108b400640 1 -- 172.21.15.98:0/1137546312 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1084009c10 con 0x7f1070005e60 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 -- 172.21.15.98:0/1137546312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1070005e60 msgr2=0x7f10700062e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 --2- 172.21.15.98:0/1137546312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1070005e60 0x7f10700062e0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f1084009fc0 tx=0x7f108401cc80 comp rx=0 tx=0).stop 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1091800640 1 -- 172.21.15.98:0/1137546312 reap_dead start 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 -- 172.21.15.98:0/1137546312 shutdown_connections 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 -- 172.21.15.98:0/1137546312 >> 172.21.15.98:0/1137546312 conn(0x7f108c067530 msgr2=0x7f108c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 -- 172.21.15.98:0/1137546312 shutdown_connections 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 -- 172.21.15.98:0/1137546312 wait complete. 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 Processor -- start 2024-05-27T00:30:53.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 -- start start 2024-05-27T00:30:53.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1070005e60 0x7f108c1487b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:53.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f108c157ab0 0x7f108c1539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:53.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c14a940 0x7f108c153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:53.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f108c0c80e0 con 0x7f108c157ab0 2024-05-27T00:30:53.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f108c0c1a20 con 0x7f1070005e60 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f1093800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f108c0b1a60 con 0x7f108c14a940 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.348+0000 7f108be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f108c157ab0 0x7f108c1539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f108be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f108c157ab0 0x7f108c1539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50430/0 (socket says 172.21.15.98:50430) 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f108be00640 1 -- 172.21.15.98:0/122191528 learned_addr learned my addr 172.21.15.98:0/122191528 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1090e00640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1070005e60 0x7f108c1487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1091800640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c14a940 0x7f108c153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f108be00640 1 -- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c14a940 msgr2=0x7f108c153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f108be00640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c14a940 0x7f108c153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f108be00640 1 -- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1070005e60 msgr2=0x7f108c1487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f108be00640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1070005e60 0x7f108c1487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f108be00640 1 -- 172.21.15.98:0/122191528 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108c155b40 con 0x7f108c157ab0 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f108be00640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f108c157ab0 0x7f108c1539b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f107800e5b0 tx=0x7f107800ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1088c00640 1 -- 172.21.15.98:0/122191528 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10780052e0 con 0x7f108c157ab0 2024-05-27T00:30:53.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1088c00640 1 -- 172.21.15.98:0/122191528 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1078018070 con 0x7f108c157ab0 2024-05-27T00:30:53.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1088c00640 1 -- 172.21.15.98:0/122191528 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078003820 con 0x7f108c157ab0 2024-05-27T00:30:53.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1093800640 1 -- 172.21.15.98:0/122191528 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f108c14aec0 con 0x7f108c157ab0 2024-05-27T00:30:53.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1093800640 1 -- 172.21.15.98:0/122191528 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f108c154fc0 con 0x7f108c157ab0 2024-05-27T00:30:53.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1093800640 1 -- 172.21.15.98:0/122191528 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f108c075cb0 con 0x7f108c157ab0 2024-05-27T00:30:53.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1088c00640 1 -- 172.21.15.98:0/122191528 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f10780039c0 con 0x7f108c157ab0 2024-05-27T00:30:53.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1088c00640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f106c0678f0 0x7f106c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:53.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.352+0000 7f1088c00640 1 -- 172.21.15.98:0/122191528 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f10780858f0 con 0x7f108c157ab0 2024-05-27T00:30:53.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.356+0000 7f1090e00640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f106c0678f0 0x7f106c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:53.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.356+0000 7f1090e00640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f106c0678f0 0x7f106c069db0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f107c001ec0 tx=0x7f107c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:53.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.356+0000 7f1088c00640 1 -- 172.21.15.98:0/122191528 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f108c155b40 con 0x7f108c157ab0 2024-05-27T00:30:53.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.544+0000 7f1093800640 1 -- 172.21.15.98:0/122191528 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f108c0713c0 con 0x7f108c157ab0 2024-05-27T00:30:53.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.544+0000 7f1088c00640 1 -- 172.21.15.98:0/122191528 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+6010 (secure 0 0 0) 0x7f1078055ff0 con 0x7f108c157ab0 2024-05-27T00:30:53.549 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:53.549 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:30:50.372999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:30:53.549 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 64 2024-05-27T00:30:53.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 -- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f106c0678f0 msgr2=0x7f106c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:53.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f106c0678f0 0x7f106c069db0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f107c001ec0 tx=0x7f107c009040 comp rx=0 tx=0).stop 2024-05-27T00:30:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 -- 172.21.15.98:0/122191528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f108c157ab0 msgr2=0x7f108c1539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f108c157ab0 0x7f108c1539b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f107800e5b0 tx=0x7f107800ea70 comp rx=0 tx=0).stop 2024-05-27T00:30:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 -- 172.21.15.98:0/122191528 shutdown_connections 2024-05-27T00:30:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f108c14a940 0x7f108c153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f108c157ab0 0x7f108c1539b0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f106c0678f0 0x7f106c069db0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 --2- 172.21.15.98:0/122191528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1070005e60 0x7f108c1487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 -- 172.21.15.98:0/122191528 >> 172.21.15.98:0/122191528 conn(0x7f108c067530 msgr2=0x7f108c0cbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 -- 172.21.15.98:0/122191528 shutdown_connections 2024-05-27T00:30:53.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:53.548+0000 7f1093800640 1 -- 172.21.15.98:0/122191528 wait complete. 2024-05-27T00:30:53.570 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 64, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 64, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6240": { "addr": "172.21.15.98:6834/3070334683", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 3070334683, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 3070334683, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6240, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:30:50.372999+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6240 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:30:53.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:30:53.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:30:55.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.748+0000 7fbfa0a00640 1 Processor -- start 2024-05-27T00:30:55.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.748+0000 7fbfa0a00640 1 -- start start 2024-05-27T00:30:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbfa0a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fbf9c0c8920 0x7fbf9c0cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbfa0a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf9c15a8c0 con 0x7fbf9c154800 2024-05-27T00:30:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbfa0a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf9c15aba0 con 0x7fbf9c0ce3f0 2024-05-27T00:30:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbfa0a00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fbf9c0715d0 con 0x7fbf9c0c8920 2024-05-27T00:30:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf9a800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbf9c154800 0x7fbf9c15e5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45810/0 (socket says 172.21.15.98:45810) 2024-05-27T00:30:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf9a800640 1 -- 172.21.15.98:0/3609098041 learned_addr learned my addr 172.21.15.98:0/3609098041 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:30:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf91e00640 1 --2- 172.21.15.98:0/3609098041 >> v2:172.21.15.204:3300/0 conn(0x7fbf9c0c8920 0x7fbf9c0cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1099571348 0 0) 0x7fbf9c15a8c0 con 0x7fbf9c154800 2024-05-27T00:30:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf80003530 con 0x7fbf9c154800 2024-05-27T00:30:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 246155216 0 0) 0x7fbf80003530 con 0x7fbf9c154800 2024-05-27T00:30:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 >> v2:172.21.15.204:3300/0 conn(0x7fbf9c0c8920 msgr2=0x7fbf9c0cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 --2- 172.21.15.98:0/3609098041 >> v2:172.21.15.204:3300/0 conn(0x7fbf9c0c8920 0x7fbf9c0cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 >> v1:172.21.15.204:6789/0 conn(0x7fbf9c0ce3f0 legacy=0x7fbf9c0c37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9c15ae80 con 0x7fbf9c154800 2024-05-27T00:30:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fbf8c001a00 con 0x7fbf9c154800 2024-05-27T00:30:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 >> v1:172.21.15.98:6789/0 conn(0x7fbf9c154800 legacy=0x7fbf9c15e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf80003a10 0x7fbf80003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf80004810 0x7fbf80004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:55.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800051d0 0x7fbf8000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c0715d0 con 0x7fbf800051d0 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8000c0b0 con 0x7fbf80004810 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8000c2b0 con 0x7fbf80003a10 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf9b200640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800051d0 0x7fbf8000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf91e00640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf80004810 0x7fbf80004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf9a800640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf80003a10 0x7fbf80003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf9b200640 1 -- 172.21.15.98:0/3609098041 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf80003a10 msgr2=0x7fbf80003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf9b200640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf80003a10 0x7fbf80003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf9b200640 1 -- 172.21.15.98:0/3609098041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf80004810 msgr2=0x7fbf80004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf9b200640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf80004810 0x7fbf80004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf9b200640 1 -- 172.21.15.98:0/3609098041 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf80003530 con 0x7fbf800051d0 2024-05-27T00:30:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.752+0000 7fbf9a800640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf80003a10 0x7fbf80003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:30:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbf9b200640 1 -- 172.21.15.98:0/3609098041 reap_dead start 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbf9b200640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800051d0 0x7fbf8000b9d0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbf94001780 tx=0x7fbf9401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2273a4c1a2bd4b12 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf940013a0 con 0x7fbf800051d0 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf940a9070 con 0x7fbf800051d0 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbf99e00640 1 -- 172.21.15.98:0/3609098041 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94003600 con 0x7fbf800051d0 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3609098041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800051d0 msgr2=0x7fbf8000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbfa0a00640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800051d0 0x7fbf8000b9d0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fbf94001780 tx=0x7fbf9401c420 comp rx=0 tx=0).stop 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3609098041 shutdown_connections 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbfa0a00640 1 --2- 172.21.15.98:0/3609098041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf800051d0 0x7fbf8000b9d0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3609098041 >> 172.21.15.98:0/3609098041 conn(0x7fbf9c072f90 msgr2=0x7fbf9c0733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3609098041 shutdown_connections 2024-05-27T00:30:55.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.756+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3609098041 wait complete. 2024-05-27T00:30:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 Processor -- start 2024-05-27T00:30:55.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 -- start start 2024-05-27T00:30:55.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf800051d0 0x7fbf9c0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf9c0c7b30 0x7fbf9c0c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf9c1e41c0 0x7fbf9c1f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c0c3290 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c15b440 con 0x7fbf800051d0 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbf9c154be0 con 0x7fbf9c1e41c0 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf91e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf9c0c7b30 0x7fbf9c0c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf91e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf9c0c7b30 0x7fbf9c0c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50450/0 (socket says 172.21.15.98:50450) 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf91e00640 1 -- 172.21.15.98:0/3418917667 learned_addr learned my addr 172.21.15.98:0/3418917667 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf9b200640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf9c1e41c0 0x7fbf9c1f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf9a800640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf800051d0 0x7fbf9c0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:55.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf91e00640 1 -- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf9c1e41c0 msgr2=0x7fbf9c1f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf91e00640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf9c1e41c0 0x7fbf9c1f39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf91e00640 1 -- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf800051d0 msgr2=0x7fbf9c0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf91e00640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf800051d0 0x7fbf9c0c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf91e00640 1 -- 172.21.15.98:0/3418917667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9c0c62e0 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf91e00640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf9c0c7b30 0x7fbf9c0c6fe0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbf84011990 tx=0x7fbf84011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf93400640 1 -- 172.21.15.98:0/3418917667 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8400fad0 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf93400640 1 -- 172.21.15.98:0/3418917667 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf8400fc70 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbf93400640 1 -- 172.21.15.98:0/3418917667 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84014440 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3418917667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9c151410 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3418917667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf9c0c55a0 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.760+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3418917667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf9c075c40 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.764+0000 7fbf93400640 1 -- 172.21.15.98:0/3418917667 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbf84012040 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.764+0000 7fbf93400640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf780678c0 0x7fbf78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.764+0000 7fbf93400640 1 -- 172.21.15.98:0/3418917667 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7fbf840867d0 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.764+0000 7fbf9a800640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf780678c0 0x7fbf78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:55.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.764+0000 7fbf9a800640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf780678c0 0x7fbf78069d80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c001380 tx=0x7fbf8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:55.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.768+0000 7fbf93400640 1 -- 172.21.15.98:0/3418917667 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fbf84019030 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.956+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3418917667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf9c0c81f0 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.956+0000 7fbf93400640 1 -- 172.21.15.98:0/3418917667 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+6010 (secure 0 0 0) 0x7fbf84056e40 con 0x7fbf9c0c7b30 2024-05-27T00:30:55.959 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:55.960 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:30:50.372999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:30:55.960 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 64 2024-05-27T00:30:55.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf780678c0 msgr2=0x7fbf78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf780678c0 0x7fbf78069d80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c001380 tx=0x7fbf8c009040 comp rx=0 tx=0).stop 2024-05-27T00:30:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3418917667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf9c0c7b30 msgr2=0x7fbf9c0c6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf9c0c7b30 0x7fbf9c0c6fe0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fbf84011990 tx=0x7fbf84011e50 comp rx=0 tx=0).stop 2024-05-27T00:30:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3418917667 shutdown_connections 2024-05-27T00:30:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbf9c1e41c0 0x7fbf9c1f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf9c0c7b30 0x7fbf9c0c6fe0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbf780678c0 0x7fbf78069d80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 --2- 172.21.15.98:0/3418917667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf800051d0 0x7fbf9c0c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3418917667 >> 172.21.15.98:0/3418917667 conn(0x7fbf9c072f90 msgr2=0x7fbf9c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:55.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3418917667 shutdown_connections 2024-05-27T00:30:55.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:55.960+0000 7fbfa0a00640 1 -- 172.21.15.98:0/3418917667 wait complete. 2024-05-27T00:30:55.980 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:30:57.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:30:58.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.124+0000 7f1d08c00640 1 Processor -- start 2024-05-27T00:30:58.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.124+0000 7f1d08c00640 1 -- start start 2024-05-27T00:30:58.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.124+0000 7f1d08c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1d040cc1f0 0x7f1d041592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:58.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.124+0000 7f1d08c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1d04073570 0x7f1d041559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:58.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.124+0000 7f1d08c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d040c1f60 con 0x7f1d040cbe10 2024-05-27T00:30:58.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.124+0000 7f1d08c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1d04159810 con 0x7f1d04073570 2024-05-27T00:30:58.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.124+0000 7f1d08c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f1d040c6dd0 con 0x7f1d040cc1f0 2024-05-27T00:30:58.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.124+0000 7f1d01e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1d040cbe10 0x7f1d040c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45812/0 (socket says 172.21.15.98:45812) 2024-05-27T00:30:58.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.124+0000 7f1d01e00640 1 -- 172.21.15.98:0/1842065558 learned_addr learned my addr 172.21.15.98:0/1842065558 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:30:58.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.128+0000 7f1d02800640 1 --2- 172.21.15.98:0/1842065558 >> v2:172.21.15.98:3300/0 conn(0x7f1d04073570 0x7f1d041559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:58.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.128+0000 7f1cf9400640 1 --2- 172.21.15.98:0/1842065558 >> v2:172.21.15.204:3300/0 conn(0x7f1d040cc1f0 0x7f1d041592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:58.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.128+0000 7f1d02800640 1 -- 172.21.15.98:0/1842065558 >> v2:172.21.15.204:3300/0 conn(0x7f1d040cc1f0 msgr2=0x7f1d041592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:30:58.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.128+0000 7f1d02800640 1 --2- 172.21.15.98:0/1842065558 >> v2:172.21.15.204:3300/0 conn(0x7f1d040cc1f0 0x7f1d041592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:58.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.128+0000 7f1d02800640 1 -- 172.21.15.98:0/1842065558 >> v1:172.21.15.98:6789/0 conn(0x7f1d040cbe10 legacy=0x7f1d040c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:58.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.128+0000 7f1d02800640 1 -- 172.21.15.98:0/1842065558 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d040c70b0 con 0x7f1d04073570 2024-05-27T00:30:58.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.128+0000 7f1d02800640 1 --2- 172.21.15.98:0/1842065558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 0x7f1d041559e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1d040cc6d0 tx=0x7f1cfc01f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=279f434af379809f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:58.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.128+0000 7f1d01400640 1 -- 172.21.15.98:0/1842065558 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc0b4070 con 0x7f1d04073570 2024-05-27T00:30:58.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.128+0000 7f1d01400640 1 -- 172.21.15.98:0/1842065558 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cfc0af600 con 0x7f1d04073570 2024-05-27T00:30:58.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.128+0000 7f1d01400640 1 -- 172.21.15.98:0/1842065558 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc0af9d0 con 0x7f1d04073570 2024-05-27T00:30:58.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 -- 172.21.15.98:0/1842065558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 msgr2=0x7f1d041559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 --2- 172.21.15.98:0/1842065558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 0x7f1d041559e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1d040cc6d0 tx=0x7f1cfc01f110 comp rx=0 tx=0).stop 2024-05-27T00:30:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 -- 172.21.15.98:0/1842065558 shutdown_connections 2024-05-27T00:30:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 --2- 172.21.15.98:0/1842065558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 0x7f1d041559e0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 --2- 172.21.15.98:0/1842065558 >> v2:172.21.15.204:3300/0 conn(0x7f1d040cc1f0 0x7f1d041592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 -- 172.21.15.98:0/1842065558 >> 172.21.15.98:0/1842065558 conn(0x7f1d04067530 msgr2=0x7f1d0406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 -- 172.21.15.98:0/1842065558 shutdown_connections 2024-05-27T00:30:58.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 -- 172.21.15.98:0/1842065558 wait complete. 2024-05-27T00:30:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 Processor -- start 2024-05-27T00:30:58.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 -- start start 2024-05-27T00:30:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 0x7f1d04072850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d040cbe10 0x7f1d04153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d040cc1f0 0x7f1d040ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1d040c1330 con 0x7f1d04073570 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1d040c1050 con 0x7f1d040cc1f0 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d08c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1d040c79d0 con 0x7f1d040cbe10 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d01e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 0x7f1d04072850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d01e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 0x7f1d04072850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50476/0 (socket says 172.21.15.98:50476) 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d01e00640 1 -- 172.21.15.98:0/2555822739 learned_addr learned my addr 172.21.15.98:0/2555822739 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d02800640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d040cc1f0 0x7f1d040ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1cf9400640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d040cbe10 0x7f1d04153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d01e00640 1 -- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d040cbe10 msgr2=0x7f1d04153b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d01e00640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d040cbe10 0x7f1d04153b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d01e00640 1 -- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d040cc1f0 msgr2=0x7f1d040ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:58.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d01e00640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d040cc1f0 0x7f1d040ca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:58.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d01e00640 1 -- 172.21.15.98:0/2555822739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d04071a50 con 0x7f1d04073570 2024-05-27T00:30:58.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1d01e00640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 0x7f1d04072850 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400c450 tx=0x7f1cf400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.132+0000 7f1cfaa00640 1 -- 172.21.15.98:0/2555822739 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4017070 con 0x7f1d04073570 2024-05-27T00:30:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.136+0000 7f1cfaa00640 1 -- 172.21.15.98:0/2555822739 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cf4005cf0 con 0x7f1d04073570 2024-05-27T00:30:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.136+0000 7f1cfaa00640 1 -- 172.21.15.98:0/2555822739 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4007290 con 0x7f1d04073570 2024-05-27T00:30:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.136+0000 7f1d08c00640 1 -- 172.21.15.98:0/2555822739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d04071d30 con 0x7f1d04073570 2024-05-27T00:30:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.136+0000 7f1d08c00640 1 -- 172.21.15.98:0/2555822739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d040722f0 con 0x7f1d04073570 2024-05-27T00:30:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.136+0000 7f1d08c00640 1 -- 172.21.15.98:0/2555822739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d04075cb0 con 0x7f1d04073570 2024-05-27T00:30:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.136+0000 7f1cfaa00640 1 -- 172.21.15.98:0/2555822739 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1cf401a070 con 0x7f1d04073570 2024-05-27T00:30:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.136+0000 7f1cfaa00640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1ce00678c0 0x7f1ce0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:30:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.136+0000 7f1cfaa00640 1 -- 172.21.15.98:0/2555822739 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f1cf40854a0 con 0x7f1d04073570 2024-05-27T00:30:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.136+0000 7f1cf9400640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1ce00678c0 0x7f1ce0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:30:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.140+0000 7f1cf9400640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1ce00678c0 0x7f1ce0069d80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1d040687d0 tx=0x7f1cec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:30:58.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.140+0000 7f1cfaa00640 1 -- 172.21.15.98:0/2555822739 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1cf4055b10 con 0x7f1d04073570 2024-05-27T00:30:58.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.328+0000 7f1d08c00640 1 -- 172.21.15.98:0/2555822739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d04072d90 con 0x7f1d04073570 2024-05-27T00:30:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1cfaa00640 1 -- 172.21.15.98:0/2555822739 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+6007 (secure 0 0 0) 0x7f1cf4055410 con 0x7f1d04073570 2024-05-27T00:30:58.334 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:30:58.334 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:30:55.733504+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:30:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 65 2024-05-27T00:30:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 -- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1ce00678c0 msgr2=0x7f1ce0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1ce00678c0 0x7f1ce0069d80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1d040687d0 tx=0x7f1cec008040 comp rx=0 tx=0).stop 2024-05-27T00:30:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 -- 172.21.15.98:0/2555822739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 msgr2=0x7f1d04072850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:30:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 0x7f1d04072850 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1cf400c450 tx=0x7f1cf400c910 comp rx=0 tx=0).stop 2024-05-27T00:30:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 -- 172.21.15.98:0/2555822739 shutdown_connections 2024-05-27T00:30:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1ce00678c0 0x7f1ce0069d80 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d040cc1f0 0x7f1d040ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d040cbe10 0x7f1d04153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 --2- 172.21.15.98:0/2555822739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d04073570 0x7f1d04072850 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:30:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 -- 172.21.15.98:0/2555822739 >> 172.21.15.98:0/2555822739 conn(0x7f1d04067530 msgr2=0x7f1d040c7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:30:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 -- 172.21.15.98:0/2555822739 shutdown_connections 2024-05-27T00:30:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:30:58.332+0000 7f1d08c00640 1 -- 172.21.15.98:0/2555822739 wait complete. 2024-05-27T00:30:58.355 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:31:00.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:31:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc188e00640 1 Processor -- start 2024-05-27T00:31:00.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc188e00640 1 -- start start 2024-05-27T00:31:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc188e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc184157cd0 0x7fc1840c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc188e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1840b3cb0 0x7fc1840c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc188e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1840c59d0 con 0x7fc1841580b0 2024-05-27T00:31:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc188e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc1840c51e0 con 0x7fc184157cd0 2024-05-27T00:31:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc188e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc1840cabb0 con 0x7fc1840b3cb0 2024-05-27T00:31:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc183400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc184157cd0 0x7fc1840c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc183400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc184157cd0 0x7fc1840c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50484/0 (socket says 172.21.15.98:50484) 2024-05-27T00:31:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc183400640 1 -- 172.21.15.98:0/3274386684 learned_addr learned my addr 172.21.15.98:0/3274386684 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc183e00640 1 --2- 172.21.15.98:0/3274386684 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1840b3cb0 0x7fc1840c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:00.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc183400640 1 -- 172.21.15.98:0/3274386684 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1840b3cb0 msgr2=0x7fc1840c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc183400640 1 --2- 172.21.15.98:0/3274386684 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1840b3cb0 0x7fc1840c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc183400640 1 -- 172.21.15.98:0/3274386684 >> v1:172.21.15.204:6789/0 conn(0x7fc1841580b0 legacy=0x7fc18415e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc183400640 1 -- 172.21.15.98:0/3274386684 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1840cae50 con 0x7fc184157cd0 2024-05-27T00:31:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.496+0000 7fc183e00640 1 --2- 172.21.15.98:0/3274386684 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1840b3cb0 0x7fc1840c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:31:00.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc183400640 1 --2- 172.21.15.98:0/3274386684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 0x7fc1840c5490 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc174009e50 tx=0x7fc17401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709561d0fc3df524 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc182a00640 1 -- 172.21.15.98:0/3274386684 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1740a5670 con 0x7fc184157cd0 2024-05-27T00:31:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc182a00640 1 -- 172.21.15.98:0/3274386684 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1740a5810 con 0x7fc184157cd0 2024-05-27T00:31:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc182a00640 1 -- 172.21.15.98:0/3274386684 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1740a5be0 con 0x7fc184157cd0 2024-05-27T00:31:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc188e00640 1 -- 172.21.15.98:0/3274386684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 msgr2=0x7fc1840c5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc188e00640 1 --2- 172.21.15.98:0/3274386684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 0x7fc1840c5490 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc174009e50 tx=0x7fc17401c9f0 comp rx=0 tx=0).stop 2024-05-27T00:31:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc188e00640 1 -- 172.21.15.98:0/3274386684 shutdown_connections 2024-05-27T00:31:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc188e00640 1 --2- 172.21.15.98:0/3274386684 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1840b3cb0 0x7fc1840c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc188e00640 1 --2- 172.21.15.98:0/3274386684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 0x7fc1840c5490 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc188e00640 1 -- 172.21.15.98:0/3274386684 >> 172.21.15.98:0/3274386684 conn(0x7fc184070b30 msgr2=0x7fc184070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:00.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc188e00640 1 -- 172.21.15.98:0/3274386684 shutdown_connections 2024-05-27T00:31:00.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc188e00640 1 -- 172.21.15.98:0/3274386684 wait complete. 2024-05-27T00:31:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc188e00640 1 Processor -- start 2024-05-27T00:31:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.500+0000 7fc188e00640 1 -- start start 2024-05-27T00:31:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc188e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1840b3cb0 0x7fc1841f7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:00.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc188e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 0x7fc184148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc188e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1841580b0 0x7fc18414f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc188e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc18415a830 con 0x7fc184157cd0 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc188e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc1840cb410 con 0x7fc1840b3cb0 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc188e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc18415a9b0 con 0x7fc1841580b0 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 0x7fc184148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 0x7fc184148320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50488/0 (socket says 172.21.15.98:50488) 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17aa00640 1 -- 172.21.15.98:0/92960080 learned_addr learned my addr 172.21.15.98:0/92960080 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc183400640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1840b3cb0 0x7fc1841f7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc183e00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1841580b0 0x7fc18414f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17aa00640 1 -- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1841580b0 msgr2=0x7fc18414f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17aa00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1841580b0 0x7fc18414f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17aa00640 1 -- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1840b3cb0 msgr2=0x7fc1841f7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:00.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17aa00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1840b3cb0 0x7fc1841f7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17aa00640 1 -- 172.21.15.98:0/92960080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1841f7d80 con 0x7fc184157cd0 2024-05-27T00:31:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc183e00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1841580b0 0x7fc18414f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:31:00.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17aa00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 0x7fc184148320 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00b440 tx=0x7fc16c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17be00640 1 -- 172.21.15.98:0/92960080 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c015020 con 0x7fc184157cd0 2024-05-27T00:31:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17be00640 1 -- 172.21.15.98:0/92960080 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc16c011890 con 0x7fc184157cd0 2024-05-27T00:31:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17be00640 1 -- 172.21.15.98:0/92960080 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c011c80 con 0x7fc184157cd0 2024-05-27T00:31:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc188e00640 1 -- 172.21.15.98:0/92960080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc18414ff90 con 0x7fc184157cd0 2024-05-27T00:31:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc188e00640 1 -- 172.21.15.98:0/92960080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1840c91d0 con 0x7fc184157cd0 2024-05-27T00:31:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc188e00640 1 -- 172.21.15.98:0/92960080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc184075c40 con 0x7fc184157cd0 2024-05-27T00:31:00.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.504+0000 7fc17be00640 1 -- 172.21.15.98:0/92960080 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc16c018020 con 0x7fc184157cd0 2024-05-27T00:31:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.508+0000 7fc17be00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc1600678c0 0x7fc160069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.508+0000 7fc17be00640 1 -- 172.21.15.98:0/92960080 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7fc16c085ec0 con 0x7fc184157cd0 2024-05-27T00:31:00.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.508+0000 7fc183400640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc1600678c0 0x7fc160069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:00.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.508+0000 7fc183400640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc1600678c0 0x7fc160069d80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc1740ae000 tx=0x7fc1740a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:00.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.512+0000 7fc17be00640 1 -- 172.21.15.98:0/92960080 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc16c01f030 con 0x7fc184157cd0 2024-05-27T00:31:00.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.704+0000 7fc188e00640 1 -- 172.21.15.98:0/92960080 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1840ca570 con 0x7fc184157cd0 2024-05-27T00:31:00.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.704+0000 7fc17be00640 1 -- 172.21.15.98:0/92960080 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+6007 (secure 0 0 0) 0x7fc16c00d040 con 0x7fc184157cd0 2024-05-27T00:31:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:31:00.710 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:30:55.733504+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:31:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 65 2024-05-27T00:31:00.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 -- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc1600678c0 msgr2=0x7fc160069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc1600678c0 0x7fc160069d80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc1740ae000 tx=0x7fc1740a7070 comp rx=0 tx=0).stop 2024-05-27T00:31:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 -- 172.21.15.98:0/92960080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 msgr2=0x7fc184148320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 0x7fc184148320 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00b440 tx=0x7fc16c00b900 comp rx=0 tx=0).stop 2024-05-27T00:31:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 -- 172.21.15.98:0/92960080 shutdown_connections 2024-05-27T00:31:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1841580b0 0x7fc18414f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc184157cd0 0x7fc184148320 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc1600678c0 0x7fc160069d80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 --2- 172.21.15.98:0/92960080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1840b3cb0 0x7fc1841f7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 -- 172.21.15.98:0/92960080 >> 172.21.15.98:0/92960080 conn(0x7fc184070b30 msgr2=0x7fc1840c3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 -- 172.21.15.98:0/92960080 shutdown_connections 2024-05-27T00:31:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:00.708+0000 7fc188e00640 1 -- 172.21.15.98:0/92960080 wait complete. 2024-05-27T00:31:00.730 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:31:02.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:31:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.864+0000 7fc4f9800640 1 Processor -- start 2024-05-27T00:31:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.864+0000 7fc4f9800640 1 -- start start 2024-05-27T00:31:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f9800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40cc1f0 0x7fc4f41592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f9800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f40c1f60 con 0x7fc4f4073570 2024-05-27T00:31:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f9800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f40c6dd0 con 0x7fc4f40cbe10 2024-05-27T00:31:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f9800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f4159810 con 0x7fc4f40cc1f0 2024-05-27T00:31:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f3400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc4f4073570 0x7fc4f41559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45818/0 (socket says 172.21.15.98:45818) 2024-05-27T00:31:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f3400640 1 -- 172.21.15.98:0/354899175 learned_addr learned my addr 172.21.15.98:0/354899175 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:31:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4ea000640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40cc1f0 0x7fc4f41592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 95341133 0 0) 0x7fc4f40c1f60 con 0x7fc4f4073570 2024-05-27T00:31:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4d8003530 con 0x7fc4f4073570 2024-05-27T00:31:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2892461953 0 0) 0x7fc4d8003530 con 0x7fc4f4073570 2024-05-27T00:31:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40cc1f0 msgr2=0x7fc4f41592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40cc1f0 0x7fc4f41592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 >> v1:172.21.15.204:6789/0 conn(0x7fc4f40cbe10 legacy=0x7fc4f40c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f40c70b0 con 0x7fc4f4073570 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fc4ec001a00 con 0x7fc4f4073570 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 >> v1:172.21.15.98:6789/0 conn(0x7fc4f4073570 legacy=0x7fc4f41559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d80039f0 0x7fc4d8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40cc1f0 0x7fc4d8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d8004d50 0x7fc4d80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f4159810 con 0x7fc4d80039f0 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d800bb40 con 0x7fc4d8004d50 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d800bd40 con 0x7fc4f40cc1f0 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2a00640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d80039f0 0x7fc4d8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f3400640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d8004d50 0x7fc4d80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4ea000640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40cc1f0 0x7fc4d8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2a00640 1 -- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40cc1f0 msgr2=0x7fc4d8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2a00640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40cc1f0 0x7fc4d8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2a00640 1 -- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d8004d50 msgr2=0x7fc4d80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2a00640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d8004d50 0x7fc4d80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f2a00640 1 -- 172.21.15.98:0/354899175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d8003530 con 0x7fc4d80039f0 2024-05-27T00:31:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4f3400640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d8004d50 0x7fc4d80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:31:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.868+0000 7fc4ea000640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40cc1f0 0x7fc4d8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:31:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f2a00640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d80039f0 0x7fc4d8003e10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc001750 tx=0x7fc4dc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1bd185644206213 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0013a0 con 0x7fc4d80039f0 2024-05-27T00:31:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4dc0052f0 con 0x7fc4d80039f0 2024-05-27T00:31:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f2000640 1 -- 172.21.15.98:0/354899175 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc003840 con 0x7fc4d80039f0 2024-05-27T00:31:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f9800640 1 -- 172.21.15.98:0/354899175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d80039f0 msgr2=0x7fc4d8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f9800640 1 --2- 172.21.15.98:0/354899175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4d80039f0 0x7fc4d8003e10 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc001750 tx=0x7fc4dc01c3e0 comp rx=0 tx=0).stop 2024-05-27T00:31:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f3400640 1 -- 172.21.15.98:0/354899175 reap_dead start 2024-05-27T00:31:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f9800640 1 -- 172.21.15.98:0/354899175 shutdown_connections 2024-05-27T00:31:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f9800640 1 -- 172.21.15.98:0/354899175 >> 172.21.15.98:0/354899175 conn(0x7fc4f4067530 msgr2=0x7fc4f406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f9800640 1 -- 172.21.15.98:0/354899175 shutdown_connections 2024-05-27T00:31:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.872+0000 7fc4f9800640 1 -- 172.21.15.98:0/354899175 wait complete. 2024-05-27T00:31:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 Processor -- start 2024-05-27T00:31:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 -- start start 2024-05-27T00:31:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d80039f0 0x7fc4f41f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40729e0 0x7fc4f4156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f4148e30 0x7fc4f4157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f40c1330 con 0x7fc4f4148e30 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f40c1050 con 0x7fc4d80039f0 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f40c79d0 con 0x7fc4f40729e0 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f4148e30 0x7fc4f4157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f4148e30 0x7fc4f4157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50514/0 (socket says 172.21.15.98:50514) 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f3400640 1 -- 172.21.15.98:0/3542826815 learned_addr learned my addr 172.21.15.98:0/3542826815 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4ea000640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40729e0 0x7fc4f4156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f2a00640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d80039f0 0x7fc4f41f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f3400640 1 -- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40729e0 msgr2=0x7fc4f4156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f3400640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40729e0 0x7fc4f4156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f3400640 1 -- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d80039f0 msgr2=0x7fc4f41f3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f3400640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d80039f0 0x7fc4f41f3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f3400640 1 -- 172.21.15.98:0/3542826815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f406ecb0 con 0x7fc4f4148e30 2024-05-27T00:31:02.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f2a00640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d80039f0 0x7fc4f41f3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:31:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f3400640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f4148e30 0x7fc4f4157350 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec011790 tx=0x7fc4ec011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4eb400640 1 -- 172.21.15.98:0/3542826815 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec010040 con 0x7fc4f4148e30 2024-05-27T00:31:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4eb400640 1 -- 172.21.15.98:0/3542826815 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4ec01a070 con 0x7fc4f4148e30 2024-05-27T00:31:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4eb400640 1 -- 172.21.15.98:0/3542826815 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec00fb10 con 0x7fc4f4148e30 2024-05-27T00:31:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 -- 172.21.15.98:0/3542826815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f41f3ad0 con 0x7fc4f4148e30 2024-05-27T00:31:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 -- 172.21.15.98:0/3542826815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f4152150 con 0x7fc4f4148e30 2024-05-27T00:31:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.876+0000 7fc4f9800640 1 -- 172.21.15.98:0/3542826815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f4075cb0 con 0x7fc4f4148e30 2024-05-27T00:31:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.880+0000 7fc4eb400640 1 -- 172.21.15.98:0/3542826815 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc4ec00fcb0 con 0x7fc4f4148e30 2024-05-27T00:31:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.880+0000 7fc4eb400640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d00678f0 0x7fc4d0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.880+0000 7fc4eb400640 1 -- 172.21.15.98:0/3542826815 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7fc4ec088e90 con 0x7fc4f4148e30 2024-05-27T00:31:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.880+0000 7fc4f2a00640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d00678f0 0x7fc4d0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:02.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.880+0000 7fc4f2a00640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d00678f0 0x7fc4d0069db0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80057f0 tx=0x7fc4dc0011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:02.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:02.884+0000 7fc4eb400640 1 -- 172.21.15.98:0/3542826815 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc4ec059470 con 0x7fc4f4148e30 2024-05-27T00:31:03.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 -- 172.21.15.98:0/3542826815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4f4156bf0 con 0x7fc4f4148e30 2024-05-27T00:31:03.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4eb400640 1 -- 172.21.15.98:0/3542826815 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6013 (secure 0 0 0) 0x7fc4ec08a020 con 0x7fc4f4148e30 2024-05-27T00:31:03.074 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:31:03.074 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:02.340148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:clientreplay","state_seq":28,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:31:03.075 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 66 2024-05-27T00:31:03.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 -- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d00678f0 msgr2=0x7fc4d0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:03.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d00678f0 0x7fc4d0069db0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80057f0 tx=0x7fc4dc0011a0 comp rx=0 tx=0).stop 2024-05-27T00:31:03.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 -- 172.21.15.98:0/3542826815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f4148e30 msgr2=0x7fc4f4157350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:03.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f4148e30 0x7fc4f4157350 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec011790 tx=0x7fc4ec011c50 comp rx=0 tx=0).stop 2024-05-27T00:31:03.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 -- 172.21.15.98:0/3542826815 shutdown_connections 2024-05-27T00:31:03.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4f4148e30 0x7fc4f4157350 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:03.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f40729e0 0x7fc4f4156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:03.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc4d00678f0 0x7fc4d0069db0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:03.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 --2- 172.21.15.98:0/3542826815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4d80039f0 0x7fc4f41f3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:03.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 -- 172.21.15.98:0/3542826815 >> 172.21.15.98:0/3542826815 conn(0x7fc4f4067530 msgr2=0x7fc4f406b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:03.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 -- 172.21.15.98:0/3542826815 shutdown_connections 2024-05-27T00:31:03.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:03.072+0000 7fc4f9800640 1 -- 172.21.15.98:0/3542826815 wait complete. 2024-05-27T00:31:03.095 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:31:05.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:31:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dfe00640 1 Processor -- start 2024-05-27T00:31:05.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dfe00640 1 -- start start 2024-05-27T00:31:05.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dfe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19d80731f0 0x7f19d80c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dfe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19d80c2220 0x7f19d8151310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dfe00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d80c28f0 con 0x7f19d80735d0 2024-05-27T00:31:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dfe00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f19d8071a90 con 0x7f19d80731f0 2024-05-27T00:31:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dfe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f19d8071600 con 0x7f19d80c2220 2024-05-27T00:31:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dd400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19d80731f0 0x7f19d80c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dd400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19d80731f0 0x7f19d80c79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51240/0 (socket says 172.21.15.98:51240) 2024-05-27T00:31:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dd400640 1 -- 172.21.15.98:0/564142923 learned_addr learned my addr 172.21.15.98:0/564142923 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dca00640 1 --1- 172.21.15.98:0/564142923 >> v1:172.21.15.98:6789/0 conn(0x7f19d80735d0 0x7f19d815aba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38662/0 (socket says 172.21.15.98:38662) 2024-05-27T00:31:05.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.240+0000 7f19dde00640 1 --2- 172.21.15.98:0/564142923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19d80c2220 0x7f19d8151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dd400640 1 -- 172.21.15.98:0/564142923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19d80c2220 msgr2=0x7f19d8151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dd400640 1 --2- 172.21.15.98:0/564142923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19d80c2220 0x7f19d8151310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dd400640 1 -- 172.21.15.98:0/564142923 >> v1:172.21.15.98:6789/0 conn(0x7f19d80735d0 legacy=0x7f19d815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dd400640 1 -- 172.21.15.98:0/564142923 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d80c2bd0 con 0x7f19d80731f0 2024-05-27T00:31:05.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19d7e00640 1 -- 172.21.15.98:0/564142923 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1270790726 0 0) 0x7f19d80c28f0 con 0x7f19d80735d0 2024-05-27T00:31:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dd400640 1 --2- 172.21.15.98:0/564142923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 0x7f19d80c79f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009e50 tx=0x7f19cc01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602251c259c392ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19d7e00640 1 -- 172.21.15.98:0/564142923 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0a58d0 con 0x7f19d80731f0 2024-05-27T00:31:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19d7e00640 1 -- 172.21.15.98:0/564142923 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19cc0a5a70 con 0x7f19d80731f0 2024-05-27T00:31:05.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19d7e00640 1 -- 172.21.15.98:0/564142923 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0a5e40 con 0x7f19d80731f0 2024-05-27T00:31:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dfe00640 1 -- 172.21.15.98:0/564142923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 msgr2=0x7f19d80c79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dfe00640 1 --2- 172.21.15.98:0/564142923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 0x7f19d80c79f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009e50 tx=0x7f19cc01ca00 comp rx=0 tx=0).stop 2024-05-27T00:31:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dfe00640 1 -- 172.21.15.98:0/564142923 shutdown_connections 2024-05-27T00:31:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dfe00640 1 --2- 172.21.15.98:0/564142923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19d80c2220 0x7f19d8151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dfe00640 1 --2- 172.21.15.98:0/564142923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 0x7f19d80c79f0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.244+0000 7f19dfe00640 1 -- 172.21.15.98:0/564142923 >> 172.21.15.98:0/564142923 conn(0x7f19d8067530 msgr2=0x7f19d806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 -- 172.21.15.98:0/564142923 shutdown_connections 2024-05-27T00:31:05.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 -- 172.21.15.98:0/564142923 wait complete. 2024-05-27T00:31:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 Processor -- start 2024-05-27T00:31:05.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 -- start start 2024-05-27T00:31:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 0x7f19d80c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19d80735d0 0x7f19d8163250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19d80c2220 0x7f19d80c38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19d815b2b0 con 0x7f19d80731f0 2024-05-27T00:31:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f19d8151c60 con 0x7f19d80c2220 2024-05-27T00:31:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f19d815b430 con 0x7f19d80735d0 2024-05-27T00:31:05.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dd400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 0x7f19d80c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dd400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 0x7f19d80c52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51250/0 (socket says 172.21.15.98:51250) 2024-05-27T00:31:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dd400640 1 -- 172.21.15.98:0/685984047 learned_addr learned my addr 172.21.15.98:0/685984047 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dca00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19d80735d0 0x7f19d8163250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dde00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19d80c2220 0x7f19d80c38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dd400640 1 -- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19d80735d0 msgr2=0x7f19d8163250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dd400640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19d80735d0 0x7f19d8163250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dd400640 1 -- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19d80c2220 msgr2=0x7f19d80c38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dd400640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19d80c2220 0x7f19d80c38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dd400640 1 -- 172.21.15.98:0/685984047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d8163c20 con 0x7f19d80731f0 2024-05-27T00:31:05.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dde00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19d80c2220 0x7f19d80c38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:31:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dd400640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 0x7f19d80c52d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f19cc01a630 tx=0x7f19cc0197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19d5600640 1 -- 172.21.15.98:0/685984047 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0a58d0 con 0x7f19d80731f0 2024-05-27T00:31:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19d5600640 1 -- 172.21.15.98:0/685984047 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19cc0be070 con 0x7f19d80731f0 2024-05-27T00:31:05.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19dfe00640 1 -- 172.21.15.98:0/685984047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d8163ec0 con 0x7f19d80731f0 2024-05-27T00:31:05.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.248+0000 7f19d5600640 1 -- 172.21.15.98:0/685984047 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc009510 con 0x7f19d80731f0 2024-05-27T00:31:05.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.252+0000 7f19dfe00640 1 -- 172.21.15.98:0/685984047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d80c33d0 con 0x7f19d80731f0 2024-05-27T00:31:05.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.252+0000 7f19dfe00640 1 -- 172.21.15.98:0/685984047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d8075cb0 con 0x7f19d80731f0 2024-05-27T00:31:05.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.252+0000 7f19d5600640 1 -- 172.21.15.98:0/685984047 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f19cc0b86b0 con 0x7f19d80731f0 2024-05-27T00:31:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.252+0000 7f19d5600640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f19b80678c0 0x7f19b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.252+0000 7f19d5600640 1 -- 172.21.15.98:0/685984047 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f19cc121580 con 0x7f19d80731f0 2024-05-27T00:31:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.252+0000 7f19dca00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f19b80678c0 0x7f19b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:05.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.256+0000 7f19dca00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f19b80678c0 0x7f19b8069d80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f19d81645c0 tx=0x7f19c4007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:05.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.256+0000 7f19d5600640 1 -- 172.21.15.98:0/685984047 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f19cc016620 con 0x7f19d80731f0 2024-05-27T00:31:05.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.448+0000 7f19dfe00640 1 -- 172.21.15.98:0/685984047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19d80c1fe0 con 0x7f19d80731f0 2024-05-27T00:31:05.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19d5600640 1 -- 172.21.15.98:0/685984047 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6013 (secure 0 0 0) 0x7f19cc0f1c90 con 0x7f19d80731f0 2024-05-27T00:31:05.454 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:31:05.454 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:02.340148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:clientreplay","state_seq":28,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:31:05.455 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 66 2024-05-27T00:31:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 -- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f19b80678c0 msgr2=0x7f19b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f19b80678c0 0x7f19b8069d80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f19d81645c0 tx=0x7f19c4007df0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 -- 172.21.15.98:0/685984047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 msgr2=0x7f19d80c52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 0x7f19d80c52d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f19cc01a630 tx=0x7f19cc0197a0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 -- 172.21.15.98:0/685984047 shutdown_connections 2024-05-27T00:31:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f19b80678c0 0x7f19b8069d80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19d80c2220 0x7f19d80c38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19d80735d0 0x7f19d8163250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 --2- 172.21.15.98:0/685984047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19d80731f0 0x7f19d80c52d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 -- 172.21.15.98:0/685984047 >> 172.21.15.98:0/685984047 conn(0x7f19d8067530 msgr2=0x7f19d80c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 -- 172.21.15.98:0/685984047 shutdown_connections 2024-05-27T00:31:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:05.452+0000 7f19dfe00640 1 -- 172.21.15.98:0/685984047 wait complete. 2024-05-27T00:31:05.475 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 66, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 66, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6240": { "addr": "172.21.15.98:6834/3070334683", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6832", "nonce": 3070334683, "type": "v2" }, { "addr": "172.21.15.98:6834", "nonce": 3070334683, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6240, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:clientreplay", "state_seq": 28 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:31:02.340148+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6240 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:31:07.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:31:07.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c9800640 1 Processor -- start 2024-05-27T00:31:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c9800640 1 -- start start 2024-05-27T00:31:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c9800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1c40b3260 0x7fe1c40c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c9800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe1c4070ad0 0x7fe1c40c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c9800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c4071010 con 0x7fe1c4070660 2024-05-27T00:31:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c9800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe1c40c3890 con 0x7fe1c40b3260 2024-05-27T00:31:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c9800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fe1c40c88c0 con 0x7fe1c4070ad0 2024-05-27T00:31:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c3e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1c40b3260 0x7fe1c40c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c3e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1c40b3260 0x7fe1c40c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51258/0 (socket says 172.21.15.98:51258) 2024-05-27T00:31:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c3e00640 1 -- 172.21.15.98:0/4084394481 learned_addr learned my addr 172.21.15.98:0/4084394481 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c8e00640 1 --2- 172.21.15.98:0/4084394481 >> v2:172.21.15.204:3300/0 conn(0x7fe1c4070ad0 0x7fe1c40c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c3e00640 1 -- 172.21.15.98:0/4084394481 >> v2:172.21.15.204:3300/0 conn(0x7fe1c4070ad0 msgr2=0x7fe1c40c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c3e00640 1 --2- 172.21.15.98:0/4084394481 >> v2:172.21.15.204:3300/0 conn(0x7fe1c4070ad0 0x7fe1c40c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c3e00640 1 -- 172.21.15.98:0/4084394481 >> v1:172.21.15.204:6789/0 conn(0x7fe1c4070660 legacy=0x7fe1c40c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.624+0000 7fe1c3e00640 1 -- 172.21.15.98:0/4084394481 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c40c8b80 con 0x7fe1c40b3260 2024-05-27T00:31:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c3e00640 1 --2- 172.21.15.98:0/4084394481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40b3260 0x7fe1c40c7cc0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40cee80 tx=0x7fe1b401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ac5ce0f75c611c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c2a00640 1 -- 172.21.15.98:0/4084394481 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40b5070 con 0x7fe1c40b3260 2024-05-27T00:31:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c2a00640 1 -- 172.21.15.98:0/4084394481 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b40a8a10 con 0x7fe1c40b3260 2024-05-27T00:31:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c2a00640 1 -- 172.21.15.98:0/4084394481 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40a8de0 con 0x7fe1c40b3260 2024-05-27T00:31:07.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c9800640 1 -- 172.21.15.98:0/4084394481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40b3260 msgr2=0x7fe1c40c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:07.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c9800640 1 --2- 172.21.15.98:0/4084394481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40b3260 0x7fe1c40c7cc0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40cee80 tx=0x7fe1b401ed10 comp rx=0 tx=0).stop 2024-05-27T00:31:07.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c9800640 1 -- 172.21.15.98:0/4084394481 shutdown_connections 2024-05-27T00:31:07.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c9800640 1 --2- 172.21.15.98:0/4084394481 >> v2:172.21.15.204:3300/0 conn(0x7fe1c4070ad0 0x7fe1c40c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:07.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c9800640 1 --2- 172.21.15.98:0/4084394481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40b3260 0x7fe1c40c7cc0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:07.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c9800640 1 -- 172.21.15.98:0/4084394481 >> 172.21.15.98:0/4084394481 conn(0x7fe1c40cdeb0 msgr2=0x7fe1c40ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:07.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c9800640 1 -- 172.21.15.98:0/4084394481 shutdown_connections 2024-05-27T00:31:07.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.628+0000 7fe1c9800640 1 -- 172.21.15.98:0/4084394481 wait complete. 2024-05-27T00:31:07.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c9800640 1 Processor -- start 2024-05-27T00:31:07.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c9800640 1 -- start start 2024-05-27T00:31:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c9800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c4070660 0x7fe1c406c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c9800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1c4070ad0 0x7fe1c41d9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c9800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1c41e14d0 0x7fe1c41da320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c9800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c40c9420 con 0x7fe1c4070660 2024-05-27T00:31:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c9800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c40c9140 con 0x7fe1c41e14d0 2024-05-27T00:31:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c9800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe1c40c8200 con 0x7fe1c4070ad0 2024-05-27T00:31:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c4070660 0x7fe1c406c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c4070660 0x7fe1c406c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51268/0 (socket says 172.21.15.98:51268) 2024-05-27T00:31:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c3e00640 1 -- 172.21.15.98:0/1118419262 learned_addr learned my addr 172.21.15.98:0/1118419262 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:07.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c3400640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1c4070ad0 0x7fe1c41d9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c8e00640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1c41e14d0 0x7fe1c41da320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c3e00640 1 -- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1c4070ad0 msgr2=0x7fe1c41d9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c3e00640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1c4070ad0 0x7fe1c41d9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c3e00640 1 -- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1c41e14d0 msgr2=0x7fe1c41da320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c3e00640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1c41e14d0 0x7fe1c41da320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c3e00640 1 -- 172.21.15.98:0/1118419262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c406c9d0 con 0x7fe1c4070660 2024-05-27T00:31:07.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c3e00640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c4070660 0x7fe1c406c490 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe1b40b3040 tx=0x7fe1b401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1bbe00640 1 -- 172.21.15.98:0/1118419262 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40b5070 con 0x7fe1c4070660 2024-05-27T00:31:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1bbe00640 1 -- 172.21.15.98:0/1118419262 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b40b0b30 con 0x7fe1c4070660 2024-05-27T00:31:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1bbe00640 1 -- 172.21.15.98:0/1118419262 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40bb410 con 0x7fe1c4070660 2024-05-27T00:31:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c9800640 1 -- 172.21.15.98:0/1118419262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c40cb550 con 0x7fe1c4070660 2024-05-27T00:31:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.632+0000 7fe1c9800640 1 -- 172.21.15.98:0/1118419262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1c41dd7f0 con 0x7fe1c4070660 2024-05-27T00:31:07.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.636+0000 7fe1c9800640 1 -- 172.21.15.98:0/1118419262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1c4075c40 con 0x7fe1c4070660 2024-05-27T00:31:07.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.636+0000 7fe1bbe00640 1 -- 172.21.15.98:0/1118419262 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe1b40b9030 con 0x7fe1c4070660 2024-05-27T00:31:07.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.636+0000 7fe1bbe00640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1a40678c0 0x7fe1a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:07.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.636+0000 7fe1bbe00640 1 -- 172.21.15.98:0/1118419262 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7fe1b41231f0 con 0x7fe1c4070660 2024-05-27T00:31:07.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.636+0000 7fe1c3400640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1a40678c0 0x7fe1a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:07.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.636+0000 7fe1c3400640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1a40678c0 0x7fe1a4069d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe1c406fde0 tx=0x7fe1b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:07.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.640+0000 7fe1bbe00640 1 -- 172.21.15.98:0/1118419262 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fe1b4018870 con 0x7fe1c4070660 2024-05-27T00:31:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.880+0000 7fe1c9800640 1 -- 172.21.15.98:0/1118419262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1c40cb7d0 con 0x7fe1c4070660 2024-05-27T00:31:07.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.880+0000 7fe1bbe00640 1 -- 172.21.15.98:0/1118419262 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6013 (secure 0 0 0) 0x7fe1b40f38f0 con 0x7fe1c4070660 2024-05-27T00:31:07.884 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:31:07.884 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:02.340148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:clientreplay","state_seq":28,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:31:07.884 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 66 2024-05-27T00:31:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 -- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1a40678c0 msgr2=0x7fe1a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1a40678c0 0x7fe1a4069d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe1c406fde0 tx=0x7fe1b0009040 comp rx=0 tx=0).stop 2024-05-27T00:31:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 -- 172.21.15.98:0/1118419262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c4070660 msgr2=0x7fe1c406c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c4070660 0x7fe1c406c490 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe1b40b3040 tx=0x7fe1b401bab0 comp rx=0 tx=0).stop 2024-05-27T00:31:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 -- 172.21.15.98:0/1118419262 shutdown_connections 2024-05-27T00:31:07.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe1a40678c0 0x7fe1a4069d80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1c41e14d0 0x7fe1c41da320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1c4070ad0 0x7fe1c41d9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 --2- 172.21.15.98:0/1118419262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c4070660 0x7fe1c406c490 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 -- 172.21.15.98:0/1118419262 >> 172.21.15.98:0/1118419262 conn(0x7fe1c40cdeb0 msgr2=0x7fe1c41e1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 -- 172.21.15.98:0/1118419262 shutdown_connections 2024-05-27T00:31:07.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:07.884+0000 7fe1c9800640 1 -- 172.21.15.98:0/1118419262 wait complete. 2024-05-27T00:31:07.919 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:31:09.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:31:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa18e00640 1 Processor -- start 2024-05-27T00:31:10.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa18e00640 1 -- start start 2024-05-27T00:31:10.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa18e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faa14071560 0x7faa1415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:10.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa18e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa14158d10 con 0x7faa14152850 2024-05-27T00:31:10.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa18e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa14158fe0 con 0x7faa140c2fb0 2024-05-27T00:31:10.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa18e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7faa140719f0 con 0x7faa14071560 2024-05-27T00:31:10.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa11e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7faa14152850 0x7faa14158600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38672/0 (socket says 172.21.15.98:38672) 2024-05-27T00:31:10.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa11e00640 1 -- 172.21.15.98:0/910993414 learned_addr learned my addr 172.21.15.98:0/910993414 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:31:10.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa12800640 1 --2- 172.21.15.98:0/910993414 >> v2:172.21.15.98:3300/0 conn(0x7faa14071560 0x7faa1415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:10.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa12800640 1 -- 172.21.15.98:0/910993414 >> v1:172.21.15.204:6789/0 conn(0x7faa140c2fb0 legacy=0x7faa140c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa10a00640 1 -- 172.21.15.98:0/910993414 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1472095965 0 0) 0x7faa14158d10 con 0x7faa14152850 2024-05-27T00:31:10.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa12800640 1 -- 172.21.15.98:0/910993414 >> v1:172.21.15.98:6789/0 conn(0x7faa14152850 legacy=0x7faa14158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.056+0000 7faa12800640 1 -- 172.21.15.98:0/910993414 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa141592c0 con 0x7faa14071560 2024-05-27T00:31:10.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa12800640 1 --2- 172.21.15.98:0/910993414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14071560 0x7faa1415c600 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009d90 tx=0x7faa0c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade90e22879c7d0d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:10.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa10a00640 1 -- 172.21.15.98:0/910993414 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0b2070 con 0x7faa14071560 2024-05-27T00:31:10.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa10a00640 1 -- 172.21.15.98:0/910993414 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa0c0a59c0 con 0x7faa14071560 2024-05-27T00:31:10.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa10a00640 1 -- 172.21.15.98:0/910993414 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0a5d90 con 0x7faa14071560 2024-05-27T00:31:10.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa18e00640 1 -- 172.21.15.98:0/910993414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14071560 msgr2=0x7faa1415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa18e00640 1 --2- 172.21.15.98:0/910993414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14071560 0x7faa1415c600 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009d90 tx=0x7faa0c01c8e0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa18e00640 1 -- 172.21.15.98:0/910993414 shutdown_connections 2024-05-27T00:31:10.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa18e00640 1 --2- 172.21.15.98:0/910993414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14071560 0x7faa1415c600 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa18e00640 1 -- 172.21.15.98:0/910993414 >> 172.21.15.98:0/910993414 conn(0x7faa140cdf90 msgr2=0x7faa140ce3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:10.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa18e00640 1 -- 172.21.15.98:0/910993414 shutdown_connections 2024-05-27T00:31:10.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa18e00640 1 -- 172.21.15.98:0/910993414 wait complete. 2024-05-27T00:31:10.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa18e00640 1 Processor -- start 2024-05-27T00:31:10.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.060+0000 7faa18e00640 1 -- start start 2024-05-27T00:31:10.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa18e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa14071560 0x7faa141546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:10.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa18e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa140c2fb0 0x7faa141531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:10.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa18e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14152850 0x7faa141f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:10.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa18e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa141595a0 con 0x7faa14152850 2024-05-27T00:31:10.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa18e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa140c4160 con 0x7faa14071560 2024-05-27T00:31:10.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa18e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faa14152c30 con 0x7faa140c2fb0 2024-05-27T00:31:10.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa12800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14152850 0x7faa141f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa12800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14152850 0x7faa141f0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51282/0 (socket says 172.21.15.98:51282) 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa12800640 1 -- 172.21.15.98:0/2749110008 learned_addr learned my addr 172.21.15.98:0/2749110008 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa11400640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa140c2fb0 0x7faa141531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa11e00640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa14071560 0x7faa141546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa12800640 1 -- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa140c2fb0 msgr2=0x7faa141531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa12800640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa140c2fb0 0x7faa141531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa12800640 1 -- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa14071560 msgr2=0x7faa141546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa12800640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa14071560 0x7faa141546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa12800640 1 -- 172.21.15.98:0/2749110008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa14154f40 con 0x7faa14152850 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa11e00640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa14071560 0x7faa141546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa12800640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14152850 0x7faa141f0c80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7faa0c01a510 tx=0x7faa0c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa0a000640 1 -- 172.21.15.98:0/2749110008 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0bd020 con 0x7faa14152850 2024-05-27T00:31:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa0a000640 1 -- 172.21.15.98:0/2749110008 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa0c0ad6c0 con 0x7faa14152850 2024-05-27T00:31:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa18e00640 1 -- 172.21.15.98:0/2749110008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa141551c0 con 0x7faa14152850 2024-05-27T00:31:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa0a000640 1 -- 172.21.15.98:0/2749110008 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0b2040 con 0x7faa14152850 2024-05-27T00:31:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa18e00640 1 -- 172.21.15.98:0/2749110008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa141f44d0 con 0x7faa14152850 2024-05-27T00:31:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa18e00640 1 -- 172.21.15.98:0/2749110008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa14075cb0 con 0x7faa14152850 2024-05-27T00:31:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.064+0000 7faa0a000640 1 -- 172.21.15.98:0/2749110008 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faa0c0c0020 con 0x7faa14152850 2024-05-27T00:31:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.068+0000 7faa0a000640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9f40678f0 0x7fa9f4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.068+0000 7faa0a000640 1 -- 172.21.15.98:0/2749110008 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7faa0c120fe0 con 0x7faa14152850 2024-05-27T00:31:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.068+0000 7faa11e00640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9f40678f0 0x7fa9f4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.068+0000 7faa11e00640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9f40678f0 0x7fa9f4069db0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faa04000f70 tx=0x7faa04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.072+0000 7faa0a000640 1 -- 172.21.15.98:0/2749110008 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7faa0c0c7030 con 0x7faa14152850 2024-05-27T00:31:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.260+0000 7faa18e00640 1 -- 172.21.15.98:0/2749110008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa140701a0 con 0x7faa14152850 2024-05-27T00:31:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa0a000640 1 -- 172.21.15.98:0/2749110008 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6014 (secure 0 0 0) 0x7faa14154f40 con 0x7faa14152850 2024-05-27T00:31:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:31:10.266 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:09.534234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":30,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6240,"qdb_cluster":[6240]},"id":1}]} 2024-05-27T00:31:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 67 2024-05-27T00:31:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 -- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9f40678f0 msgr2=0x7fa9f4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9f40678f0 0x7fa9f4069db0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faa04000f70 tx=0x7faa04009040 comp rx=0 tx=0).stop 2024-05-27T00:31:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 -- 172.21.15.98:0/2749110008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14152850 msgr2=0x7faa141f0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14152850 0x7faa141f0c80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7faa0c01a510 tx=0x7faa0c019680 comp rx=0 tx=0).stop 2024-05-27T00:31:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 -- 172.21.15.98:0/2749110008 shutdown_connections 2024-05-27T00:31:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa14152850 0x7faa141f0c80 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa140c2fb0 0x7faa141531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa9f40678f0 0x7fa9f4069db0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 --2- 172.21.15.98:0/2749110008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa14071560 0x7faa141546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 -- 172.21.15.98:0/2749110008 >> 172.21.15.98:0/2749110008 conn(0x7faa140cdf90 msgr2=0x7faa14151c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 -- 172.21.15.98:0/2749110008 shutdown_connections 2024-05-27T00:31:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.264+0000 7faa18e00640 1 -- 172.21.15.98:0/2749110008 wait complete. 2024-05-27T00:31:10.287 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6078 2024-05-27T00:31:10.287 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2024-05-27T00:31:10.287 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-05-27T00:31:10.287 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-27T00:31:10.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-27T00:31:10.291 INFO:tasks.ceph.mds.c:Started 2024-05-27T00:31:10.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:31:10.375 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2024-05-27T00:31:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.424+0000 7f4514a00640 1 Processor -- start 2024-05-27T00:31:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.424+0000 7f4514a00640 1 -- start start 2024-05-27T00:31:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f4514a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4510073570 0x7f45101592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f4514a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45100c1f60 con 0x7f45100cbe10 2024-05-27T00:31:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f4514a00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45100c6dd0 con 0x7f45100cc1f0 2024-05-27T00:31:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f4514a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4510159810 con 0x7f4510073570 2024-05-27T00:31:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f450de00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f45100cbe10 0x7f45100c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38698/0 (socket says 172.21.15.98:38698) 2024-05-27T00:31:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f450de00640 1 -- 172.21.15.98:0/1990179569 learned_addr learned my addr 172.21.15.98:0/1990179569 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:31:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f450e800640 1 --2- 172.21.15.98:0/1990179569 >> v2:172.21.15.98:3300/0 conn(0x7f4510073570 0x7f45101592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:10.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f450e800640 1 -- 172.21.15.98:0/1990179569 >> v1:172.21.15.204:6789/0 conn(0x7f45100cc1f0 legacy=0x7f45101559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f450e800640 1 -- 172.21.15.98:0/1990179569 >> v1:172.21.15.98:6789/0 conn(0x7f45100cbe10 legacy=0x7f45100c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f450e800640 1 -- 172.21.15.98:0/1990179569 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45100c70b0 con 0x7f4510073570 2024-05-27T00:31:10.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.428+0000 7f450d400640 1 -- 172.21.15.98:0/1990179569 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250491703 0 0) 0x7f45100c1f60 con 0x7f45100cbe10 2024-05-27T00:31:10.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f450e800640 1 --2- 172.21.15.98:0/1990179569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 0x7f45101592d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4508009d90 tx=0x7f450801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2fea2694aba57ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:10.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f450d400640 1 -- 172.21.15.98:0/1990179569 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45080b4070 con 0x7f4510073570 2024-05-27T00:31:10.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f450d400640 1 -- 172.21.15.98:0/1990179569 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45080af600 con 0x7f4510073570 2024-05-27T00:31:10.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f450d400640 1 -- 172.21.15.98:0/1990179569 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45080af9d0 con 0x7f4510073570 2024-05-27T00:31:10.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f4514a00640 1 -- 172.21.15.98:0/1990179569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 msgr2=0x7f45101592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f4514a00640 1 --2- 172.21.15.98:0/1990179569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 0x7f45101592d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4508009d90 tx=0x7f450801f150 comp rx=0 tx=0).stop 2024-05-27T00:31:10.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f4514a00640 1 -- 172.21.15.98:0/1990179569 shutdown_connections 2024-05-27T00:31:10.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f4514a00640 1 --2- 172.21.15.98:0/1990179569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 0x7f45101592d0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f4514a00640 1 -- 172.21.15.98:0/1990179569 >> 172.21.15.98:0/1990179569 conn(0x7f4510067530 msgr2=0x7f451006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:10.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f4514a00640 1 -- 172.21.15.98:0/1990179569 shutdown_connections 2024-05-27T00:31:10.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f4514a00640 1 -- 172.21.15.98:0/1990179569 wait complete. 2024-05-27T00:31:10.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f4514a00640 1 Processor -- start 2024-05-27T00:31:10.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.432+0000 7f4514a00640 1 -- start start 2024-05-27T00:31:10.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4514a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 0x7f451014e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:10.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4514a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45100cbe10 0x7f45100721e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:10.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4514a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45100cc1f0 0x7f4510072720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:10.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4514a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45100c1330 con 0x7f4510073570 2024-05-27T00:31:10.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4514a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f45100c1050 con 0x7f45100cbe10 2024-05-27T00:31:10.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4514a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f45100c79d0 con 0x7f45100cc1f0 2024-05-27T00:31:10.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f450de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 0x7f451014e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:10.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f450de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 0x7f451014e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51316/0 (socket says 172.21.15.98:51316) 2024-05-27T00:31:10.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f450de00640 1 -- 172.21.15.98:0/4185460070 learned_addr learned my addr 172.21.15.98:0/4185460070 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:10.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f450e800640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45100cc1f0 0x7f4510072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:10.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4507e00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45100cbe10 0x7f45100721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:10.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f450de00640 1 -- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45100cc1f0 msgr2=0x7f4510072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f450de00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45100cc1f0 0x7f4510072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f450de00640 1 -- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45100cbe10 msgr2=0x7f45100721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f450de00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45100cbe10 0x7f45100721e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f450de00640 1 -- 172.21.15.98:0/4185460070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4510154040 con 0x7f4510073570 2024-05-27T00:31:10.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4507e00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45100cbe10 0x7f45100721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:31:10.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f450de00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 0x7f451014e310 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f450000c4a0 tx=0x7f450000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:10.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4506000640 1 -- 172.21.15.98:0/4185460070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500018070 con 0x7f4510073570 2024-05-27T00:31:10.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4506000640 1 -- 172.21.15.98:0/4185460070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45000074e0 con 0x7f4510073570 2024-05-27T00:31:10.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4506000640 1 -- 172.21.15.98:0/4185460070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500004a30 con 0x7f4510073570 2024-05-27T00:31:10.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4514a00640 1 -- 172.21.15.98:0/4185460070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4510070f40 con 0x7f4510073570 2024-05-27T00:31:10.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4514a00640 1 -- 172.21.15.98:0/4185460070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f451014e850 con 0x7f4510073570 2024-05-27T00:31:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4514a00640 1 -- 172.21.15.98:0/4185460070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4510075cb0 con 0x7f4510073570 2024-05-27T00:31:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.436+0000 7f4506000640 1 -- 172.21.15.98:0/4185460070 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4500004bd0 con 0x7f4510073570 2024-05-27T00:31:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.440+0000 7f4506000640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44ec0678c0 0x7f44ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:10.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.440+0000 7f4506000640 1 -- 172.21.15.98:0/4185460070 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f4500085420 con 0x7f4510073570 2024-05-27T00:31:10.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.440+0000 7f4507e00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44ec0678c0 0x7f44ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:10.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.440+0000 7f4507e00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44ec0678c0 0x7f44ec069d80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4510153190 tx=0x7f44f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:10.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.444+0000 7f4506000640 1 -- 172.21.15.98:0/4185460070 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4500055b20 con 0x7f4510073570 2024-05-27T00:31:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 -- 172.21.15.98:0/4185460070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45100b2110 con 0x7f4510073570 2024-05-27T00:31:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4506000640 1 -- 172.21.15.98:0/4185460070 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6839 (secure 0 0 0) 0x7f4500055420 con 0x7f4510073570 2024-05-27T00:31:10.639 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:31:10.639 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:09.534234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":30,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6240,"qdb_cluster":[6240]},"id":1}]} 2024-05-27T00:31:10.639 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 68 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 -- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44ec0678c0 msgr2=0x7f44ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44ec0678c0 0x7f44ec069d80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4510153190 tx=0x7f44f8009040 comp rx=0 tx=0).stop 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 -- 172.21.15.98:0/4185460070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 msgr2=0x7f451014e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 0x7f451014e310 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f450000c4a0 tx=0x7f450000c960 comp rx=0 tx=0).stop 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 -- 172.21.15.98:0/4185460070 shutdown_connections 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45100cc1f0 0x7f4510072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f44ec0678c0 0x7f44ec069d80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45100cbe10 0x7f45100721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 --2- 172.21.15.98:0/4185460070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4510073570 0x7f451014e310 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 -- 172.21.15.98:0/4185460070 >> 172.21.15.98:0/4185460070 conn(0x7f4510067530 msgr2=0x7f451006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 -- 172.21.15.98:0/4185460070 shutdown_connections 2024-05-27T00:31:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:10.636+0000 7f4514a00640 1 -- 172.21.15.98:0/4185460070 wait complete. 2024-05-27T00:31:10.660 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-05-27T00:31:10.660 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2024-05-27T00:31:14.886 INFO:tasks.workunit.client.0.smithi098.stdout:v0.70 2024-05-27T00:31:14.886 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 70 -eq 48 ] 2024-05-27T00:31:14.886 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.70 2024-05-27T00:31:14.886 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.70 2024-05-27T00:31:14.886 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.70 2024-05-27T00:31:17.540 INFO:tasks.ceph.mds.a.smithi098.stderr:2024-05-27T00:31:17.536+0000 7fbbc7a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:31:17.940 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at e3bb0656d92 v0.70 2024-05-27T00:31:17.942 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.70 2024-05-27T00:31:17.958 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 71 -eq 48 ] 2024-05-27T00:31:17.959 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.71 2024-05-27T00:31:17.959 INFO:tasks.workunit.client.0.smithi098.stdout:v0.71 2024-05-27T00:31:17.959 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.71 2024-05-27T00:31:17.959 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.71 2024-05-27T00:31:20.253 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 57% (231/405) Updating files: 58% (235/405) Updating files: 59% (239/405) Updating files: 60% (243/405) Updating files: 61% (248/405) Updating files: 62% (252/405) Updating files: 63% (256/405) Updating files: 64% (260/405) Updating files: 65% (264/405) Updating files: 66% (268/405) Updating files: 67% (272/405) Updating files: 68% (276/405) Updating files: 69% (280/405) Updating files: 70% (284/405) Updating files: 71% (288/405) Updating files: 72% (292/405) Updating files: 73% (296/405) Updating files: 74% (300/405) Updating files: 75% (304/405) Updating files: 76% (308/405) Updating files: 77% (312/405) Updating files: 78% (316/405) Updating files: 79% (320/405) Updating files: 80% (324/405) Updating files: 81% (329/405) Updating files: 82% (333/405) Updating files: 83% (337/405) Updating files: 84% (341/405) Updating files: 85% (345/405) Updating files: 86% (349/405) Updating files: 87% (353/405) Updating files: 88% (357/405) Updating files: 89% (361/405) Updating files: 90% (365/405) Updating files: 91% (369/405) Updating files: 92% (373/405) Updating files: 93% (377/405) Updating files: 94% (381/405) Updating files: 95% (385/405) Updating files: 96% (389/405) Updating files: 97% (393/405) Updating files: 97% (396/405) Updating files: 98% (397/405) Updating files: 99% (401/405) Updating files: 100% (405/405) Updating files: 100% (405/405), done. 2024-05-27T00:31:20.298 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at e509cb1e69c v0.71 2024-05-27T00:31:20.308 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.71 2024-05-27T00:31:20.408 INFO:tasks.workunit.client.0.smithi098.stdout:v0.72 2024-05-27T00:31:20.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 72 -eq 48 ] 2024-05-27T00:31:20.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.72 2024-05-27T00:31:20.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.72 2024-05-27T00:31:20.408 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.72 2024-05-27T00:31:21.684 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 5832e2603c7 v0.72 2024-05-27T00:31:21.695 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.72 2024-05-27T00:31:21.778 INFO:tasks.workunit.client.0.smithi098.stdout:v0.73 2024-05-27T00:31:21.778 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 73 -eq 48 ] 2024-05-27T00:31:21.778 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.73 2024-05-27T00:31:21.778 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.73 2024-05-27T00:31:21.778 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.73 2024-05-27T00:31:22.817 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at d8ad51ee8ab v0.73 2024-05-27T00:31:22.822 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.73 2024-05-27T00:31:22.983 INFO:tasks.workunit.client.0.smithi098.stdout:v0.74 2024-05-27T00:31:22.983 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 74 -eq 48 ] 2024-05-27T00:31:22.983 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.74 2024-05-27T00:31:22.983 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.74 2024-05-27T00:31:22.983 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.74 2024-05-27T00:31:23.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:31:23.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:31:24.294 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at c165483bc72 v0.74 2024-05-27T00:31:24.306 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.74 2024-05-27T00:31:24.346 INFO:tasks.workunit.client.0.smithi098.stdout:v0.75 2024-05-27T00:31:24.346 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 75 -eq 48 ] 2024-05-27T00:31:24.346 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.75 2024-05-27T00:31:24.347 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.75 2024-05-27T00:31:24.347 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.75 2024-05-27T00:31:35.834 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 38% (112/291) Updating files: 39% (114/291) Updating files: 40% (117/291) Updating files: 41% (120/291) Updating files: 42% (123/291) Updating files: 43% (126/291) Updating files: 44% (129/291) Updating files: 45% (131/291) Updating files: 46% (134/291) Updating files: 47% (137/291) Updating files: 48% (140/291) Updating files: 49% (143/291) Updating files: 50% (146/291) Updating files: 51% (149/291) Updating files: 52% (152/291) Updating files: 53% (155/291) Updating files: 54% (158/291) Updating files: 55% (161/291) Updating files: 56% (163/291) Updating files: 57% (166/291) Updating files: 58% (169/291) Updating files: 59% (172/291) Updating files: 60% (175/291) Updating files: 61% (178/291) Updating files: 62% (181/291) Updating files: 63% (184/291) Updating files: 64% (187/291) Updating files: 65% (190/291) Updating files: 66% (193/291) Updating files: 67% (195/291) Updating files: 68% (198/291) Updating files: 69% (201/291) Updating files: 70% (204/291) Updating files: 71% (207/291) Updating files: 72% (210/291) Updating files: 73% (213/291) Updating files: 74% (216/291) Updating files: 75% (219/291) Updating files: 76% (222/291) Updating files: 77% (225/291) Updating files: 78% (227/291) Updating files: 79% (230/291) Updating files: 80% (233/291) Updating files: 81% (236/291) Updating files: 82% (239/291) Updating files: 83% (242/291) Updating files: 84% (245/291) Updating files: 85% (248/291) Updating files: 86% (251/291) Updating files: 87% (254/291) Updating files: 88% (257/291) Updating files: 89% (259/291) Updating files: 90% (262/291) Updating files: 91% (265/291) Updating files: 92% (268/291) Updating files: 93% (271/291) Updating files: 94% (274/291) Updating files: 95% (277/291) Updating files: 96% (280/291) Updating files: 97% (283/291) Updating files: 98% (286/291) Updating files: 99% (289/291) Updating files: 100% (291/291) Updating files: 100% (291/291), done. 2024-05-27T00:31:35.886 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 946d6036958 v0.75 2024-05-27T00:31:35.894 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.75 2024-05-27T00:31:36.090 INFO:tasks.workunit.client.0.smithi098.stdout:v0.76 2024-05-27T00:31:36.090 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 76 -eq 48 ] 2024-05-27T00:31:36.090 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.76 2024-05-27T00:31:36.091 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.76 2024-05-27T00:31:36.091 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.76 2024-05-27T00:31:40.408 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 38% (65/168) Updating files: 39% (66/168) Updating files: 40% (68/168) Updating files: 41% (69/168) Updating files: 42% (71/168) Updating files: 43% (73/168) Updating files: 44% (74/168) Updating files: 45% (76/168) Updating files: 46% (78/168) Updating files: 47% (79/168) Updating files: 48% (81/168) Updating files: 49% (83/168) Updating files: 50% (84/168) Updating files: 51% (86/168) Updating files: 52% (88/168) Updating files: 53% (90/168) Updating files: 54% (91/168) Updating files: 55% (93/168) Updating files: 56% (95/168) Updating files: 57% (96/168) Updating files: 58% (98/168) Updating files: 59% (100/168) Updating files: 60% (101/168) Updating files: 61% (103/168) Updating files: 62% (105/168) Updating files: 63% (106/168) Updating files: 64% (108/168) Updating files: 65% (110/168) Updating files: 66% (111/168) Updating files: 67% (113/168) Updating files: 68% (115/168) Updating files: 69% (116/168) Updating files: 70% (118/168) Updating files: 71% (120/168) Updating files: 72% (121/168) Updating files: 73% (123/168) Updating files: 74% (125/168) Updating files: 75% (126/168) Updating files: 76% (128/168) Updating files: 77% (130/168) Updating files: 78% (132/168) Updating files: 79% (133/168) Updating files: 80% (135/168) Updating files: 81% (137/168) Updating files: 82% (138/168) Updating files: 83% (140/168) Updating files: 84% (142/168) Updating files: 85% (143/168) Updating files: 86% (145/168) Updating files: 87% (147/168) Updating files: 88% (148/168) Updating files: 89% (150/168) Updating files: 90% (152/168) Updating files: 91% (153/168) Updating files: 92% (155/168) Updating files: 93% (157/168) Updating files: 94% (158/168) Updating files: 95% (160/168) Updating files: 96% (162/168) Updating files: 97% (163/168) Updating files: 98% (165/168) Updating files: 99% (167/168) Updating files: 100% (168/168) Updating files: 100% (168/168), done. 2024-05-27T00:31:40.472 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 3b990136bfa v0.76 2024-05-27T00:31:40.484 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.76 2024-05-27T00:31:40.584 INFO:tasks.workunit.client.0.smithi098.stdout:v0.77 2024-05-27T00:31:40.584 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 77 -eq 48 ] 2024-05-27T00:31:40.584 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.77 2024-05-27T00:31:40.584 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.77 2024-05-27T00:31:40.584 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.77 2024-05-27T00:31:49.676 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 63% (188/298) Updating files: 64% (191/298) Updating files: 65% (194/298) Updating files: 66% (197/298) Updating files: 67% (200/298) Updating files: 68% (203/298) Updating files: 69% (206/298) Updating files: 70% (209/298) Updating files: 71% (212/298) Updating files: 72% (215/298) Updating files: 73% (218/298) Updating files: 74% (221/298) Updating files: 75% (224/298) Updating files: 76% (227/298) Updating files: 77% (230/298) Updating files: 78% (233/298) Updating files: 79% (236/298) Updating files: 80% (239/298) Updating files: 81% (242/298) Updating files: 82% (245/298) Updating files: 83% (248/298) Updating files: 84% (251/298) Updating files: 85% (254/298) Updating files: 86% (257/298) Updating files: 87% (260/298) Updating files: 88% (263/298) Updating files: 89% (266/298) Updating files: 90% (269/298) Updating files: 91% (272/298) Updating files: 92% (275/298) Updating files: 93% (278/298) Updating files: 94% (281/298) Updating files: 95% (284/298) Updating files: 96% (287/298) Updating files: 97% (290/298) Updating files: 98% (293/298) Updating files: 99% (296/298) Updating files: 100% (298/298) Updating files: 100% (298/298), done. 2024-05-27T00:31:49.800 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 1bca9c5c412 v0.77 2024-05-27T00:31:49.806 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.77 2024-05-27T00:31:49.823 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 78 -eq 48 ] 2024-05-27T00:31:49.823 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.78 2024-05-27T00:31:49.824 INFO:tasks.workunit.client.0.smithi098.stdout:v0.78 2024-05-27T00:31:49.824 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.78 2024-05-27T00:31:49.824 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.78 2024-05-27T00:31:52.100 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 52% (262/496) Updating files: 53% (263/496) Updating files: 54% (268/496) Updating files: 55% (273/496) Updating files: 56% (278/496) Updating files: 57% (283/496) Updating files: 58% (288/496) Updating files: 59% (293/496) Updating files: 60% (298/496) Updating files: 61% (303/496) Updating files: 62% (308/496) Updating files: 63% (313/496) Updating files: 64% (318/496) Updating files: 65% (323/496) Updating files: 66% (328/496) Updating files: 67% (333/496) Updating files: 68% (338/496) Updating files: 69% (343/496) Updating files: 70% (348/496) Updating files: 71% (353/496) Updating files: 72% (358/496) Updating files: 73% (363/496) Updating files: 74% (368/496) Updating files: 75% (372/496) Updating files: 76% (377/496) Updating files: 77% (382/496) Updating files: 78% (387/496) Updating files: 79% (392/496) Updating files: 80% (397/496) Updating files: 81% (402/496) Updating files: 82% (407/496) Updating files: 83% (412/496) Updating files: 84% (417/496) Updating files: 85% (422/496) Updating files: 86% (427/496) Updating files: 87% (432/496) Updating files: 88% (437/496) Updating files: 89% (442/496) Updating files: 90% (447/496) Updating files: 91% (452/496) Updating files: 92% (457/496) Updating files: 93% (462/496) Updating files: 94% (467/496) Updating files: 95% (472/496) Updating files: 96% (477/496) Updating files: 97% (482/496) Updating files: 98% (487/496) Updating files: 99% (492/496) Updating files: 99% (494/496) Updating files: 100% (496/496) Updating files: 100% (496/496), done. 2024-05-27T00:31:52.232 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at f6c746c314d 0.78 2024-05-27T00:31:52.245 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.78 2024-05-27T00:31:52.337 INFO:tasks.workunit.client.0.smithi098.stdout:v0.79 2024-05-27T00:31:52.338 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 79 -eq 48 ] 2024-05-27T00:31:52.338 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.79 2024-05-27T00:31:52.338 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.79 2024-05-27T00:31:52.338 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.79 2024-05-27T00:31:53.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:31:53.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bc1a00640 1 Processor -- start 2024-05-27T00:31:53.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bc1a00640 1 -- start start 2024-05-27T00:31:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bc1a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7bbc0731f0 0x7f7bbc0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bc1a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bbc0735d0 0x7f7bbc151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bc1a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bbc0c17d0 con 0x7f7bbc0c79f0 2024-05-27T00:31:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bc1a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7bbc071990 con 0x7f7bbc0731f0 2024-05-27T00:31:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bc1a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7bbc070e10 con 0x7f7bbc0735d0 2024-05-27T00:31:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bbaa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7bbc0731f0 0x7f7bbc0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bbaa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7bbc0731f0 0x7f7bbc0c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45082/0 (socket says 172.21.15.98:45082) 2024-05-27T00:31:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bbaa00640 1 -- 172.21.15.98:0/4199012800 learned_addr learned my addr 172.21.15.98:0/4199012800 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bbb400640 1 --1- 172.21.15.98:0/4199012800 >> v1:172.21.15.98:6789/0 conn(0x7f7bbc0c79f0 0x7f7bbc157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37170/0 (socket says 172.21.15.98:37170) 2024-05-27T00:31:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bba000640 1 --2- 172.21.15.98:0/4199012800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bbc0735d0 0x7f7bbc151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bbaa00640 1 -- 172.21.15.98:0/4199012800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bbc0735d0 msgr2=0x7f7bbc151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bbaa00640 1 --2- 172.21.15.98:0/4199012800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bbc0735d0 0x7f7bbc151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bbaa00640 1 -- 172.21.15.98:0/4199012800 >> v1:172.21.15.98:6789/0 conn(0x7f7bbc0c79f0 legacy=0x7f7bbc157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bbaa00640 1 -- 172.21.15.98:0/4199012800 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bbc0710f0 con 0x7f7bbc0731f0 2024-05-27T00:31:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.807+0000 7f7bb9600640 1 -- 172.21.15.98:0/4199012800 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3516024786 0 0) 0x7f7bbc0c17d0 con 0x7f7bbc0c79f0 2024-05-27T00:31:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bbaa00640 1 --2- 172.21.15.98:0/4199012800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 0x7f7bbc0c0eb0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009870 tx=0x7f7bac01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c469df01f3bbd68a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bb9600640 1 -- 172.21.15.98:0/4199012800 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0b5070 con 0x7f7bbc0731f0 2024-05-27T00:31:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bb9600640 1 -- 172.21.15.98:0/4199012800 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bac0a8a50 con 0x7f7bbc0731f0 2024-05-27T00:31:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bb9600640 1 -- 172.21.15.98:0/4199012800 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0a8e20 con 0x7f7bbc0731f0 2024-05-27T00:31:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bc1a00640 1 -- 172.21.15.98:0/4199012800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 msgr2=0x7f7bbc0c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bc1a00640 1 --2- 172.21.15.98:0/4199012800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 0x7f7bbc0c0eb0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009870 tx=0x7f7bac01ed50 comp rx=0 tx=0).stop 2024-05-27T00:31:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bc1a00640 1 -- 172.21.15.98:0/4199012800 shutdown_connections 2024-05-27T00:31:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bc1a00640 1 --2- 172.21.15.98:0/4199012800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bbc0735d0 0x7f7bbc151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bc1a00640 1 --2- 172.21.15.98:0/4199012800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 0x7f7bbc0c0eb0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bc1a00640 1 -- 172.21.15.98:0/4199012800 >> 172.21.15.98:0/4199012800 conn(0x7f7bbc067530 msgr2=0x7f7bbc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bc1a00640 1 -- 172.21.15.98:0/4199012800 shutdown_connections 2024-05-27T00:31:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.811+0000 7f7bc1a00640 1 -- 172.21.15.98:0/4199012800 wait complete. 2024-05-27T00:31:53.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bc1a00640 1 Processor -- start 2024-05-27T00:31:53.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bc1a00640 1 -- start start 2024-05-27T00:31:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bc1a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 0x7f7bbc0c6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bc1a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bbc0735d0 0x7f7bbc0cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bc1a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bbc0c79f0 0x7f7bbc0cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bc1a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc0c8400 con 0x7f7bbc0731f0 2024-05-27T00:31:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bc1a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc0716b0 con 0x7f7bbc0735d0 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bc1a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7bbc0c13f0 con 0x7f7bbc0c79f0 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bbaa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 0x7f7bbc0c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bbaa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 0x7f7bbc0c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45090/0 (socket says 172.21.15.98:45090) 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bbaa00640 1 -- 172.21.15.98:0/2863476343 learned_addr learned my addr 172.21.15.98:0/2863476343 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bbb400640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bbc0c79f0 0x7f7bbc0cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bba000640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bbc0735d0 0x7f7bbc0cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bbaa00640 1 -- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bbc0c79f0 msgr2=0x7f7bbc0cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bbaa00640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bbc0c79f0 0x7f7bbc0cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bbaa00640 1 -- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bbc0735d0 msgr2=0x7f7bbc0cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bbaa00640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bbc0735d0 0x7f7bbc0cabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bbaa00640 1 -- 172.21.15.98:0/2863476343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bbc1f1660 con 0x7f7bbc0731f0 2024-05-27T00:31:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bba000640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bbc0735d0 0x7f7bbc0cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:31:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bbaa00640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 0x7f7bbc0c6120 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0b3040 tx=0x7f7bac01baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bb2a00640 1 -- 172.21.15.98:0/2863476343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0c0070 con 0x7f7bbc0731f0 2024-05-27T00:31:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bb2a00640 1 -- 172.21.15.98:0/2863476343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bac0b0c40 con 0x7f7bbc0731f0 2024-05-27T00:31:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bc1a00640 1 -- 172.21.15.98:0/2863476343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bbc0c9b70 con 0x7f7bbc0731f0 2024-05-27T00:31:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bb2a00640 1 -- 172.21.15.98:0/2863476343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0b5040 con 0x7f7bbc0731f0 2024-05-27T00:31:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.815+0000 7f7bc1a00640 1 -- 172.21.15.98:0/2863476343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bbc0c9e10 con 0x7f7bbc0731f0 2024-05-27T00:31:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.819+0000 7f7bc1a00640 1 -- 172.21.15.98:0/2863476343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bbc075cb0 con 0x7f7bbc0731f0 2024-05-27T00:31:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.819+0000 7f7bb2a00640 1 -- 172.21.15.98:0/2863476343 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7bac0c3070 con 0x7f7bbc0731f0 2024-05-27T00:31:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.819+0000 7f7bb2a00640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b9c0678c0 0x7f7b9c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.819+0000 7f7bb2a00640 1 -- 172.21.15.98:0/2863476343 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f7bac1236e0 con 0x7f7bbc0731f0 2024-05-27T00:31:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.819+0000 7f7bba000640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b9c0678c0 0x7f7b9c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.819+0000 7f7bba000640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b9c0678c0 0x7f7b9c069d80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc14b680 tx=0x7f7ba8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:53.823+0000 7f7bb2a00640 1 -- 172.21.15.98:0/2863476343 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7bac0188b0 con 0x7f7bbc0731f0 2024-05-27T00:31:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.027+0000 7f7bc1a00640 1 -- 172.21.15.98:0/2863476343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bbc151b10 con 0x7f7bbc0731f0 2024-05-27T00:31:54.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:31:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.027+0000 7f7bb2a00640 1 -- 172.21.15.98:0/2863476343 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6839 (secure 0 0 0) 0x7f7bac0f3d50 con 0x7f7bbc0731f0 2024-05-27T00:31:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:31:54.032 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:09.534234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":30,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6240,"qdb_cluster":[6240]},"id":1}]} 2024-05-27T00:31:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 68 2024-05-27T00:31:54.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:31:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 -- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b9c0678c0 msgr2=0x7f7b9c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b9c0678c0 0x7f7b9c069d80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc14b680 tx=0x7f7ba8008040 comp rx=0 tx=0).stop 2024-05-27T00:31:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 -- 172.21.15.98:0/2863476343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 msgr2=0x7f7bbc0c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 0x7f7bbc0c6120 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0b3040 tx=0x7f7bac01baf0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 -- 172.21.15.98:0/2863476343 shutdown_connections 2024-05-27T00:31:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bbc0c79f0 0x7f7bbc0cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b9c0678c0 0x7f7b9c069d80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bbc0735d0 0x7f7bbc0cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 --2- 172.21.15.98:0/2863476343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc0731f0 0x7f7bbc0c6120 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 -- 172.21.15.98:0/2863476343 >> 172.21.15.98:0/2863476343 conn(0x7f7bbc067530 msgr2=0x7f7bbc0c7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 -- 172.21.15.98:0/2863476343 shutdown_connections 2024-05-27T00:31:54.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.031+0000 7f7bc1a00640 1 -- 172.21.15.98:0/2863476343 wait complete. 2024-05-27T00:31:54.071 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2024-05-27T00:31:54.071 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-27T00:31:54.071 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:31:54.137 INFO:tasks.ceph.mds.a:Stopped 2024-05-27T00:31:54.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:31:54.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.307+0000 7f53b8200640 1 Processor -- start 2024-05-27T00:31:54.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.307+0000 7f53b8200640 1 -- start start 2024-05-27T00:31:54.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b8200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f53b00735d0 0x7f53b0151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:54.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b8200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b00c17d0 con 0x7f53b00731f0 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b8200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b0070e10 con 0x7f53b00c79f0 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b8200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f53b0071990 con 0x7f53b00735d0 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b5800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f53b00731f0 0x7f53b00c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37176/0 (socket says 172.21.15.98:37176) 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b5800640 1 -- 172.21.15.98:0/1236097449 learned_addr learned my addr 172.21.15.98:0/1236097449 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b4e00640 1 --2- 172.21.15.98:0/1236097449 >> v2:172.21.15.204:3300/0 conn(0x7f53b00735d0 0x7f53b0151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217449353 0 0) 0x7f53b00c17d0 con 0x7f53b00731f0 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5398003530 con 0x7f53b00731f0 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 265656634 0 0) 0x7f5398003530 con 0x7f53b00731f0 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 >> v2:172.21.15.204:3300/0 conn(0x7f53b00735d0 msgr2=0x7f53b0151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 --2- 172.21.15.98:0/1236097449 >> v2:172.21.15.204:3300/0 conn(0x7f53b00735d0 0x7f53b0151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 >> v1:172.21.15.204:6789/0 conn(0x7f53b00c79f0 legacy=0x7f53b0157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b00710f0 con 0x7f53b00731f0 2024-05-27T00:31:54.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f53a4001680 con 0x7f53b00731f0 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 >> v1:172.21.15.98:6789/0 conn(0x7f53b00731f0 legacy=0x7f53b00c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5398003a10 0x7f5398003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5398004810 0x7f5398004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53980051d0 0x7f539800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53b0071990 con 0x7f5398003a10 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f539800c100 con 0x7f5398004810 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f539800c300 con 0x7f53980051d0 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b5800640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5398003a10 0x7f5398003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b4e00640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5398004810 0x7f5398004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b6200640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53980051d0 0x7f539800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b5800640 1 -- 172.21.15.98:0/1236097449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53980051d0 msgr2=0x7f539800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b5800640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53980051d0 0x7f539800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b5800640 1 -- 172.21.15.98:0/1236097449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5398004810 msgr2=0x7f5398004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:54.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b5800640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5398004810 0x7f5398004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b5800640 1 -- 172.21.15.98:0/1236097449 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5398003530 con 0x7f5398003a10 2024-05-27T00:31:54.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b4e00640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5398004810 0x7f5398004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:31:54.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.311+0000 7f53b6200640 1 -- 172.21.15.98:0/1236097449 reap_dead start 2024-05-27T00:31:54.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53b5800640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5398003a10 0x7f5398003e50 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f53a400c110 tx=0x7f53a401cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a79bdd3f37756b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:54.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4007c50 con 0x7f5398003a10 2024-05-27T00:31:54.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a40a9020 con 0x7f5398003a10 2024-05-27T00:31:54.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53afe00640 1 -- 172.21.15.98:0/1236097449 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4005ae0 con 0x7f5398003a10 2024-05-27T00:31:54.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53b8200640 1 -- 172.21.15.98:0/1236097449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5398003a10 msgr2=0x7f5398003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:54.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53b8200640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5398003a10 0x7f5398003e50 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f53a400c110 tx=0x7f53a401cb90 comp rx=0 tx=0).stop 2024-05-27T00:31:54.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53b8200640 1 -- 172.21.15.98:0/1236097449 shutdown_connections 2024-05-27T00:31:54.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53b8200640 1 --2- 172.21.15.98:0/1236097449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5398003a10 0x7f5398003e50 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53b8200640 1 -- 172.21.15.98:0/1236097449 >> 172.21.15.98:0/1236097449 conn(0x7f53b0067530 msgr2=0x7f53b006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:54.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53b8200640 1 -- 172.21.15.98:0/1236097449 shutdown_connections 2024-05-27T00:31:54.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53b8200640 1 -- 172.21.15.98:0/1236097449 wait complete. 2024-05-27T00:31:54.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53b8200640 1 Processor -- start 2024-05-27T00:31:54.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.315+0000 7f53b8200640 1 -- start start 2024-05-27T00:31:54.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b8200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5398003a10 0x7f53b00c1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:54.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b8200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53b01f19a0 0x7f53b01f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:54.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b8200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b01f2140 0x7f53b00c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:54.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b8200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53b01515b0 con 0x7f53b01f2140 2024-05-27T00:31:54.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b8200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f53b0151430 con 0x7f53b01f19a0 2024-05-27T00:31:54.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b8200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f53b0151730 con 0x7f5398003a10 2024-05-27T00:31:54.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b6200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b01f2140 0x7f53b00c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:54.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b6200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b01f2140 0x7f53b00c8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45108/0 (socket says 172.21.15.98:45108) 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b6200640 1 -- 172.21.15.98:0/940763296 learned_addr learned my addr 172.21.15.98:0/940763296 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b5800640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5398003a10 0x7f53b00c1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b4e00640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53b01f19a0 0x7f53b01f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b6200640 1 -- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5398003a10 msgr2=0x7f53b00c1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b6200640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5398003a10 0x7f53b00c1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b6200640 1 -- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53b01f19a0 msgr2=0x7f53b01f4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b6200640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53b01f19a0 0x7f53b01f4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b6200640 1 -- 172.21.15.98:0/940763296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b0152180 con 0x7f53b01f2140 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b6200640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b01f2140 0x7f53b00c8940 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f53b0072e90 tx=0x7f53a800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53ad600640 1 -- 172.21.15.98:0/940763296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8003980 con 0x7f53b01f2140 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53ad600640 1 -- 172.21.15.98:0/940763296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a8018070 con 0x7f53b01f2140 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53ad600640 1 -- 172.21.15.98:0/940763296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8003d80 con 0x7f53b01f2140 2024-05-27T00:31:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b8200640 1 -- 172.21.15.98:0/940763296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b00c2120 con 0x7f53b01f2140 2024-05-27T00:31:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b8200640 1 -- 172.21.15.98:0/940763296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b01f0ec0 con 0x7f53b01f2140 2024-05-27T00:31:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.319+0000 7f53b8200640 1 -- 172.21.15.98:0/940763296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b01f0cb0 con 0x7f53b01f2140 2024-05-27T00:31:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.323+0000 7f53ad600640 1 -- 172.21.15.98:0/940763296 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f53a80050d0 con 0x7f53b01f2140 2024-05-27T00:31:54.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.323+0000 7f53ad600640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53900678f0 0x7f5390069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:54.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.323+0000 7f53ad600640 1 -- 172.21.15.98:0/940763296 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f53a8085c50 con 0x7f53b01f2140 2024-05-27T00:31:54.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.323+0000 7f53b5800640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53900678f0 0x7f5390069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:54.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.323+0000 7f53b5800640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53900678f0 0x7f5390069db0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f53a40aa000 tx=0x7f53b00c19a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:54.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.327+0000 7f53ad600640 1 -- 172.21.15.98:0/940763296 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f53b0152180 con 0x7f53b01f2140 2024-05-27T00:31:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 -- 172.21.15.98:0/940763296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53b0075cb0 con 0x7f53b01f2140 2024-05-27T00:31:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53ad600640 1 -- 172.21.15.98:0/940763296 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6839 (secure 0 0 0) 0x7f53a80562c0 con 0x7f53b01f2140 2024-05-27T00:31:54.522 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:31:54.523 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:09.534234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":30,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6240,"qdb_cluster":[6240]},"id":1}]} 2024-05-27T00:31:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 68 2024-05-27T00:31:54.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 -- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53900678f0 msgr2=0x7f5390069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:54.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53900678f0 0x7f5390069db0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f53a40aa000 tx=0x7f53b00c19a0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 -- 172.21.15.98:0/940763296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b01f2140 msgr2=0x7f53b00c8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b01f2140 0x7f53b00c8940 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f53b0072e90 tx=0x7f53a800eb90 comp rx=0 tx=0).stop 2024-05-27T00:31:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 -- 172.21.15.98:0/940763296 shutdown_connections 2024-05-27T00:31:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53b01f2140 0x7f53b00c8940 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f53900678f0 0x7f5390069db0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53b01f19a0 0x7f53b01f4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 --2- 172.21.15.98:0/940763296 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5398003a10 0x7f53b00c1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 -- 172.21.15.98:0/940763296 >> 172.21.15.98:0/940763296 conn(0x7f53b0067530 msgr2=0x7f53b014f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 -- 172.21.15.98:0/940763296 shutdown_connections 2024-05-27T00:31:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:54.519+0000 7f53b8200640 1 -- 172.21.15.98:0/940763296 wait complete. 2024-05-27T00:31:54.545 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:31:56.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:31:56.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.687+0000 7f24c6000640 1 Processor -- start 2024-05-27T00:31:56.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.687+0000 7f24c6000640 1 -- start start 2024-05-27T00:31:56.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24c6000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24c00c79f0 0x7f24c0157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:56.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24c6000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24c00c17d0 con 0x7f24c00731f0 2024-05-27T00:31:56.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24c6000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24c0070e10 con 0x7f24c00735d0 2024-05-27T00:31:56.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24c6000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f24c0071990 con 0x7f24c00c79f0 2024-05-27T00:31:56.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bf400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f24c00731f0 0x7f24c00c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37190/0 (socket says 172.21.15.98:37190) 2024-05-27T00:31:56.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bf400640 1 -- 172.21.15.98:0/3865684482 learned_addr learned my addr 172.21.15.98:0/3865684482 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:31:56.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bfe00640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24c00c79f0 0x7f24c0157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719698897 0 0) 0x7f24c00c17d0 con 0x7f24c00731f0 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24a4005740 con 0x7f24c00731f0 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2602307834 0 0) 0x7f24c0070e10 con 0x7f24c00735d0 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24c00c17d0 con 0x7f24c00735d0 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2672856105 0 0) 0x7f24a4005740 con 0x7f24c00731f0 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24c00c79f0 msgr2=0x7f24c0157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24c00c79f0 0x7f24c0157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 >> v1:172.21.15.204:6789/0 conn(0x7f24c00735d0 legacy=0x7f24c0151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c00710f0 con 0x7f24c00731f0 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f24b0001680 con 0x7f24c00731f0 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 >> v1:172.21.15.98:6789/0 conn(0x7f24c00731f0 legacy=0x7f24c00c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:56.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24a4005d90 0x7f24a40061b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24a4006a20 0x7f24a4006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24c00c79f0 0x7f24a40073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24c0071990 con 0x7f24a4006a20 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f24a400dc90 con 0x7f24a4005d90 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f24a400de90 con 0x7f24c00c79f0 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24b6a00640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24a4006a20 0x7f24a4006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bf400640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24a4005d90 0x7f24a40061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24bfe00640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24c00c79f0 0x7f24a40073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24b6a00640 1 -- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24c00c79f0 msgr2=0x7f24a40073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24b6a00640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24c00c79f0 0x7f24a40073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24b6a00640 1 -- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24a4005d90 msgr2=0x7f24a40061b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24b6a00640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24a4005d90 0x7f24a40061b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:56.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.691+0000 7f24b6a00640 1 -- 172.21.15.98:0/3865684482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a4005740 con 0x7f24a4006a20 2024-05-27T00:31:56.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24b6a00640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24a4006a20 0x7f24a4006ea0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f24a8007fd0 tx=0x7f24a801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33942e993caebd14 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:56.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a80b0070 con 0x7f24a4006a20 2024-05-27T00:31:56.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a80052f0 con 0x7f24a4006a20 2024-05-27T00:31:56.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24bea00640 1 -- 172.21.15.98:0/3865684482 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8001520 con 0x7f24a4006a20 2024-05-27T00:31:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24c6000640 1 -- 172.21.15.98:0/3865684482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24a4006a20 msgr2=0x7f24a4006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24c6000640 1 --2- 172.21.15.98:0/3865684482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24a4006a20 0x7f24a4006ea0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f24a8007fd0 tx=0x7f24a801c500 comp rx=0 tx=0).stop 2024-05-27T00:31:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24bfe00640 1 -- 172.21.15.98:0/3865684482 reap_dead start 2024-05-27T00:31:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24c6000640 1 -- 172.21.15.98:0/3865684482 shutdown_connections 2024-05-27T00:31:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24c6000640 1 -- 172.21.15.98:0/3865684482 >> 172.21.15.98:0/3865684482 conn(0x7f24c0067530 msgr2=0x7f24c006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24c6000640 1 -- 172.21.15.98:0/3865684482 shutdown_connections 2024-05-27T00:31:56.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24c6000640 1 -- 172.21.15.98:0/3865684482 wait complete. 2024-05-27T00:31:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24c6000640 1 Processor -- start 2024-05-27T00:31:56.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.695+0000 7f24c6000640 1 -- start start 2024-05-27T00:31:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24c6000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24a4006a20 0x7f24c01f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24c6000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24c014c9a0 0x7f24c015afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:56.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24c6000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24c0144740 0x7f24c0144bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24c6000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24c00c8400 con 0x7f24c014c9a0 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24c6000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f24c00716b0 con 0x7f24c0144740 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24c6000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f24c0151b40 con 0x7f24a4006a20 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24c014c9a0 0x7f24c015afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24c014c9a0 0x7f24c015afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45130/0 (socket says 172.21.15.98:45130) 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b6a00640 1 -- 172.21.15.98:0/2179136440 learned_addr learned my addr 172.21.15.98:0/2179136440 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24bf400640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24a4006a20 0x7f24c01f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24bfe00640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24c0144740 0x7f24c0144bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b6a00640 1 -- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24a4006a20 msgr2=0x7f24c01f42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b6a00640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24a4006a20 0x7f24c01f42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b6a00640 1 -- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24c0144740 msgr2=0x7f24c0144bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b6a00640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24c0144740 0x7f24c0144bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b6a00640 1 -- 172.21.15.98:0/2179136440 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c0145100 con 0x7f24c014c9a0 2024-05-27T00:31:56.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b6a00640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24c014c9a0 0x7f24c015afe0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f24a80ae040 tx=0x7f24a4007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:56.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b7e00640 1 -- 172.21.15.98:0/2179136440 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a80bb070 con 0x7f24c014c9a0 2024-05-27T00:31:56.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b7e00640 1 -- 172.21.15.98:0/2179136440 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a80054a0 con 0x7f24c014c9a0 2024-05-27T00:31:56.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b7e00640 1 -- 172.21.15.98:0/2179136440 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a80b0040 con 0x7f24c014c9a0 2024-05-27T00:31:56.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24c6000640 1 -- 172.21.15.98:0/2179136440 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c015b9a0 con 0x7f24c014c9a0 2024-05-27T00:31:56.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24c6000640 1 -- 172.21.15.98:0/2179136440 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c01f0980 con 0x7f24c014c9a0 2024-05-27T00:31:56.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24c6000640 1 -- 172.21.15.98:0/2179136440 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24c0075cb0 con 0x7f24c014c9a0 2024-05-27T00:31:56.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.699+0000 7f24b7e00640 1 -- 172.21.15.98:0/2179136440 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f24a80b6440 con 0x7f24c014c9a0 2024-05-27T00:31:56.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.703+0000 7f24b7e00640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f249c0678c0 0x7f249c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:56.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.703+0000 7f24b7e00640 1 -- 172.21.15.98:0/2179136440 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f24a811fea0 con 0x7f24c014c9a0 2024-05-27T00:31:56.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.703+0000 7f24bf400640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f249c0678c0 0x7f249c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:56.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.703+0000 7f24bf400640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f249c0678c0 0x7f249c069d80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f24b0001060 tx=0x7f24b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.707+0000 7f24b7e00640 1 -- 172.21.15.98:0/2179136440 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f24a8016210 con 0x7f24c014c9a0 2024-05-27T00:31:56.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.895+0000 7f24c6000640 1 -- 172.21.15.98:0/2179136440 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24c0068690 con 0x7f24c014c9a0 2024-05-27T00:31:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.895+0000 7f24b7e00640 1 -- 172.21.15.98:0/2179136440 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6839 (secure 0 0 0) 0x7f24a80efec0 con 0x7f24c014c9a0 2024-05-27T00:31:56.900 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:31:56.900 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:09.534234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":30,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6240,"qdb_cluster":[6240]},"id":1}]} 2024-05-27T00:31:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 68 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 -- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f249c0678c0 msgr2=0x7f249c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f249c0678c0 0x7f249c069d80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f24b0001060 tx=0x7f24b0009040 comp rx=0 tx=0).stop 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 -- 172.21.15.98:0/2179136440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24c014c9a0 msgr2=0x7f24c015afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24c014c9a0 0x7f24c015afe0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f24a80ae040 tx=0x7f24a4007a50 comp rx=0 tx=0).stop 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 -- 172.21.15.98:0/2179136440 shutdown_connections 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f249c0678c0 0x7f249c069d80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24c0144740 0x7f24c0144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24c014c9a0 0x7f24c015afe0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 --2- 172.21.15.98:0/2179136440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f24a4006a20 0x7f24c01f42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 -- 172.21.15.98:0/2179136440 >> 172.21.15.98:0/2179136440 conn(0x7f24c0067530 msgr2=0x7f24c0157870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 -- 172.21.15.98:0/2179136440 shutdown_connections 2024-05-27T00:31:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:56.899+0000 7f24c6000640 1 -- 172.21.15.98:0/2179136440 wait complete. 2024-05-27T00:31:56.921 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:31:58.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:31:59.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.063+0000 7f85e7800640 1 Processor -- start 2024-05-27T00:31:59.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.063+0000 7f85e7800640 1 -- start start 2024-05-27T00:31:59.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e7800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f85e015b470 0x7f85e00c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:59.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e7800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85e00c1a90 con 0x7f85e0154830 2024-05-27T00:31:59.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e7800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85e00c1d70 con 0x7f85e00c6e00 2024-05-27T00:31:59.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e7800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f85e015b8d0 con 0x7f85e015b470 2024-05-27T00:31:59.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e4e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f85e0154830 0x7f85e00cbbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37198/0 (socket says 172.21.15.98:37198) 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e4e00640 1 -- 172.21.15.98:0/4230531123 learned_addr learned my addr 172.21.15.98:0/4230531123 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85dfe00640 1 --2- 172.21.15.98:0/4230531123 >> v2:172.21.15.204:3300/0 conn(0x7f85e015b470 0x7f85e00c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3071790599 0 0) 0x7f85e00c1a90 con 0x7f85e0154830 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85c8003530 con 0x7f85e0154830 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 15189652 0 0) 0x7f85c8003530 con 0x7f85e0154830 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 >> v2:172.21.15.204:3300/0 conn(0x7f85e015b470 msgr2=0x7f85e00c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 --2- 172.21.15.98:0/4230531123 >> v2:172.21.15.204:3300/0 conn(0x7f85e015b470 0x7f85e00c68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 >> v1:172.21.15.204:6789/0 conn(0x7f85e00c6e00 legacy=0x7f85e0071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e00c2050 con 0x7f85e0154830 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f85d4001680 con 0x7f85e0154830 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 >> v1:172.21.15.98:6789/0 conn(0x7f85e0154830 legacy=0x7f85e00cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85c80039c0 0x7f85c8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:59.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85c8004810 0x7f85c8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:59.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c80051d0 0x7f85c800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:59.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85e015b8d0 con 0x7f85c80051d0 2024-05-27T00:31:59.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f85c800c070 con 0x7f85c80039c0 2024-05-27T00:31:59.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f85c800c270 con 0x7f85c8004810 2024-05-27T00:31:59.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e5800640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c80051d0 0x7f85c800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:59.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e4e00640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85c80039c0 0x7f85c8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:59.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85dfe00640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85c8004810 0x7f85c8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:59.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e5800640 1 -- 172.21.15.98:0/4230531123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85c8004810 msgr2=0x7f85c8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:59.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e5800640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85c8004810 0x7f85c8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:59.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e5800640 1 -- 172.21.15.98:0/4230531123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85c80039c0 msgr2=0x7f85c8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:31:59.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e5800640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85c80039c0 0x7f85c8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:59.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.067+0000 7f85e5800640 1 -- 172.21.15.98:0/4230531123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c8003530 con 0x7f85c80051d0 2024-05-27T00:31:59.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e5800640 1 -- 172.21.15.98:0/4230531123 reap_dead start 2024-05-27T00:31:59.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e5800640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c80051d0 0x7f85c800b9d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f85d8001e70 tx=0x7f85d801c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1e0d03bf5bf273 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:59.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8001a90 con 0x7f85c80051d0 2024-05-27T00:31:59.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d8003bc0 con 0x7f85c80051d0 2024-05-27T00:31:59.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85df400640 1 -- 172.21.15.98:0/4230531123 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8005020 con 0x7f85c80051d0 2024-05-27T00:31:59.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e7800640 1 -- 172.21.15.98:0/4230531123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c80051d0 msgr2=0x7f85c800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:59.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e7800640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c80051d0 0x7f85c800b9d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f85d8001e70 tx=0x7f85d801c780 comp rx=0 tx=0).stop 2024-05-27T00:31:59.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e7800640 1 -- 172.21.15.98:0/4230531123 shutdown_connections 2024-05-27T00:31:59.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e7800640 1 --2- 172.21.15.98:0/4230531123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c80051d0 0x7f85c800b9d0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:59.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e7800640 1 -- 172.21.15.98:0/4230531123 >> 172.21.15.98:0/4230531123 conn(0x7f85e0067530 msgr2=0x7f85e006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:59.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e7800640 1 -- 172.21.15.98:0/4230531123 shutdown_connections 2024-05-27T00:31:59.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e7800640 1 -- 172.21.15.98:0/4230531123 wait complete. 2024-05-27T00:31:59.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e7800640 1 Processor -- start 2024-05-27T00:31:59.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.071+0000 7f85e7800640 1 -- start start 2024-05-27T00:31:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e7800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85c80051d0 0x7f85e00c4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e7800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85e014a820 0x7f85e0159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e7800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e00c3840 0x7f85e00c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e7800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85e00c26a0 con 0x7f85e00c3840 2024-05-27T00:31:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e7800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f85e00c2b30 con 0x7f85e014a820 2024-05-27T00:31:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e7800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f85e00c2330 con 0x7f85c80051d0 2024-05-27T00:31:59.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e5800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e00c3840 0x7f85e00c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:59.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85dfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85e014a820 0x7f85e0159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:59.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e5800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e00c3840 0x7f85e00c8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45148/0 (socket says 172.21.15.98:45148) 2024-05-27T00:31:59.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e5800640 1 -- 172.21.15.98:0/2492896921 learned_addr learned my addr 172.21.15.98:0/2492896921 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:31:59.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e4e00640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85c80051d0 0x7f85e00c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e5800640 1 -- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85c80051d0 msgr2=0x7f85e00c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e5800640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85c80051d0 0x7f85e00c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e5800640 1 -- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85e014a820 msgr2=0x7f85e0159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e5800640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85e014a820 0x7f85e0159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e5800640 1 -- 172.21.15.98:0/2492896921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e01f10d0 con 0x7f85e00c3840 2024-05-27T00:31:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e5800640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e00c3840 0x7f85e00c8690 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f85d801ccf0 tx=0x7f85c8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85dcc00640 1 -- 172.21.15.98:0/2492896921 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8007dc0 con 0x7f85e00c3840 2024-05-27T00:31:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85dcc00640 1 -- 172.21.15.98:0/2492896921 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d80012e0 con 0x7f85e00c3840 2024-05-27T00:31:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85dcc00640 1 -- 172.21.15.98:0/2492896921 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8006b60 con 0x7f85e00c3840 2024-05-27T00:31:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e7800640 1 -- 172.21.15.98:0/2492896921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e0158970 con 0x7f85e00c3840 2024-05-27T00:31:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e7800640 1 -- 172.21.15.98:0/2492896921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e0158f30 con 0x7f85e00c3840 2024-05-27T00:31:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85e7800640 1 -- 172.21.15.98:0/2492896921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e00c1300 con 0x7f85e00c3840 2024-05-27T00:31:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.075+0000 7f85dcc00640 1 -- 172.21.15.98:0/2492896921 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f85d8006d00 con 0x7f85e00c3840 2024-05-27T00:31:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.079+0000 7f85dcc00640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f85c00678f0 0x7f85c0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:31:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.079+0000 7f85dcc00640 1 -- 172.21.15.98:0/2492896921 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f85d811f5c0 con 0x7f85e00c3840 2024-05-27T00:31:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.079+0000 7f85e4e00640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f85c00678f0 0x7f85c0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:31:59.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.079+0000 7f85e4e00640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f85c00678f0 0x7f85c0069db0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f85d4001060 tx=0x7f85d4007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:31:59.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.083+0000 7f85dcc00640 1 -- 172.21.15.98:0/2492896921 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f85d8016490 con 0x7f85e00c3840 2024-05-27T00:31:59.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 -- 172.21.15.98:0/2492896921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85e00c7420 con 0x7f85e00c3840 2024-05-27T00:31:59.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85dcc00640 1 -- 172.21.15.98:0/2492896921 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6839 (secure 0 0 0) 0x7f85d80ef670 con 0x7f85e00c3840 2024-05-27T00:31:59.275 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:31:59.275 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:09.534234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":30,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6240,"qdb_cluster":[6240]},"id":1}]} 2024-05-27T00:31:59.275 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 68 2024-05-27T00:31:59.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 -- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f85c00678f0 msgr2=0x7f85c0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:59.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f85c00678f0 0x7f85c0069db0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f85d4001060 tx=0x7f85d4007900 comp rx=0 tx=0).stop 2024-05-27T00:31:59.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 -- 172.21.15.98:0/2492896921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e00c3840 msgr2=0x7f85e00c8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:31:59.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e00c3840 0x7f85e00c8690 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f85d801ccf0 tx=0x7f85c8005840 comp rx=0 tx=0).stop 2024-05-27T00:31:59.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 -- 172.21.15.98:0/2492896921 shutdown_connections 2024-05-27T00:31:59.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e00c3840 0x7f85e00c8690 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:59.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f85c00678f0 0x7f85c0069db0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:59.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85e014a820 0x7f85e0159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:59.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 --2- 172.21.15.98:0/2492896921 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85c80051d0 0x7f85e00c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:31:59.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 -- 172.21.15.98:0/2492896921 >> 172.21.15.98:0/2492896921 conn(0x7f85e0067530 msgr2=0x7f85e014ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:31:59.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 -- 172.21.15.98:0/2492896921 shutdown_connections 2024-05-27T00:31:59.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:31:59.271+0000 7f85e7800640 1 -- 172.21.15.98:0/2492896921 wait complete. 2024-05-27T00:31:59.296 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:32:01.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:01.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.439+0000 7f1a2a800640 1 Processor -- start 2024-05-27T00:32:01.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.439+0000 7f1a2a800640 1 -- start start 2024-05-27T00:32:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a2a800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1a240cc1f0 0x7f1a241592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a2a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24073570 0x7f1a241559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a2a800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a240c1f60 con 0x7f1a240cbe10 2024-05-27T00:32:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a2a800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f1a24159810 con 0x7f1a240cc1f0 2024-05-27T00:32:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a2a800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1a240c6dd0 con 0x7f1a24073570 2024-05-27T00:32:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a23400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1a240cbe10 0x7f1a240c1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37208/0 (socket says 172.21.15.98:37208) 2024-05-27T00:32:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a23400640 1 -- 172.21.15.98:0/1163918679 learned_addr learned my addr 172.21.15.98:0/1163918679 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:32:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a23e00640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24073570 0x7f1a241559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a1be00640 1 --2- 172.21.15.98:0/1163918679 >> v2:172.21.15.204:3300/0 conn(0x7f1a240cc1f0 0x7f1a241592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1976248857 0 0) 0x7f1a240c1f60 con 0x7f1a240cbe10 2024-05-27T00:32:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a08003530 con 0x7f1a240cbe10 2024-05-27T00:32:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 190467120 0 0) 0x7f1a08003530 con 0x7f1a240cbe10 2024-05-27T00:32:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24073570 msgr2=0x7f1a241559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24073570 0x7f1a241559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 >> v2:172.21.15.204:3300/0 conn(0x7f1a240cc1f0 msgr2=0x7f1a241592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 --2- 172.21.15.98:0/1163918679 >> v2:172.21.15.204:3300/0 conn(0x7f1a240cc1f0 0x7f1a241592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a240c70b0 con 0x7f1a240cbe10 2024-05-27T00:32:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f1a14001680 con 0x7f1a240cbe10 2024-05-27T00:32:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 >> v1:172.21.15.98:6789/0 conn(0x7f1a240cbe10 legacy=0x7f1a240c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080039c0 0x7f1a08003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a08004810 0x7f1a08004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24073570 0x7f1a080051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:01.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1a24159810 con 0x7f1a080039c0 2024-05-27T00:32:01.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1a240c6dd0 con 0x7f1a08004810 2024-05-27T00:32:01.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1a0800b9d0 con 0x7f1a24073570 2024-05-27T00:32:01.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a23400640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080039c0 0x7f1a08003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:01.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a1be00640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a08004810 0x7f1a08004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:01.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a23e00640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24073570 0x7f1a080051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:01.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a23400640 1 -- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24073570 msgr2=0x7f1a080051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:01.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a23400640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a24073570 0x7f1a080051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a23400640 1 -- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a08004810 msgr2=0x7f1a08004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:01.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a23400640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a08004810 0x7f1a08004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.443+0000 7f1a23400640 1 -- 172.21.15.98:0/1163918679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a08003530 con 0x7f1a080039c0 2024-05-27T00:32:01.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a23400640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080039c0 0x7f1a08003de0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1a1400c2c0 tx=0x7f1a1401f4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e198cd0167167ed4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:01.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a14007d60 con 0x7f1a080039c0 2024-05-27T00:32:01.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a14005b00 con 0x7f1a080039c0 2024-05-27T00:32:01.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a22a00640 1 -- 172.21.15.98:0/1163918679 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a14007070 con 0x7f1a080039c0 2024-05-27T00:32:01.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a2a800640 1 -- 172.21.15.98:0/1163918679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080039c0 msgr2=0x7f1a08003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:01.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a2a800640 1 --2- 172.21.15.98:0/1163918679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a080039c0 0x7f1a08003de0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1a1400c2c0 tx=0x7f1a1401f4b0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a23e00640 1 -- 172.21.15.98:0/1163918679 reap_dead start 2024-05-27T00:32:01.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a2a800640 1 -- 172.21.15.98:0/1163918679 shutdown_connections 2024-05-27T00:32:01.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a2a800640 1 -- 172.21.15.98:0/1163918679 >> 172.21.15.98:0/1163918679 conn(0x7f1a24067530 msgr2=0x7f1a2406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:01.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a2a800640 1 -- 172.21.15.98:0/1163918679 shutdown_connections 2024-05-27T00:32:01.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a2a800640 1 -- 172.21.15.98:0/1163918679 wait complete. 2024-05-27T00:32:01.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a2a800640 1 Processor -- start 2024-05-27T00:32:01.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.447+0000 7f1a2a800640 1 -- start start 2024-05-27T00:32:01.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a2a800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a080039c0 0x7f1a2414d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:01.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a2a800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a241f3e90 0x7f1a24156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:01.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a2a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a2414d600 0x7f1a24156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:01.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a2a800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1a240c1330 con 0x7f1a241f3e90 2024-05-27T00:32:01.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a2a800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1a240c1050 con 0x7f1a080039c0 2024-05-27T00:32:01.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a2a800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1a240c79d0 con 0x7f1a2414d600 2024-05-27T00:32:01.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a241f3e90 0x7f1a24156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a241f3e90 0x7f1a24156910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45172/0 (socket says 172.21.15.98:45172) 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1be00640 1 -- 172.21.15.98:0/4028975568 learned_addr learned my addr 172.21.15.98:0/4028975568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a23e00640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a2414d600 0x7f1a24156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a23400640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a080039c0 0x7f1a2414d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1be00640 1 -- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a2414d600 msgr2=0x7f1a24156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1be00640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a2414d600 0x7f1a24156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1be00640 1 -- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a080039c0 msgr2=0x7f1a2414d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1be00640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a080039c0 0x7f1a2414d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1be00640 1 -- 172.21.15.98:0/4028975568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a241f0050 con 0x7f1a241f3e90 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a23e00640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a2414d600 0x7f1a24156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1be00640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a241f3e90 0x7f1a24156910 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c011610 tx=0x7f1a0c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1b400640 1 -- 172.21.15.98:0/4028975568 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c00f8f0 con 0x7f1a241f3e90 2024-05-27T00:32:01.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1b400640 1 -- 172.21.15.98:0/4028975568 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c00fa90 con 0x7f1a241f3e90 2024-05-27T00:32:01.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a1b400640 1 -- 172.21.15.98:0/4028975568 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c00fe40 con 0x7f1a241f3e90 2024-05-27T00:32:01.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a2a800640 1 -- 172.21.15.98:0/4028975568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a241f02f0 con 0x7f1a241f3e90 2024-05-27T00:32:01.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a2a800640 1 -- 172.21.15.98:0/4028975568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a241f08b0 con 0x7f1a241f3e90 2024-05-27T00:32:01.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.451+0000 7f1a2a800640 1 -- 172.21.15.98:0/4028975568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a24075cb0 con 0x7f1a241f3e90 2024-05-27T00:32:01.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.455+0000 7f1a1b400640 1 -- 172.21.15.98:0/4028975568 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1a0c015460 con 0x7f1a241f3e90 2024-05-27T00:32:01.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.455+0000 7f1a1b400640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 0x7f1a00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:01.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.455+0000 7f1a1b400640 1 -- 172.21.15.98:0/4028975568 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f1a0c086630 con 0x7f1a241f3e90 2024-05-27T00:32:01.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.455+0000 7f1a23400640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 0x7f1a00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:01.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.455+0000 7f1a23400640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 0x7f1a00069d80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1a140ae000 tx=0x7f1a14001a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:01.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.459+0000 7f1a1b400640 1 -- 172.21.15.98:0/4028975568 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1a0c019030 con 0x7f1a241f3e90 2024-05-27T00:32:01.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 -- 172.21.15.98:0/4028975568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a240c9be0 con 0x7f1a241f3e90 2024-05-27T00:32:01.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a1b400640 1 -- 172.21.15.98:0/4028975568 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6839 (secure 0 0 0) 0x7f1a0c056cb0 con 0x7f1a241f3e90 2024-05-27T00:32:01.650 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:01.651 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:09.534234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":30,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6240,"qdb_cluster":[6240]},"id":1}]} 2024-05-27T00:32:01.651 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 68 2024-05-27T00:32:01.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 -- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 msgr2=0x7f1a00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:01.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 0x7f1a00069d80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1a140ae000 tx=0x7f1a14001a50 comp rx=0 tx=0).stop 2024-05-27T00:32:01.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 -- 172.21.15.98:0/4028975568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a241f3e90 msgr2=0x7f1a24156910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a241f3e90 0x7f1a24156910 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c011610 tx=0x7f1a0c011ad0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 -- 172.21.15.98:0/4028975568 shutdown_connections 2024-05-27T00:32:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1a2414d600 0x7f1a24156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a241f3e90 0x7f1a24156910 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1a000678c0 0x7f1a00069d80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 --2- 172.21.15.98:0/4028975568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a080039c0 0x7f1a2414d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 -- 172.21.15.98:0/4028975568 >> 172.21.15.98:0/4028975568 conn(0x7f1a24067530 msgr2=0x7f1a240cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 -- 172.21.15.98:0/4028975568 shutdown_connections 2024-05-27T00:32:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:01.647+0000 7f1a2a800640 1 -- 172.21.15.98:0/4028975568 wait complete. 2024-05-27T00:32:01.672 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:32:03.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:03.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8ec800640 1 Processor -- start 2024-05-27T00:32:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8ec800640 1 -- start start 2024-05-27T00:32:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8ec800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e4158e60 0x7fd8e4164000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8ec800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e415be40 con 0x7fd8e4152a50 2024-05-27T00:32:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8ec800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e40c50e0 con 0x7fd8e40cb7e0 2024-05-27T00:32:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8ec800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e40c8a00 con 0x7fd8e4158e60 2024-05-27T00:32:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e9e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd8e4152a50 0x7fd8e40709a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44414/0 (socket says 172.21.15.98:44414) 2024-05-27T00:32:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e9e00640 1 -- 172.21.15.98:0/1120785540 learned_addr learned my addr 172.21.15.98:0/1120785540 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:32:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8ea800640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e4158e60 0x7fd8e4164000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4264302472 0 0) 0x7fd8e415be40 con 0x7fd8e4152a50 2024-05-27T00:32:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8cc003530 con 0x7fd8e4152a50 2024-05-27T00:32:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1978148164 0 0) 0x7fd8cc003530 con 0x7fd8e4152a50 2024-05-27T00:32:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e4158e60 msgr2=0x7fd8e4164000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e4158e60 0x7fd8e4164000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 >> v1:172.21.15.204:6789/0 conn(0x7fd8e40cb7e0 legacy=0x7fd8e4158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e40c8c50 con 0x7fd8e4152a50 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fd8d80016a0 con 0x7fd8e4152a50 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 >> v1:172.21.15.98:6789/0 conn(0x7fd8e4152a50 legacy=0x7fd8e40709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 0x7fd8cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e4158e60 0x7fd8cc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8cc004d50 0x7fd8cc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e40c8a00 con 0x7fd8cc0039f0 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd8cc00bab0 con 0x7fd8cc004d50 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd8cc00bcb0 con 0x7fd8e4158e60 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e9e00640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 0x7fd8cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e9400640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e4158e60 0x7fd8cc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8ea800640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8cc004d50 0x7fd8cc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:03.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e9e00640 1 -- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e4158e60 msgr2=0x7fd8cc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e9e00640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e4158e60 0x7fd8cc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e9e00640 1 -- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8cc004d50 msgr2=0x7fd8cc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e9e00640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8cc004d50 0x7fd8cc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e9e00640 1 -- 172.21.15.98:0/1120785540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8cc003530 con 0x7fd8cc0039f0 2024-05-27T00:32:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.819+0000 7fd8e9400640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e4158e60 0x7fd8cc004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8e9e00640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 0x7fd8cc003dd0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800c210 tx=0x7fd8d801cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2345a0d6d36ee8fb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8007d70 con 0x7fd8cc0039f0 2024-05-27T00:32:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8d80a9070 con 0x7fd8cc0039f0 2024-05-27T00:32:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8e8a00640 1 -- 172.21.15.98:0/1120785540 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d80a5d90 con 0x7fd8cc0039f0 2024-05-27T00:32:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8ec800640 1 -- 172.21.15.98:0/1120785540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 msgr2=0x7fd8cc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8ec800640 1 --2- 172.21.15.98:0/1120785540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 0x7fd8cc003dd0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800c210 tx=0x7fd8d801cc70 comp rx=0 tx=0).stop 2024-05-27T00:32:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8ea800640 1 -- 172.21.15.98:0/1120785540 reap_dead start 2024-05-27T00:32:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8ec800640 1 -- 172.21.15.98:0/1120785540 shutdown_connections 2024-05-27T00:32:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8ec800640 1 -- 172.21.15.98:0/1120785540 >> 172.21.15.98:0/1120785540 conn(0x7fd8e4072f90 msgr2=0x7fd8e40733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8ec800640 1 -- 172.21.15.98:0/1120785540 shutdown_connections 2024-05-27T00:32:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.823+0000 7fd8ec800640 1 -- 172.21.15.98:0/1120785540 wait complete. 2024-05-27T00:32:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 Processor -- start 2024-05-27T00:32:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 -- start start 2024-05-27T00:32:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 0x7fd8e4151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8e41530a0 0x7fd8e40c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e40c7cb0 0x7fd8e40cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e4152e30 con 0x7fd8cc0039f0 2024-05-27T00:32:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e40cb570 con 0x7fd8e41530a0 2024-05-27T00:32:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e40c9c10 con 0x7fd8e40c7cb0 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8e9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 0x7fd8e4151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8e9e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 0x7fd8e4151b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55282/0 (socket says 172.21.15.98:55282) 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8e9e00640 1 -- 172.21.15.98:0/1041013736 learned_addr learned my addr 172.21.15.98:0/1041013736 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8e9400640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8e41530a0 0x7fd8e40c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ea800640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e40c7cb0 0x7fd8e40cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8e9e00640 1 -- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e40c7cb0 msgr2=0x7fd8e40cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8e9e00640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e40c7cb0 0x7fd8e40cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8e9e00640 1 -- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8e41530a0 msgr2=0x7fd8e40c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8e9e00640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8e41530a0 0x7fd8e40c7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8e9e00640 1 -- 172.21.15.98:0/1041013736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e40c69f0 con 0x7fd8cc0039f0 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8e9e00640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 0x7fd8e4151b40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800c210 tx=0x7fd8cc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8de000640 1 -- 172.21.15.98:0/1041013736 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8001070 con 0x7fd8cc0039f0 2024-05-27T00:32:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8de000640 1 -- 172.21.15.98:0/1041013736 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8d80a5950 con 0x7fd8cc0039f0 2024-05-27T00:32:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8de000640 1 -- 172.21.15.98:0/1041013736 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d80047f0 con 0x7fd8cc0039f0 2024-05-27T00:32:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 -- 172.21.15.98:0/1041013736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e41f3780 con 0x7fd8cc0039f0 2024-05-27T00:32:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 -- 172.21.15.98:0/1041013736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e40c6c90 con 0x7fd8cc0039f0 2024-05-27T00:32:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.827+0000 7fd8ec800640 1 -- 172.21.15.98:0/1041013736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e4075cb0 con 0x7fd8cc0039f0 2024-05-27T00:32:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.831+0000 7fd8de000640 1 -- 172.21.15.98:0/1041013736 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd8d8004990 con 0x7fd8cc0039f0 2024-05-27T00:32:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.831+0000 7fd8de000640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8c80678c0 0x7fd8c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.831+0000 7fd8de000640 1 -- 172.21.15.98:0/1041013736 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7fd8d811fef0 con 0x7fd8cc0039f0 2024-05-27T00:32:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.831+0000 7fd8e9400640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8c80678c0 0x7fd8c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.831+0000 7fd8e9400640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8c80678c0 0x7fd8c8069d80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4007b50 tx=0x7fd8d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:03.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:03.835+0000 7fd8de000640 1 -- 172.21.15.98:0/1041013736 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd8d80169b0 con 0x7fd8cc0039f0 2024-05-27T00:32:04.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 -- 172.21.15.98:0/1041013736 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8e41f3a40 con 0x7fd8cc0039f0 2024-05-27T00:32:04.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8de000640 1 -- 172.21.15.98:0/1041013736 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6839 (secure 0 0 0) 0x7fd8d80f05f0 con 0x7fd8cc0039f0 2024-05-27T00:32:04.031 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:04.031 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:09.534234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":30,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6240,"qdb_cluster":[6240]},"id":1}]} 2024-05-27T00:32:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 68 2024-05-27T00:32:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 -- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8c80678c0 msgr2=0x7fd8c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8c80678c0 0x7fd8c8069d80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4007b50 tx=0x7fd8d4009040 comp rx=0 tx=0).stop 2024-05-27T00:32:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 -- 172.21.15.98:0/1041013736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 msgr2=0x7fd8e4151b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 0x7fd8e4151b40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800c210 tx=0x7fd8cc005840 comp rx=0 tx=0).stop 2024-05-27T00:32:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 -- 172.21.15.98:0/1041013736 shutdown_connections 2024-05-27T00:32:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8e40c7cb0 0x7fd8e40cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd8c80678c0 0x7fd8c8069d80 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8e41530a0 0x7fd8e40c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 --2- 172.21.15.98:0/1041013736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8cc0039f0 0x7fd8e4151b40 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.027+0000 7fd8ec800640 1 -- 172.21.15.98:0/1041013736 >> 172.21.15.98:0/1041013736 conn(0x7fd8e4072f90 msgr2=0x7fd8e41584a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.031+0000 7fd8ec800640 1 -- 172.21.15.98:0/1041013736 shutdown_connections 2024-05-27T00:32:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:04.031+0000 7fd8ec800640 1 -- 172.21.15.98:0/1041013736 wait complete. 2024-05-27T00:32:04.053 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:32:06.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:06.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.195+0000 7fc37f800640 1 Processor -- start 2024-05-27T00:32:06.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.195+0000 7fc37f800640 1 -- start start 2024-05-27T00:32:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37f800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3780735d0 0x7fc378151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37f800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc3780c79f0 0x7fc378151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37f800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3780c17d0 con 0x7fc3780731f0 2024-05-27T00:32:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37f800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc378071990 con 0x7fc3780c79f0 2024-05-27T00:32:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37f800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc378070e10 con 0x7fc3780735d0 2024-05-27T00:32:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37ce00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc3780731f0 0x7fc3780c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44418/0 (socket says 172.21.15.98:44418) 2024-05-27T00:32:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37ce00640 1 -- 172.21.15.98:0/3581171230 learned_addr learned my addr 172.21.15.98:0/3581171230 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:32:06.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37d800640 1 --2- 172.21.15.98:0/3581171230 >> v2:172.21.15.98:3300/0 conn(0x7fc3780c79f0 0x7fc378151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:06.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc377e00640 1 --2- 172.21.15.98:0/3581171230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3780735d0 0x7fc378151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:06.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc377400640 1 -- 172.21.15.98:0/3581171230 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3756343399 0 0) 0x7fc3780c17d0 con 0x7fc3780731f0 2024-05-27T00:32:06.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc377400640 1 -- 172.21.15.98:0/3581171230 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc360003530 con 0x7fc3780731f0 2024-05-27T00:32:06.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37d800640 1 -- 172.21.15.98:0/3581171230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3780735d0 msgr2=0x7fc378151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:06.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37d800640 1 --2- 172.21.15.98:0/3581171230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3780735d0 0x7fc378151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:06.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37d800640 1 -- 172.21.15.98:0/3581171230 >> v1:172.21.15.98:6789/0 conn(0x7fc3780731f0 legacy=0x7fc3780c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:06.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.199+0000 7fc37d800640 1 -- 172.21.15.98:0/3581171230 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3780710f0 con 0x7fc3780c79f0 2024-05-27T00:32:06.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37d800640 1 --2- 172.21.15.98:0/3581171230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780c79f0 0x7fc378151970 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc370009d90 tx=0x7fc37001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b864400f99c53d4f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc377400640 1 -- 172.21.15.98:0/3581171230 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3700b4070 con 0x7fc3780c79f0 2024-05-27T00:32:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc377400640 1 -- 172.21.15.98:0/3581171230 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3700af600 con 0x7fc3780c79f0 2024-05-27T00:32:06.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc377400640 1 -- 172.21.15.98:0/3581171230 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3700af9d0 con 0x7fc3780c79f0 2024-05-27T00:32:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37f800640 1 -- 172.21.15.98:0/3581171230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780c79f0 msgr2=0x7fc378151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37f800640 1 --2- 172.21.15.98:0/3581171230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780c79f0 0x7fc378151970 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc370009d90 tx=0x7fc37001f150 comp rx=0 tx=0).stop 2024-05-27T00:32:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37f800640 1 -- 172.21.15.98:0/3581171230 shutdown_connections 2024-05-27T00:32:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37f800640 1 --2- 172.21.15.98:0/3581171230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780c79f0 0x7fc378151970 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37f800640 1 --2- 172.21.15.98:0/3581171230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3780735d0 0x7fc378151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37f800640 1 -- 172.21.15.98:0/3581171230 >> 172.21.15.98:0/3581171230 conn(0x7fc378067530 msgr2=0x7fc37806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37f800640 1 -- 172.21.15.98:0/3581171230 shutdown_connections 2024-05-27T00:32:06.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37f800640 1 -- 172.21.15.98:0/3581171230 wait complete. 2024-05-27T00:32:06.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37f800640 1 Processor -- start 2024-05-27T00:32:06.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.203+0000 7fc37f800640 1 -- start start 2024-05-27T00:32:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37f800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3780731f0 0x7fc37815b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:06.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37f800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780735d0 0x7fc3781591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37f800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3780c79f0 0x7fc3781544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37f800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3780c8400 con 0x7fc3780735d0 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37f800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc3780716b0 con 0x7fc3780c79f0 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37f800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc378157970 con 0x7fc3780731f0 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc377e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780735d0 0x7fc3781591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc377e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780735d0 0x7fc3781591f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55314/0 (socket says 172.21.15.98:55314) 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc377e00640 1 -- 172.21.15.98:0/3796911945 learned_addr learned my addr 172.21.15.98:0/3796911945 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37ce00640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3780731f0 0x7fc37815b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc377e00640 1 -- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3780731f0 msgr2=0x7fc37815b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37d800640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3780c79f0 0x7fc3781544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc377e00640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3780731f0 0x7fc37815b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:06.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc377e00640 1 -- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3780c79f0 msgr2=0x7fc3781544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc377e00640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3780c79f0 0x7fc3781544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc377e00640 1 -- 172.21.15.98:0/3796911945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc378152010 con 0x7fc3780735d0 2024-05-27T00:32:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37d800640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3780c79f0 0x7fc3781544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:32:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc377e00640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780735d0 0x7fc3781591f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc36400d800 tx=0x7fc36400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc374c00640 1 -- 172.21.15.98:0/3796911945 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36400b6c0 con 0x7fc3780735d0 2024-05-27T00:32:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc374c00640 1 -- 172.21.15.98:0/3796911945 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc364019070 con 0x7fc3780735d0 2024-05-27T00:32:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37f800640 1 -- 172.21.15.98:0/3796911945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3781522f0 con 0x7fc3780735d0 2024-05-27T00:32:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc374c00640 1 -- 172.21.15.98:0/3796911945 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36400baa0 con 0x7fc3780735d0 2024-05-27T00:32:06.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37f800640 1 -- 172.21.15.98:0/3796911945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3781528b0 con 0x7fc3780735d0 2024-05-27T00:32:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc37f800640 1 -- 172.21.15.98:0/3796911945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc378075cb0 con 0x7fc3780735d0 2024-05-27T00:32:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.207+0000 7fc374c00640 1 -- 172.21.15.98:0/3796911945 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc36400bc40 con 0x7fc3780735d0 2024-05-27T00:32:06.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.211+0000 7fc374c00640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3580678c0 0x7fc358069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.211+0000 7fc374c00640 1 -- 172.21.15.98:0/3796911945 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7fc3640864c0 con 0x7fc3780735d0 2024-05-27T00:32:06.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.211+0000 7fc37ce00640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3580678c0 0x7fc358069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:06.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.211+0000 7fc37ce00640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3580678c0 0x7fc358069d80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc36c005890 tx=0x7fc36c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:06.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.215+0000 7fc374c00640 1 -- 172.21.15.98:0/3796911945 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc364056c60 con 0x7fc3780735d0 2024-05-27T00:32:06.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 -- 172.21.15.98:0/3796911945 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc378152a80 con 0x7fc3780735d0 2024-05-27T00:32:06.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc374c00640 1 -- 172.21.15.98:0/3796911945 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6839 (secure 0 0 0) 0x7fc364056560 con 0x7fc3780735d0 2024-05-27T00:32:06.407 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:06.407 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:09.534234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":30,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6240,"qdb_cluster":[6240]},"id":1}]} 2024-05-27T00:32:06.408 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 68 2024-05-27T00:32:06.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 -- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3580678c0 msgr2=0x7fc358069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:06.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3580678c0 0x7fc358069d80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc36c005890 tx=0x7fc36c009040 comp rx=0 tx=0).stop 2024-05-27T00:32:06.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 -- 172.21.15.98:0/3796911945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780735d0 msgr2=0x7fc3781591f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:06.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780735d0 0x7fc3781591f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc36400d800 tx=0x7fc36400dcc0 comp rx=0 tx=0).stop 2024-05-27T00:32:06.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 -- 172.21.15.98:0/3796911945 shutdown_connections 2024-05-27T00:32:06.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc3580678c0 0x7fc358069d80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:06.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3780c79f0 0x7fc3781544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:06.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3780735d0 0x7fc3781591f0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:06.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 --2- 172.21.15.98:0/3796911945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc3780731f0 0x7fc37815b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:06.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.403+0000 7fc37f800640 1 -- 172.21.15.98:0/3796911945 >> 172.21.15.98:0/3796911945 conn(0x7fc378067530 msgr2=0x7fc3780c5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:06.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.407+0000 7fc37f800640 1 -- 172.21.15.98:0/3796911945 shutdown_connections 2024-05-27T00:32:06.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:06.407+0000 7fc37f800640 1 -- 172.21.15.98:0/3796911945 wait complete. 2024-05-27T00:32:06.428 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:32:08.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.571+0000 7f3014c00640 1 Processor -- start 2024-05-27T00:32:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f3014c00640 1 -- start start 2024-05-27T00:32:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f3014c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30100cc750 0x7f30100c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f3014c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f30100ccb30 0x7f3010154620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f3014c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30100c74f0 con 0x7f30100c7fc0 2024-05-27T00:32:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f3014c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f301015b580 con 0x7f30100cc750 2024-05-27T00:32:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f3014c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f301015a970 con 0x7f30100ccb30 2024-05-27T00:32:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f300f400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f30100cc750 0x7f30100c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f300fe00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f30100c7fc0 0x7f30100c8480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44434/0 (socket says 172.21.15.98:44434) 2024-05-27T00:32:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f300fe00640 1 -- 172.21.15.98:0/377123164 learned_addr learned my addr 172.21.15.98:0/377123164 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:32:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f300ea00640 1 --2- 172.21.15.98:0/377123164 >> v2:172.21.15.204:3300/0 conn(0x7f30100ccb30 0x7f3010154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f300f400640 1 -- 172.21.15.98:0/377123164 >> v2:172.21.15.204:3300/0 conn(0x7f30100ccb30 msgr2=0x7f3010154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f300f400640 1 --2- 172.21.15.98:0/377123164 >> v2:172.21.15.204:3300/0 conn(0x7f30100ccb30 0x7f3010154620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f300f400640 1 -- 172.21.15.98:0/377123164 >> v1:172.21.15.98:6789/0 conn(0x7f30100c7fc0 legacy=0x7f30100c8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f300f400640 1 -- 172.21.15.98:0/377123164 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301015ac50 con 0x7f30100cc750 2024-05-27T00:32:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.575+0000 7f300e000640 1 -- 172.21.15.98:0/377123164 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3569441561 0 0) 0x7f30100c74f0 con 0x7f30100c7fc0 2024-05-27T00:32:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f300f400640 1 --2- 172.21.15.98:0/377123164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 0x7f30100c6e60 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3000009d90 tx=0x7f300001f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca8f0533bd3a578 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f300e000640 1 -- 172.21.15.98:0/377123164 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30000b4070 con 0x7f30100cc750 2024-05-27T00:32:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f300e000640 1 -- 172.21.15.98:0/377123164 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30000af600 con 0x7f30100cc750 2024-05-27T00:32:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f300e000640 1 -- 172.21.15.98:0/377123164 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30000af9d0 con 0x7f30100cc750 2024-05-27T00:32:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f3014c00640 1 -- 172.21.15.98:0/377123164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 msgr2=0x7f30100c6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f3014c00640 1 --2- 172.21.15.98:0/377123164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 0x7f30100c6e60 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3000009d90 tx=0x7f300001f120 comp rx=0 tx=0).stop 2024-05-27T00:32:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f3014c00640 1 -- 172.21.15.98:0/377123164 shutdown_connections 2024-05-27T00:32:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f3014c00640 1 --2- 172.21.15.98:0/377123164 >> v2:172.21.15.204:3300/0 conn(0x7f30100ccb30 0x7f3010154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f3014c00640 1 --2- 172.21.15.98:0/377123164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 0x7f30100c6e60 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f3014c00640 1 -- 172.21.15.98:0/377123164 >> 172.21.15.98:0/377123164 conn(0x7f301006dd00 msgr2=0x7f301006b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f3014c00640 1 -- 172.21.15.98:0/377123164 shutdown_connections 2024-05-27T00:32:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f3014c00640 1 -- 172.21.15.98:0/377123164 wait complete. 2024-05-27T00:32:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f3014c00640 1 Processor -- start 2024-05-27T00:32:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.579+0000 7f3014c00640 1 -- start start 2024-05-27T00:32:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3014c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30100c7fc0 0x7f30100cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3014c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 0x7f3010151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3014c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30100ccb30 0x7f30101522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3014c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30100c8b90 con 0x7f30100cc750 2024-05-27T00:32:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3014c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f301015b210 con 0x7f30100ccb30 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3014c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f30100c8d10 con 0x7f30100c7fc0 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 0x7f3010151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 0x7f3010151da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55326/0 (socket says 172.21.15.98:55326) 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300ea00640 1 -- 172.21.15.98:0/2245251176 learned_addr learned my addr 172.21.15.98:0/2245251176 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300fe00640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30100ccb30 0x7f30101522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300f400640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30100c7fc0 0x7f30100cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300ea00640 1 -- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30100c7fc0 msgr2=0x7f30100cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300ea00640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30100c7fc0 0x7f30100cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300ea00640 1 -- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30100ccb30 msgr2=0x7f30101522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300ea00640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30100ccb30 0x7f30101522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300ea00640 1 -- 172.21.15.98:0/2245251176 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30101528c0 con 0x7f30100cc750 2024-05-27T00:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f300ea00640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 0x7f3010151da0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00d620 tx=0x7f2ffc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3007400640 1 -- 172.21.15.98:0/2245251176 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc00b620 con 0x7f30100cc750 2024-05-27T00:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3007400640 1 -- 172.21.15.98:0/2245251176 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ffc019070 con 0x7f30100cc750 2024-05-27T00:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3014c00640 1 -- 172.21.15.98:0/2245251176 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3010152ba0 con 0x7f30100cc750 2024-05-27T00:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3007400640 1 -- 172.21.15.98:0/2245251176 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc00b9e0 con 0x7f30100cc750 2024-05-27T00:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3014c00640 1 -- 172.21.15.98:0/2245251176 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3010153160 con 0x7f30100cc750 2024-05-27T00:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3014c00640 1 -- 172.21.15.98:0/2245251176 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3010075c40 con 0x7f30100cc750 2024-05-27T00:32:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.583+0000 7f3007400640 1 -- 172.21.15.98:0/2245251176 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2ffc00bb80 con 0x7f30100cc750 2024-05-27T00:32:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.587+0000 7f3007400640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ff00678f0 0x7f2ff0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.587+0000 7f3007400640 1 -- 172.21.15.98:0/2245251176 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6771+0+0 (secure 0 0 0) 0x7f2ffc086470 con 0x7f30100cc750 2024-05-27T00:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.587+0000 7f300f400640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ff00678f0 0x7f2ff0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.587+0000 7f300f400640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ff00678f0 0x7f2ff0069db0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f30000b1000 tx=0x7f30000aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:08.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.591+0000 7f3007400640 1 -- 172.21.15.98:0/2245251176 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f2ffc056b60 con 0x7f30100cc750 2024-05-27T00:32:08.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.779+0000 7f3014c00640 1 -- 172.21.15.98:0/2245251176 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f301014ee40 con 0x7f30100cc750 2024-05-27T00:32:08.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.779+0000 7f3007400640 1 -- 172.21.15.98:0/2245251176 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6839 (secure 0 0 0) 0x7f2ffc056460 con 0x7f30100cc750 2024-05-27T00:32:08.784 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:08.784 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:31:09.534234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6240":{"gid":6240,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":30,"addr":"172.21.15.98:6834/3070334683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3070334683},{"type":"v1","addr":"172.21.15.98:6834","nonce":3070334683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6240,"qdb_cluster":[6240]},"id":1}]} 2024-05-27T00:32:08.784 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 68 2024-05-27T00:32:08.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 -- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ff00678f0 msgr2=0x7f2ff0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:08.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ff00678f0 0x7f2ff0069db0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f30000b1000 tx=0x7f30000aa000 comp rx=0 tx=0).stop 2024-05-27T00:32:08.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 -- 172.21.15.98:0/2245251176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 msgr2=0x7f3010151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:08.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 0x7f3010151da0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00d620 tx=0x7f2ffc00dae0 comp rx=0 tx=0).stop 2024-05-27T00:32:08.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 -- 172.21.15.98:0/2245251176 shutdown_connections 2024-05-27T00:32:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f2ff00678f0 0x7f2ff0069db0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f30100ccb30 0x7f30101522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30100cc750 0x7f3010151da0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 --2- 172.21.15.98:0/2245251176 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30100c7fc0 0x7f30100cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 -- 172.21.15.98:0/2245251176 >> 172.21.15.98:0/2245251176 conn(0x7f301006dd00 msgr2=0x7f301014d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 -- 172.21.15.98:0/2245251176 shutdown_connections 2024-05-27T00:32:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:08.783+0000 7f3014c00640 1 -- 172.21.15.98:0/2245251176 wait complete. 2024-05-27T00:32:08.806 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T00:32:10.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa422000640 1 Processor -- start 2024-05-27T00:32:10.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa422000640 1 -- start start 2024-05-27T00:32:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa422000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa41c154de0 0x7fa41c0b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa422000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa41c0d0070 0x7fa41c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa422000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c0b3f20 con 0x7fa41c154a00 2024-05-27T00:32:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa422000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa41c0b4200 con 0x7fa41c154de0 2024-05-27T00:32:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa422000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa41c0b44e0 con 0x7fa41c0d0070 2024-05-27T00:32:10.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa41b400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa41c154a00 0x7fa41c15b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:44442/0 (socket says 172.21.15.98:44442) 2024-05-27T00:32:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa41b400640 1 -- 172.21.15.98:0/1834553402 learned_addr learned my addr 172.21.15.98:0/1834553402 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:32:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa412a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa41c154de0 0x7fa41c0b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa41be00640 1 --2- 172.21.15.98:0/1834553402 >> v2:172.21.15.204:3300/0 conn(0x7fa41c0d0070 0x7fa41c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa412a00640 1 -- 172.21.15.98:0/1834553402 >> v2:172.21.15.204:3300/0 conn(0x7fa41c0d0070 msgr2=0x7fa41c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa412a00640 1 --2- 172.21.15.98:0/1834553402 >> v2:172.21.15.204:3300/0 conn(0x7fa41c0d0070 0x7fa41c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa412a00640 1 -- 172.21.15.98:0/1834553402 >> v1:172.21.15.98:6789/0 conn(0x7fa41c154a00 legacy=0x7fa41c15b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa412a00640 1 -- 172.21.15.98:0/1834553402 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c0b47c0 con 0x7fa41c154de0 2024-05-27T00:32:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa41aa00640 1 -- 172.21.15.98:0/1834553402 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3978231811 0 0) 0x7fa41c0b3f20 con 0x7fa41c154a00 2024-05-27T00:32:10.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.951+0000 7fa41be00640 1 --2- 172.21.15.98:0/1834553402 >> v2:172.21.15.204:3300/0 conn(0x7fa41c0d0070 0x7fa41c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:32:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa412a00640 1 --2- 172.21.15.98:0/1834553402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154de0 0x7fa41c0b2c50 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0caef0 tx=0x7fa40401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c191b23582550d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa41aa00640 1 -- 172.21.15.98:0/1834553402 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4040b5070 con 0x7fa41c154de0 2024-05-27T00:32:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa41aa00640 1 -- 172.21.15.98:0/1834553402 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4040a8950 con 0x7fa41c154de0 2024-05-27T00:32:10.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa41aa00640 1 -- 172.21.15.98:0/1834553402 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4040a8d20 con 0x7fa41c154de0 2024-05-27T00:32:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa422000640 1 -- 172.21.15.98:0/1834553402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154de0 msgr2=0x7fa41c0b2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa422000640 1 --2- 172.21.15.98:0/1834553402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154de0 0x7fa41c0b2c50 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0caef0 tx=0x7fa40401ec50 comp rx=0 tx=0).stop 2024-05-27T00:32:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa422000640 1 -- 172.21.15.98:0/1834553402 shutdown_connections 2024-05-27T00:32:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa422000640 1 --2- 172.21.15.98:0/1834553402 >> v2:172.21.15.204:3300/0 conn(0x7fa41c0d0070 0x7fa41c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa422000640 1 --2- 172.21.15.98:0/1834553402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154de0 0x7fa41c0b2c50 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa422000640 1 -- 172.21.15.98:0/1834553402 >> 172.21.15.98:0/1834553402 conn(0x7fa41c067530 msgr2=0x7fa41c06c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:10.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa422000640 1 -- 172.21.15.98:0/1834553402 shutdown_connections 2024-05-27T00:32:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.955+0000 7fa422000640 1 -- 172.21.15.98:0/1834553402 wait complete. 2024-05-27T00:32:10.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 Processor -- start 2024-05-27T00:32:10.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 -- start start 2024-05-27T00:32:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa41c0d0070 0x7fa41c14c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154a00 0x7fa41c1f0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:10.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa41c154de0 0x7fa41c1f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c0b37a0 con 0x7fa41c154a00 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c0b3620 con 0x7fa41c154de0 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa41c0cc7e0 con 0x7fa41c0d0070 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa412a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154a00 0x7fa41c1f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa412a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154a00 0x7fa41c1f0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55348/0 (socket says 172.21.15.98:55348) 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa412a00640 1 -- 172.21.15.98:0/3027727310 learned_addr learned my addr 172.21.15.98:0/3027727310 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa41b400640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa41c0d0070 0x7fa41c14c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa41be00640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa41c154de0 0x7fa41c1f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa412a00640 1 -- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa41c0d0070 msgr2=0x7fa41c14c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa412a00640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa41c0d0070 0x7fa41c14c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa412a00640 1 -- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa41c154de0 msgr2=0x7fa41c1f0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa412a00640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa41c154de0 0x7fa41c1f0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:10.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa412a00640 1 -- 172.21.15.98:0/3027727310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c15a6e0 con 0x7fa41c154a00 2024-05-27T00:32:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa412a00640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154a00 0x7fa41c1f0740 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa4040b3040 tx=0x7fa40401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa413e00640 1 -- 172.21.15.98:0/3027727310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4040bf070 con 0x7fa41c154a00 2024-05-27T00:32:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa413e00640 1 -- 172.21.15.98:0/3027727310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4040b0800 con 0x7fa41c154a00 2024-05-27T00:32:10.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 -- 172.21.15.98:0/3027727310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa41c1f3980 con 0x7fa41c154a00 2024-05-27T00:32:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa413e00640 1 -- 172.21.15.98:0/3027727310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4040b5040 con 0x7fa41c154a00 2024-05-27T00:32:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 -- 172.21.15.98:0/3027727310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa41c1f3f20 con 0x7fa41c154a00 2024-05-27T00:32:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.959+0000 7fa422000640 1 -- 172.21.15.98:0/3027727310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa41c075cb0 con 0x7fa41c154a00 2024-05-27T00:32:10.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.963+0000 7fa413e00640 1 -- 172.21.15.98:0/3027727310 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa4040c2070 con 0x7fa41c154a00 2024-05-27T00:32:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.963+0000 7fa413e00640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa3f80678c0 0x7fa3f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.963+0000 7fa413e00640 1 -- 172.21.15.98:0/3027727310 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fa404122d30 con 0x7fa41c154a00 2024-05-27T00:32:10.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.963+0000 7fa41b400640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa3f80678c0 0x7fa3f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:10.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.963+0000 7fa41b400640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa3f80678c0 0x7fa3f8069d80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa40c007fd0 tx=0x7fa40c007ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:10.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:10.967+0000 7fa413e00640 1 -- 172.21.15.98:0/3027727310 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa4040187b0 con 0x7fa41c154a00 2024-05-27T00:32:11.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.155+0000 7fa422000640 1 -- 172.21.15.98:0/3027727310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa41c15b410 con 0x7fa41c154a00 2024-05-27T00:32:11.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.155+0000 7fa413e00640 1 -- 172.21.15.98:0/3027727310 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+6006 (secure 0 0 0) 0x7fa4040f32c0 con 0x7fa41c154a00 2024-05-27T00:32:11.160 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:11.161 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:10.313350+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:32:11.161 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 69 2024-05-27T00:32:11.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 -- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa3f80678c0 msgr2=0x7fa3f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:11.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa3f80678c0 0x7fa3f8069d80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa40c007fd0 tx=0x7fa40c007ee0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 -- 172.21.15.98:0/3027727310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154a00 msgr2=0x7fa41c1f0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:11.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154a00 0x7fa41c1f0740 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa4040b3040 tx=0x7fa40401b9f0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 -- 172.21.15.98:0/3027727310 shutdown_connections 2024-05-27T00:32:11.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa3f80678c0 0x7fa3f8069d80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa41c154de0 0x7fa41c1f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa41c154a00 0x7fa41c1f0740 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 --2- 172.21.15.98:0/3027727310 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa41c0d0070 0x7fa41c14c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 -- 172.21.15.98:0/3027727310 >> 172.21.15.98:0/3027727310 conn(0x7fa41c067530 msgr2=0x7fa41c0ce860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:11.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 -- 172.21.15.98:0/3027727310 shutdown_connections 2024-05-27T00:32:11.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.159+0000 7fa422000640 1 -- 172.21.15.98:0/3027727310 wait complete. 2024-05-27T00:32:11.181 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-05-27T00:32:11.182 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T00:32:11.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.311+0000 7f9533400640 1 Processor -- start 2024-05-27T00:32:11.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.311+0000 7f9533400640 1 -- start start 2024-05-27T00:32:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9533400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0731f0 0x7f952c0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9533400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f952c0735d0 0x7f952c151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9533400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f952c0c79f0 0x7f952c151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9533400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f952c071990 con 0x7f952c0c79f0 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9533400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f952c0c17d0 con 0x7f952c0735d0 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9533400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f952c070e10 con 0x7f952c0731f0 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9531400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f952c0c79f0 0x7f952c151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9531400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f952c0c79f0 0x7f952c151970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55362/0 (socket says 172.21.15.98:55362) 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9531400640 1 -- 172.21.15.98:0/2672639589 learned_addr learned my addr 172.21.15.98:0/2672639589 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9530a00640 1 --2- 172.21.15.98:0/2672639589 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0731f0 0x7f952c0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9523e00640 1 --2- 172.21.15.98:0/2672639589 >> v2:172.21.15.204:3300/0 conn(0x7f952c0735d0 0x7f952c151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9531400640 1 -- 172.21.15.98:0/2672639589 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0731f0 msgr2=0x7f952c0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9531400640 1 --2- 172.21.15.98:0/2672639589 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0731f0 0x7f952c0c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9531400640 1 -- 172.21.15.98:0/2672639589 >> v2:172.21.15.204:3300/0 conn(0x7f952c0735d0 msgr2=0x7f952c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9531400640 1 --2- 172.21.15.98:0/2672639589 >> v2:172.21.15.204:3300/0 conn(0x7f952c0735d0 0x7f952c151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.315+0000 7f9531400640 1 -- 172.21.15.98:0/2672639589 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952c0710f0 con 0x7f952c0c79f0 2024-05-27T00:32:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9531400640 1 --2- 172.21.15.98:0/2672639589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0c79f0 0x7f952c151970 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f952c0cc770 tx=0x7f952401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47d75741f61b9fb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f952be00640 1 -- 172.21.15.98:0/2672639589 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95240b4070 con 0x7f952c0c79f0 2024-05-27T00:32:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f952be00640 1 -- 172.21.15.98:0/2672639589 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95240af400 con 0x7f952c0c79f0 2024-05-27T00:32:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f952be00640 1 -- 172.21.15.98:0/2672639589 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95240af7d0 con 0x7f952c0c79f0 2024-05-27T00:32:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 -- 172.21.15.98:0/2672639589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0c79f0 msgr2=0x7f952c151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 --2- 172.21.15.98:0/2672639589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0c79f0 0x7f952c151970 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f952c0cc770 tx=0x7f952401f1e0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 -- 172.21.15.98:0/2672639589 shutdown_connections 2024-05-27T00:32:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 --2- 172.21.15.98:0/2672639589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0c79f0 0x7f952c151970 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 --2- 172.21.15.98:0/2672639589 >> v2:172.21.15.204:3300/0 conn(0x7f952c0735d0 0x7f952c151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 --2- 172.21.15.98:0/2672639589 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0731f0 0x7f952c0c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 -- 172.21.15.98:0/2672639589 >> 172.21.15.98:0/2672639589 conn(0x7f952c067530 msgr2=0x7f952c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 -- 172.21.15.98:0/2672639589 shutdown_connections 2024-05-27T00:32:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 -- 172.21.15.98:0/2672639589 wait complete. 2024-05-27T00:32:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 Processor -- start 2024-05-27T00:32:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 -- start start 2024-05-27T00:32:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f952c0731f0 0x7f952c14d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0735d0 0x7f952c163030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0c79f0 0x7f952c1f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f952c0c8400 con 0x7f952c0735d0 2024-05-27T00:32:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f952c0716b0 con 0x7f952c0731f0 2024-05-27T00:32:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9533400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f952c0c13f0 con 0x7f952c0c79f0 2024-05-27T00:32:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9523e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0735d0 0x7f952c163030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.319+0000 7f9523e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0735d0 0x7f952c163030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55374/0 (socket says 172.21.15.98:55374) 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9523e00640 1 -- 172.21.15.98:0/782523501 learned_addr learned my addr 172.21.15.98:0/782523501 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9530a00640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f952c0731f0 0x7f952c14d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9531400640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0c79f0 0x7f952c1f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9523e00640 1 -- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0c79f0 msgr2=0x7f952c1f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9523e00640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0c79f0 0x7f952c1f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9523e00640 1 -- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f952c0731f0 msgr2=0x7f952c14d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9523e00640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f952c0731f0 0x7f952c14d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9523e00640 1 -- 172.21.15.98:0/782523501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f952c163950 con 0x7f952c0735d0 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9530a00640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f952c0731f0 0x7f952c14d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9531400640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0c79f0 0x7f952c1f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9523e00640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0735d0 0x7f952c163030 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f951400d800 tx=0x7f951400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:11.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9529600640 1 -- 172.21.15.98:0/782523501 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951400b6c0 con 0x7f952c0735d0 2024-05-27T00:32:11.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9529600640 1 -- 172.21.15.98:0/782523501 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9514019070 con 0x7f952c0735d0 2024-05-27T00:32:11.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9529600640 1 -- 172.21.15.98:0/782523501 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951400baa0 con 0x7f952c0735d0 2024-05-27T00:32:11.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9533400640 1 -- 172.21.15.98:0/782523501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f952c1f3d40 con 0x7f952c0735d0 2024-05-27T00:32:11.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9533400640 1 -- 172.21.15.98:0/782523501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f952c1f4300 con 0x7f952c0735d0 2024-05-27T00:32:11.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9533400640 1 -- 172.21.15.98:0/782523501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f952c075cb0 con 0x7f952c0735d0 2024-05-27T00:32:11.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9529600640 1 -- 172.21.15.98:0/782523501 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f951400bc40 con 0x7f952c0735d0 2024-05-27T00:32:11.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9529600640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f95080678f0 0x7f9508069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:11.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.323+0000 7f9529600640 1 -- 172.21.15.98:0/782523501 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f9514086620 con 0x7f952c0735d0 2024-05-27T00:32:11.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.327+0000 7f9530a00640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f95080678f0 0x7f9508069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:11.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.327+0000 7f9530a00640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f95080678f0 0x7f9508069db0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f951c004660 tx=0x7f951c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.327+0000 7f9529600640 1 -- 172.21.15.98:0/782523501 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f9514056c40 con 0x7f952c0735d0 2024-05-27T00:32:11.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 -- 172.21.15.98:0/782523501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f952c150a80 con 0x7f952c0735d0 2024-05-27T00:32:11.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9529600640 1 -- 172.21.15.98:0/782523501 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+6006 (secure 0 0 0) 0x7f9514056540 con 0x7f952c0735d0 2024-05-27T00:32:11.528 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:11.528 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:10.313350+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:32:11.529 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 69 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 -- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f95080678f0 msgr2=0x7f9508069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f95080678f0 0x7f9508069db0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f951c004660 tx=0x7f951c008040 comp rx=0 tx=0).stop 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 -- 172.21.15.98:0/782523501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0735d0 msgr2=0x7f952c163030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0735d0 0x7f952c163030 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f951400d800 tx=0x7f951400dcc0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 -- 172.21.15.98:0/782523501 shutdown_connections 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f952c0c79f0 0x7f952c1f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c0735d0 0x7f952c163030 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f95080678f0 0x7f9508069db0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 --2- 172.21.15.98:0/782523501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f952c0731f0 0x7f952c14d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.523+0000 7f9533400640 1 -- 172.21.15.98:0/782523501 >> 172.21.15.98:0/782523501 conn(0x7f952c067530 msgr2=0x7f952c151200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.527+0000 7f9533400640 1 -- 172.21.15.98:0/782523501 shutdown_connections 2024-05-27T00:32:11.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:11.527+0000 7f9533400640 1 -- 172.21.15.98:0/782523501 wait complete. 2024-05-27T00:32:11.549 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 69, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 69, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6396": { "addr": "172.21.15.98:6837/2042067068", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 2042067068, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2042067068, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6396, "incarnation": 69, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:32:10.313350+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6396 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:32:13.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b54400640 1 Processor -- start 2024-05-27T00:32:13.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b54400640 1 -- start start 2024-05-27T00:32:13.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b54400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3b4c062720 0x7f3b4c072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:13.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b54400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b4c15e6b0 0x7f3b4c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b54400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b4c0700d0 con 0x7f3b4c154a80 2024-05-27T00:32:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b54400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3b4c0726a0 con 0x7f3b4c062720 2024-05-27T00:32:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b54400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3b4c071a90 con 0x7f3b4c15e6b0 2024-05-27T00:32:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b51a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3b4c154a80 0x7f3b4c0c6f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39354/0 (socket says 172.21.15.98:39354) 2024-05-27T00:32:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b51a00640 1 -- 172.21.15.98:0/1583961904 learned_addr learned my addr 172.21.15.98:0/1583961904 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:32:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b51000640 1 --2- 172.21.15.98:0/1583961904 >> v2:172.21.15.98:3300/0 conn(0x7f3b4c062720 0x7f3b4c072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b52400640 1 --2- 172.21.15.98:0/1583961904 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b4c15e6b0 0x7f3b4c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:13.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b4be00640 1 -- 172.21.15.98:0/1583961904 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2428463490 0 0) 0x7f3b4c0700d0 con 0x7f3b4c154a80 2024-05-27T00:32:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b51000640 1 -- 172.21.15.98:0/1583961904 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b4c15e6b0 msgr2=0x7f3b4c06ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b51000640 1 --2- 172.21.15.98:0/1583961904 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b4c15e6b0 0x7f3b4c06ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b51000640 1 -- 172.21.15.98:0/1583961904 >> v1:172.21.15.98:6789/0 conn(0x7f3b4c154a80 legacy=0x7f3b4c0c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:13.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.699+0000 7f3b51000640 1 -- 172.21.15.98:0/1583961904 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4c071d70 con 0x7f3b4c062720 2024-05-27T00:32:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b51000640 1 --2- 172.21.15.98:0/1583961904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 0x7f3b4c072b30 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3b380097c0 tx=0x7f3b3801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8c7d4e42bf3c75 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b4be00640 1 -- 172.21.15.98:0/1583961904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b380b0070 con 0x7f3b4c062720 2024-05-27T00:32:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b4be00640 1 -- 172.21.15.98:0/1583961904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b380ab600 con 0x7f3b4c062720 2024-05-27T00:32:13.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b4be00640 1 -- 172.21.15.98:0/1583961904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b380ab9d0 con 0x7f3b4c062720 2024-05-27T00:32:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b54400640 1 -- 172.21.15.98:0/1583961904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 msgr2=0x7f3b4c072b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b54400640 1 --2- 172.21.15.98:0/1583961904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 0x7f3b4c072b30 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3b380097c0 tx=0x7f3b3801c410 comp rx=0 tx=0).stop 2024-05-27T00:32:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b54400640 1 -- 172.21.15.98:0/1583961904 shutdown_connections 2024-05-27T00:32:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b54400640 1 --2- 172.21.15.98:0/1583961904 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b4c15e6b0 0x7f3b4c06ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b54400640 1 --2- 172.21.15.98:0/1583961904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 0x7f3b4c072b30 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:13.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b54400640 1 -- 172.21.15.98:0/1583961904 >> 172.21.15.98:0/1583961904 conn(0x7f3b4c0cc560 msgr2=0x7f3b4c0cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b54400640 1 -- 172.21.15.98:0/1583961904 shutdown_connections 2024-05-27T00:32:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.703+0000 7f3b54400640 1 -- 172.21.15.98:0/1583961904 wait complete. 2024-05-27T00:32:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 Processor -- start 2024-05-27T00:32:13.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 -- start start 2024-05-27T00:32:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 0x7f3b4c144400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b4c154a80 0x7f3b4c0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b4c15e6b0 0x7f3b4c1410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4c167210 con 0x7f3b4c062720 2024-05-27T00:32:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4c167090 con 0x7f3b4c15e6b0 2024-05-27T00:32:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3b4c167390 con 0x7f3b4c154a80 2024-05-27T00:32:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b51a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 0x7f3b4c144400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:13.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b51a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 0x7f3b4c144400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39226/0 (socket says 172.21.15.98:39226) 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b51a00640 1 -- 172.21.15.98:0/2213688276 learned_addr learned my addr 172.21.15.98:0/2213688276 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b51000640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b4c154a80 0x7f3b4c0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b52400640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b4c15e6b0 0x7f3b4c1410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b51a00640 1 -- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b4c154a80 msgr2=0x7f3b4c0ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b51a00640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b4c154a80 0x7f3b4c0ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b51a00640 1 -- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b4c15e6b0 msgr2=0x7f3b4c1410a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b51a00640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b4c15e6b0 0x7f3b4c1410a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b51a00640 1 -- 172.21.15.98:0/2213688276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4c0ca8a0 con 0x7f3b4c062720 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b51a00640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 0x7f3b4c144400 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3b4000c520 tx=0x7f3b4000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b49600640 1 -- 172.21.15.98:0/2213688276 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b40018070 con 0x7f3b4c062720 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b49600640 1 -- 172.21.15.98:0/2213688276 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b400074e0 con 0x7f3b4c062720 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b49600640 1 -- 172.21.15.98:0/2213688276 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b400049f0 con 0x7f3b4c062720 2024-05-27T00:32:13.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 -- 172.21.15.98:0/2213688276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b4c148a50 con 0x7f3b4c062720 2024-05-27T00:32:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 -- 172.21.15.98:0/2213688276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b4c148230 con 0x7f3b4c062720 2024-05-27T00:32:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.707+0000 7f3b54400640 1 -- 172.21.15.98:0/2213688276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b4c075cb0 con 0x7f3b4c062720 2024-05-27T00:32:13.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.711+0000 7f3b49600640 1 -- 172.21.15.98:0/2213688276 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3b40004b90 con 0x7f3b4c062720 2024-05-27T00:32:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.711+0000 7f3b49600640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b2c0678c0 0x7f3b2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.711+0000 7f3b49600640 1 -- 172.21.15.98:0/2213688276 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f3b400852d0 con 0x7f3b4c062720 2024-05-27T00:32:13.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.711+0000 7f3b51000640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b2c0678c0 0x7f3b2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:13.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.711+0000 7f3b51000640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b2c0678c0 0x7f3b2c069d80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3b380ad000 tx=0x7f3b380a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:13.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.715+0000 7f3b49600640 1 -- 172.21.15.98:0/2213688276 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f3b400553d0 con 0x7f3b4c062720 2024-05-27T00:32:13.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 -- 172.21.15.98:0/2213688276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b4c15c130 con 0x7f3b4c062720 2024-05-27T00:32:13.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b49600640 1 -- 172.21.15.98:0/2213688276 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+6006 (secure 0 0 0) 0x7f3b400596d0 con 0x7f3b4c062720 2024-05-27T00:32:13.907 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:13.907 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:10.313350+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:32:13.907 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 69 2024-05-27T00:32:13.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 -- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b2c0678c0 msgr2=0x7f3b2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b2c0678c0 0x7f3b2c069d80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3b380ad000 tx=0x7f3b380a6000 comp rx=0 tx=0).stop 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 -- 172.21.15.98:0/2213688276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 msgr2=0x7f3b4c144400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 0x7f3b4c144400 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3b4000c520 tx=0x7f3b4000c9e0 comp rx=0 tx=0).stop 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 -- 172.21.15.98:0/2213688276 shutdown_connections 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f3b2c0678c0 0x7f3b2c069d80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b4c15e6b0 0x7f3b4c1410a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3b4c154a80 0x7f3b4c0ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 --2- 172.21.15.98:0/2213688276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3b4c062720 0x7f3b4c144400 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 -- 172.21.15.98:0/2213688276 >> 172.21.15.98:0/2213688276 conn(0x7f3b4c0cc560 msgr2=0x7f3b4c0cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 -- 172.21.15.98:0/2213688276 shutdown_connections 2024-05-27T00:32:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:13.903+0000 7f3b54400640 1 -- 172.21.15.98:0/2213688276 wait complete. 2024-05-27T00:32:13.929 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:32:15.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:16.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2cb400640 1 Processor -- start 2024-05-27T00:32:16.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2cb400640 1 -- start start 2024-05-27T00:32:16.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2cb400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2c415b880 0x7fc2c40c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:16.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2cb400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2c4067fe0 con 0x7fc2c40c7960 2024-05-27T00:32:16.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2cb400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2c40c6e80 con 0x7fc2c406b470 2024-05-27T00:32:16.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2cb400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc2c40c6c30 con 0x7fc2c415b880 2024-05-27T00:32:16.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2c8a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc2c415b880 0x7fc2c40c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:16.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2c9400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc2c40c7960 0x7fc2c4165370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39368/0 (socket says 172.21.15.98:39368) 2024-05-27T00:32:16.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2c9400640 1 -- 172.21.15.98:0/2247506343 learned_addr learned my addr 172.21.15.98:0/2247506343 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:32:16.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2c8a00640 1 -- 172.21.15.98:0/2247506343 >> v1:172.21.15.204:6789/0 conn(0x7fc2c406b470 legacy=0x7fc2c4067130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:16.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2c8a00640 1 -- 172.21.15.98:0/2247506343 >> v1:172.21.15.98:6789/0 conn(0x7fc2c40c7960 legacy=0x7fc2c4165370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:16.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.087+0000 7fc2c8a00640 1 -- 172.21.15.98:0/2247506343 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c40c7160 con 0x7fc2c415b880 2024-05-27T00:32:16.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2c8a00640 1 --2- 172.21.15.98:0/2247506343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 0x7fc2c40c93d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80097c0 tx=0x7fc2b801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562ada65d626b278 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:16.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2c3400640 1 -- 172.21.15.98:0/2247506343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b80b0070 con 0x7fc2c415b880 2024-05-27T00:32:16.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2c3400640 1 -- 172.21.15.98:0/2247506343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2b80ab600 con 0x7fc2c415b880 2024-05-27T00:32:16.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2c3400640 1 -- 172.21.15.98:0/2247506343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b80a4e60 con 0x7fc2c415b880 2024-05-27T00:32:16.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2cb400640 1 -- 172.21.15.98:0/2247506343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 msgr2=0x7fc2c40c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:16.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2cb400640 1 --2- 172.21.15.98:0/2247506343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 0x7fc2c40c93d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80097c0 tx=0x7fc2b801c3d0 comp rx=0 tx=0).stop 2024-05-27T00:32:16.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2cb400640 1 -- 172.21.15.98:0/2247506343 shutdown_connections 2024-05-27T00:32:16.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2cb400640 1 --2- 172.21.15.98:0/2247506343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 0x7fc2c40c93d0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:16.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2cb400640 1 -- 172.21.15.98:0/2247506343 >> 172.21.15.98:0/2247506343 conn(0x7fc2c40cd060 msgr2=0x7fc2c40cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:16.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2cb400640 1 -- 172.21.15.98:0/2247506343 shutdown_connections 2024-05-27T00:32:16.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2cb400640 1 -- 172.21.15.98:0/2247506343 wait complete. 2024-05-27T00:32:16.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.091+0000 7fc2cb400640 1 Processor -- start 2024-05-27T00:32:16.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2cb400640 1 -- start start 2024-05-27T00:32:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2cb400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2c406b470 0x7fc2c415d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2cb400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2c40c7960 0x7fc2c4150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2cb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 0x7fc2c415c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2cb400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c40738f0 con 0x7fc2c415b880 2024-05-27T00:32:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2cb400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c4073770 con 0x7fc2c406b470 2024-05-27T00:32:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2cb400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc2c4067840 con 0x7fc2c40c7960 2024-05-27T00:32:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c9400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 0x7fc2c415c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c9400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 0x7fc2c415c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39258/0 (socket says 172.21.15.98:39258) 2024-05-27T00:32:16.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c9400640 1 -- 172.21.15.98:0/1941892008 learned_addr learned my addr 172.21.15.98:0/1941892008 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c8a00640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2c406b470 0x7fc2c415d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c3e00640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2c40c7960 0x7fc2c4150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c9400640 1 -- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2c40c7960 msgr2=0x7fc2c4150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c9400640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2c40c7960 0x7fc2c4150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c9400640 1 -- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2c406b470 msgr2=0x7fc2c415d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c9400640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2c406b470 0x7fc2c415d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c9400640 1 -- 172.21.15.98:0/1941892008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c41f3d80 con 0x7fc2c415b880 2024-05-27T00:32:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c9400640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 0x7fc2c415c1e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00cab0 tx=0x7fc2bc00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c0c00640 1 -- 172.21.15.98:0/1941892008 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc017070 con 0x7fc2c415b880 2024-05-27T00:32:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c0c00640 1 -- 172.21.15.98:0/1941892008 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2bc00ac30 con 0x7fc2c415b880 2024-05-27T00:32:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2c0c00640 1 -- 172.21.15.98:0/1941892008 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc012400 con 0x7fc2c415b880 2024-05-27T00:32:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2cb400640 1 -- 172.21.15.98:0/1941892008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c41f4060 con 0x7fc2c415b880 2024-05-27T00:32:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2cb400640 1 -- 172.21.15.98:0/1941892008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c41f4620 con 0x7fc2c415b880 2024-05-27T00:32:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.095+0000 7fc2cb400640 1 -- 172.21.15.98:0/1941892008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2c4075cb0 con 0x7fc2c415b880 2024-05-27T00:32:16.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.099+0000 7fc2c0c00640 1 -- 172.21.15.98:0/1941892008 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc2bc01a070 con 0x7fc2c415b880 2024-05-27T00:32:16.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.099+0000 7fc2c0c00640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2a40678c0 0x7fc2a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:16.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.099+0000 7fc2c0c00640 1 -- 172.21.15.98:0/1941892008 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fc2bc085d00 con 0x7fc2c415b880 2024-05-27T00:32:16.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.099+0000 7fc2c8a00640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2a40678c0 0x7fc2a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:16.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.099+0000 7fc2c8a00640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2a40678c0 0x7fc2a4069d80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80ad000 tx=0x7fc2b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:16.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.103+0000 7fc2c0c00640 1 -- 172.21.15.98:0/1941892008 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fc2c4067fe0 con 0x7fc2c415b880 2024-05-27T00:32:16.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.311+0000 7fc2cb400640 1 -- 172.21.15.98:0/1941892008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2c415bf40 con 0x7fc2c415b880 2024-05-27T00:32:16.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2c0c00640 1 -- 172.21.15.98:0/1941892008 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+6006 (secure 0 0 0) 0x7fc2bc055d60 con 0x7fc2c415b880 2024-05-27T00:32:16.318 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:16.318 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:10.313350+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:32:16.318 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 69 2024-05-27T00:32:16.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 -- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2a40678c0 msgr2=0x7fc2a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:16.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2a40678c0 0x7fc2a4069d80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80ad000 tx=0x7fc2b8008040 comp rx=0 tx=0).stop 2024-05-27T00:32:16.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 -- 172.21.15.98:0/1941892008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 msgr2=0x7fc2c415c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:16.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 0x7fc2c415c1e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00cab0 tx=0x7fc2bc00cf70 comp rx=0 tx=0).stop 2024-05-27T00:32:16.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 -- 172.21.15.98:0/1941892008 shutdown_connections 2024-05-27T00:32:16.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc2c415b880 0x7fc2c415c1e0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:16.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2c40c7960 0x7fc2c4150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:16.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fc2a40678c0 0x7fc2a4069d80 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 --2- 172.21.15.98:0/1941892008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2c406b470 0x7fc2c415d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 -- 172.21.15.98:0/1941892008 >> 172.21.15.98:0/1941892008 conn(0x7fc2c40cd060 msgr2=0x7fc2c40cbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 -- 172.21.15.98:0/1941892008 shutdown_connections 2024-05-27T00:32:16.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:16.315+0000 7fc2cb400640 1 -- 172.21.15.98:0/1941892008 wait complete. 2024-05-27T00:32:16.343 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:32:18.344 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:18.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.483+0000 7f02d1000640 1 Processor -- start 2024-05-27T00:32:18.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.483+0000 7f02d1000640 1 -- start start 2024-05-27T00:32:18.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02d1000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f02cc0b3260 0x7f02cc0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:18.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02d1000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f02cc070660 0x7f02cc0c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:18.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02d1000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02cc0c36c0 0x7f02cc0c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:18.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02d1000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f02cc071010 con 0x7f02cc070660 2024-05-27T00:32:18.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02d1000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f02cc0c88c0 con 0x7f02cc0b3260 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02d1000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f02cc0c8b80 con 0x7f02cc0c36c0 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02caa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f02cc070660 0x7f02cc0c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02caa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f02cc070660 0x7f02cc0c3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39262/0 (socket says 172.21.15.98:39262) 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02caa00640 1 -- 172.21.15.98:0/1741648918 learned_addr learned my addr 172.21.15.98:0/1741648918 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02cb400640 1 --2- 172.21.15.98:0/1741648918 >> v2:172.21.15.204:3300/0 conn(0x7f02cc0b3260 0x7f02cc0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02cbe00640 1 --2- 172.21.15.98:0/1741648918 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02cc0c36c0 0x7f02cc0c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02caa00640 1 -- 172.21.15.98:0/1741648918 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02cc0c36c0 msgr2=0x7f02cc0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02caa00640 1 --2- 172.21.15.98:0/1741648918 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02cc0c36c0 0x7f02cc0c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02caa00640 1 -- 172.21.15.98:0/1741648918 >> v2:172.21.15.204:3300/0 conn(0x7f02cc0b3260 msgr2=0x7f02cc0c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02caa00640 1 --2- 172.21.15.98:0/1741648918 >> v2:172.21.15.204:3300/0 conn(0x7f02cc0b3260 0x7f02cc0c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02caa00640 1 -- 172.21.15.98:0/1741648918 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc0c8e60 con 0x7f02cc070660 2024-05-27T00:32:18.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.487+0000 7f02cb400640 1 --2- 172.21.15.98:0/1741648918 >> v2:172.21.15.204:3300/0 conn(0x7f02cc0b3260 0x7f02cc0c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:18.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02caa00640 1 --2- 172.21.15.98:0/1741648918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc070660 0x7f02cc0c3180 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0cedc0 tx=0x7f02b801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb4b916c4f70712 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:18.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02ca000640 1 -- 172.21.15.98:0/1741648918 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b80b5070 con 0x7f02cc070660 2024-05-27T00:32:18.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02ca000640 1 -- 172.21.15.98:0/1741648918 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02b80a89d0 con 0x7f02cc070660 2024-05-27T00:32:18.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02ca000640 1 -- 172.21.15.98:0/1741648918 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b80a8da0 con 0x7f02cc070660 2024-05-27T00:32:18.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 -- 172.21.15.98:0/1741648918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc070660 msgr2=0x7f02cc0c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:18.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 --2- 172.21.15.98:0/1741648918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc070660 0x7f02cc0c3180 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0cedc0 tx=0x7f02b801ed10 comp rx=0 tx=0).stop 2024-05-27T00:32:18.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 -- 172.21.15.98:0/1741648918 shutdown_connections 2024-05-27T00:32:18.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 --2- 172.21.15.98:0/1741648918 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02cc0c36c0 0x7f02cc0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 --2- 172.21.15.98:0/1741648918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc070660 0x7f02cc0c3180 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 --2- 172.21.15.98:0/1741648918 >> v2:172.21.15.204:3300/0 conn(0x7f02cc0b3260 0x7f02cc0c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 -- 172.21.15.98:0/1741648918 >> 172.21.15.98:0/1741648918 conn(0x7f02cc0cdeb0 msgr2=0x7f02cc0ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:18.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 -- 172.21.15.98:0/1741648918 shutdown_connections 2024-05-27T00:32:18.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 -- 172.21.15.98:0/1741648918 wait complete. 2024-05-27T00:32:18.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 Processor -- start 2024-05-27T00:32:18.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.491+0000 7f02d1000640 1 -- start start 2024-05-27T00:32:18.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02d1000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02cc070660 0x7f02cc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02d1000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02cc0c36c0 0x7f02cc1e74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02d1000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc15b230 0x7f02cc159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02d1000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc0c8200 con 0x7f02cc15b230 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02d1000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc0c9420 con 0x7f02cc070660 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02d1000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f02cc0c9940 con 0x7f02cc0c36c0 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02cbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc15b230 0x7f02cc159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02cbe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc15b230 0x7f02cc159430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39274/0 (socket says 172.21.15.98:39274) 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02cbe00640 1 -- 172.21.15.98:0/2796859240 learned_addr learned my addr 172.21.15.98:0/2796859240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02cb400640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02cc070660 0x7f02cc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02caa00640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02cc0c36c0 0x7f02cc1e74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02cbe00640 1 -- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02cc0c36c0 msgr2=0x7f02cc1e74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:18.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02cbe00640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02cc0c36c0 0x7f02cc1e74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02cbe00640 1 -- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02cc070660 msgr2=0x7f02cc0af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02cbe00640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02cc070660 0x7f02cc0af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02cbe00640 1 -- 172.21.15.98:0/2796859240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc1f05c0 con 0x7f02cc15b230 2024-05-27T00:32:18.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02cbe00640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc15b230 0x7f02cc159430 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f02c4007e70 tx=0x7f02c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:18.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02c3400640 1 -- 172.21.15.98:0/2796859240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4014620 con 0x7f02cc15b230 2024-05-27T00:32:18.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02c3400640 1 -- 172.21.15.98:0/2796859240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c4019070 con 0x7f02cc15b230 2024-05-27T00:32:18.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02c3400640 1 -- 172.21.15.98:0/2796859240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4014a20 con 0x7f02cc15b230 2024-05-27T00:32:18.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02d1000640 1 -- 172.21.15.98:0/2796859240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02cc1f08a0 con 0x7f02cc15b230 2024-05-27T00:32:18.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02d1000640 1 -- 172.21.15.98:0/2796859240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02cc1f0e60 con 0x7f02cc15b230 2024-05-27T00:32:18.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.495+0000 7f02d1000640 1 -- 172.21.15.98:0/2796859240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02cc075c40 con 0x7f02cc15b230 2024-05-27T00:32:18.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.499+0000 7f02c3400640 1 -- 172.21.15.98:0/2796859240 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f02c4014bc0 con 0x7f02cc15b230 2024-05-27T00:32:18.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.499+0000 7f02c3400640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02ac0678c0 0x7f02ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:18.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.499+0000 7f02c3400640 1 -- 172.21.15.98:0/2796859240 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f02c4086970 con 0x7f02cc15b230 2024-05-27T00:32:18.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.499+0000 7f02cb400640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02ac0678c0 0x7f02ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:18.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.499+0000 7f02cb400640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02ac0678c0 0x7f02ac069d80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0098d0 tx=0x7f02bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:18.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.507+0000 7f02c3400640 1 -- 172.21.15.98:0/2796859240 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f02c4056940 con 0x7f02cc15b230 2024-05-27T00:32:18.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.751+0000 7f02d1000640 1 -- 172.21.15.98:0/2796859240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02cc0c5480 con 0x7f02cc15b230 2024-05-27T00:32:18.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.751+0000 7f02c3400640 1 -- 172.21.15.98:0/2796859240 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+6006 (secure 0 0 0) 0x7f02c405ac40 con 0x7f02cc15b230 2024-05-27T00:32:18.757 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:18.757 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:10.313350+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:replay","state_seq":1,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:32:18.758 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 69 2024-05-27T00:32:18.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 -- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02ac0678c0 msgr2=0x7f02ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:18.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02ac0678c0 0x7f02ac069d80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0098d0 tx=0x7f02bc008040 comp rx=0 tx=0).stop 2024-05-27T00:32:18.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 -- 172.21.15.98:0/2796859240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc15b230 msgr2=0x7f02cc159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:18.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc15b230 0x7f02cc159430 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f02c4007e70 tx=0x7f02c400e4b0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 -- 172.21.15.98:0/2796859240 shutdown_connections 2024-05-27T00:32:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02cc15b230 0x7f02cc159430 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02cc0c36c0 0x7f02cc1e74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f02ac0678c0 0x7f02ac069d80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 --2- 172.21.15.98:0/2796859240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02cc070660 0x7f02cc0af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 -- 172.21.15.98:0/2796859240 >> 172.21.15.98:0/2796859240 conn(0x7f02cc0cdeb0 msgr2=0x7f02cc0cd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 -- 172.21.15.98:0/2796859240 shutdown_connections 2024-05-27T00:32:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:18.755+0000 7f02d1000640 1 -- 172.21.15.98:0/2796859240 wait complete. 2024-05-27T00:32:18.781 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:32:20.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bbde00640 1 Processor -- start 2024-05-27T00:32:20.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bbde00640 1 -- start start 2024-05-27T00:32:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bbde00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7bb80731f0 0x7f7bb80c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:20.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bbde00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7bb80735d0 0x7f7bb8151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bbde00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80c79f0 0x7f7bb8151970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bbde00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7bb8071990 con 0x7f7bb80735d0 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bbde00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7bb80c17d0 con 0x7f7bb80731f0 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bbde00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb8070e10 con 0x7f7bb80c79f0 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bafe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7bb80735d0 0x7f7bb8151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bafe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7bb80735d0 0x7f7bb8151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39276/0 (socket says 172.21.15.98:39276) 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bafe00640 1 -- 172.21.15.98:0/4038144117 learned_addr learned my addr 172.21.15.98:0/4038144117 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bb6a00640 1 --2- 172.21.15.98:0/4038144117 >> v2:172.21.15.204:3300/0 conn(0x7f7bb80731f0 0x7f7bb80c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bb7400640 1 --2- 172.21.15.98:0/4038144117 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80c79f0 0x7f7bb8151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bafe00640 1 -- 172.21.15.98:0/4038144117 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80c79f0 msgr2=0x7f7bb8151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bafe00640 1 --2- 172.21.15.98:0/4038144117 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80c79f0 0x7f7bb8151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bafe00640 1 -- 172.21.15.98:0/4038144117 >> v2:172.21.15.204:3300/0 conn(0x7f7bb80731f0 msgr2=0x7f7bb80c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bafe00640 1 --2- 172.21.15.98:0/4038144117 >> v2:172.21.15.204:3300/0 conn(0x7f7bb80731f0 0x7f7bb80c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.919+0000 7f7bafe00640 1 -- 172.21.15.98:0/4038144117 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb80710f0 con 0x7f7bb80735d0 2024-05-27T00:32:20.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bafe00640 1 --2- 172.21.15.98:0/4038144117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80735d0 0x7f7bb8151430 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80cca90 tx=0x7f7ba001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97299095f706497 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bb6000640 1 -- 172.21.15.98:0/4038144117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba00b5070 con 0x7f7bb80735d0 2024-05-27T00:32:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bb6000640 1 -- 172.21.15.98:0/4038144117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ba00a89d0 con 0x7f7bb80735d0 2024-05-27T00:32:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bb6000640 1 -- 172.21.15.98:0/4038144117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba00a8da0 con 0x7f7bb80735d0 2024-05-27T00:32:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 -- 172.21.15.98:0/4038144117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80735d0 msgr2=0x7f7bb8151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 --2- 172.21.15.98:0/4038144117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80735d0 0x7f7bb8151430 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80cca90 tx=0x7f7ba001ed10 comp rx=0 tx=0).stop 2024-05-27T00:32:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 -- 172.21.15.98:0/4038144117 shutdown_connections 2024-05-27T00:32:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 --2- 172.21.15.98:0/4038144117 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80c79f0 0x7f7bb8151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 --2- 172.21.15.98:0/4038144117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80735d0 0x7f7bb8151430 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 --2- 172.21.15.98:0/4038144117 >> v2:172.21.15.204:3300/0 conn(0x7f7bb80731f0 0x7f7bb80c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 -- 172.21.15.98:0/4038144117 >> 172.21.15.98:0/4038144117 conn(0x7f7bb8067530 msgr2=0x7f7bb806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 -- 172.21.15.98:0/4038144117 shutdown_connections 2024-05-27T00:32:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 -- 172.21.15.98:0/4038144117 wait complete. 2024-05-27T00:32:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 Processor -- start 2024-05-27T00:32:20.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.923+0000 7f7bbde00640 1 -- start start 2024-05-27T00:32:20.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bbde00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80731f0 0x7f7bb815ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:20.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bbde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bb80735d0 0x7f7bb81f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:20.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bbde00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80c79f0 0x7f7bb8144cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:20.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bbde00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb80c8400 con 0x7f7bb80c79f0 2024-05-27T00:32:20.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bbde00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb80716b0 con 0x7f7bb80735d0 2024-05-27T00:32:20.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bbde00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb80c13f0 con 0x7f7bb80731f0 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80c79f0 0x7f7bb8144cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb7400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80c79f0 0x7f7bb8144cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39284/0 (socket says 172.21.15.98:39284) 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb7400640 1 -- 172.21.15.98:0/610109585 learned_addr learned my addr 172.21.15.98:0/610109585 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb6a00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80731f0 0x7f7bb815ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bafe00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bb80735d0 0x7f7bb81f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb7400640 1 -- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80731f0 msgr2=0x7f7bb815ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb7400640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80731f0 0x7f7bb815ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb7400640 1 -- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bb80735d0 msgr2=0x7f7bb81f0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb7400640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bb80735d0 0x7f7bb81f0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb7400640 1 -- 172.21.15.98:0/610109585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb8140610 con 0x7f7bb80c79f0 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb6a00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80731f0 0x7f7bb815ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bb7400640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80c79f0 0x7f7bb8144cd0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0007e90 tx=0x7f7bb000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:20.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7baf400640 1 -- 172.21.15.98:0/610109585 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0014420 con 0x7f7bb80c79f0 2024-05-27T00:32:20.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7baf400640 1 -- 172.21.15.98:0/610109585 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bb0019070 con 0x7f7bb80c79f0 2024-05-27T00:32:20.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bbde00640 1 -- 172.21.15.98:0/610109585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb81f40a0 con 0x7f7bb80c79f0 2024-05-27T00:32:20.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7baf400640 1 -- 172.21.15.98:0/610109585 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0014800 con 0x7f7bb80c79f0 2024-05-27T00:32:20.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bbde00640 1 -- 172.21.15.98:0/610109585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb81f4660 con 0x7f7bb80c79f0 2024-05-27T00:32:20.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.927+0000 7f7bbde00640 1 -- 172.21.15.98:0/610109585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb81f0760 con 0x7f7bb80c79f0 2024-05-27T00:32:20.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.931+0000 7f7baf400640 1 -- 172.21.15.98:0/610109585 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7bb00149a0 con 0x7f7bb80c79f0 2024-05-27T00:32:20.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.931+0000 7f7baf400640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b940678f0 0x7f7b94069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:20.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.931+0000 7f7baf400640 1 -- 172.21.15.98:0/610109585 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f7bb0086a80 con 0x7f7bb80c79f0 2024-05-27T00:32:20.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.931+0000 7f7bb6a00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b940678f0 0x7f7b94069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:20.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.931+0000 7f7bb6a00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b940678f0 0x7f7b94069db0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8004660 tx=0x7f7ba8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:20.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:20.935+0000 7f7baf400640 1 -- 172.21.15.98:0/610109585 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7bb0088e20 con 0x7f7bb80c79f0 2024-05-27T00:32:21.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 -- 172.21.15.98:0/610109585 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bb8075cb0 con 0x7f7bb80c79f0 2024-05-27T00:32:21.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7baf400640 1 -- 172.21.15.98:0/610109585 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+6010 (secure 0 0 0) 0x7f7bb0057010 con 0x7f7bb80c79f0 2024-05-27T00:32:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:21.138 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:20.111777+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:32:21.138 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 70 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 -- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b940678f0 msgr2=0x7f7b94069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b940678f0 0x7f7b94069db0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8004660 tx=0x7f7ba8008040 comp rx=0 tx=0).stop 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 -- 172.21.15.98:0/610109585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80c79f0 msgr2=0x7f7bb8144cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80c79f0 0x7f7bb8144cd0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0007e90 tx=0x7f7bb000e4b0 comp rx=0 tx=0).stop 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 -- 172.21.15.98:0/610109585 shutdown_connections 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bb80c79f0 0x7f7bb8144cd0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b940678f0 0x7f7b94069db0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bb80735d0 0x7f7bb81f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 --2- 172.21.15.98:0/610109585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bb80731f0 0x7f7bb815ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 -- 172.21.15.98:0/610109585 >> 172.21.15.98:0/610109585 conn(0x7f7bb8067530 msgr2=0x7f7bb80c86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 -- 172.21.15.98:0/610109585 shutdown_connections 2024-05-27T00:32:21.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:21.135+0000 7f7bbde00640 1 -- 172.21.15.98:0/610109585 wait complete. 2024-05-27T00:32:21.158 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:32:23.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f0903200640 1 Processor -- start 2024-05-27T00:32:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f0903200640 1 -- start start 2024-05-27T00:32:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f0903200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 0x7f08fc0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f0903200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08fc0c79f0 0x7f08fc157970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f0903200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08fc0c17d0 con 0x7f08fc0735d0 2024-05-27T00:32:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f0903200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f08fc071990 con 0x7f08fc0c79f0 2024-05-27T00:32:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f0903200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f08fc070e10 con 0x7f08fc0731f0 2024-05-27T00:32:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f0901200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08fc0c79f0 0x7f08fc157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f0901200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f08fc0c79f0 0x7f08fc157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55494/0 (socket says 172.21.15.98:55494) 2024-05-27T00:32:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f0901200640 1 -- 172.21.15.98:0/1872894646 learned_addr learned my addr 172.21.15.98:0/1872894646 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.303+0000 7f08fbe00640 1 --2- 172.21.15.98:0/1872894646 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 0x7f08fc0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0901200640 1 -- 172.21.15.98:0/1872894646 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 msgr2=0x7f08fc0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0901200640 1 --2- 172.21.15.98:0/1872894646 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 0x7f08fc0c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0901200640 1 -- 172.21.15.98:0/1872894646 >> v1:172.21.15.204:6789/0 conn(0x7f08fc0735d0 legacy=0x7f08fc151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0901200640 1 -- 172.21.15.98:0/1872894646 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08fc0710f0 con 0x7f08fc0c79f0 2024-05-27T00:32:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0901200640 1 --2- 172.21.15.98:0/1872894646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0c79f0 0x7f08fc157970 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009e50 tx=0x7f08f401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930747bfb9d45fdc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f08fb400640 1 -- 172.21.15.98:0/1872894646 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f40b4070 con 0x7f08fc0c79f0 2024-05-27T00:32:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f08fb400640 1 -- 172.21.15.98:0/1872894646 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08f40af400 con 0x7f08fc0c79f0 2024-05-27T00:32:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f08fb400640 1 -- 172.21.15.98:0/1872894646 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f40af7d0 con 0x7f08fc0c79f0 2024-05-27T00:32:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0903200640 1 -- 172.21.15.98:0/1872894646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0c79f0 msgr2=0x7f08fc157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0903200640 1 --2- 172.21.15.98:0/1872894646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0c79f0 0x7f08fc157970 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009e50 tx=0x7f08f401f210 comp rx=0 tx=0).stop 2024-05-27T00:32:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0903200640 1 -- 172.21.15.98:0/1872894646 shutdown_connections 2024-05-27T00:32:23.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0903200640 1 --2- 172.21.15.98:0/1872894646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0c79f0 0x7f08fc157970 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:23.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0903200640 1 --2- 172.21.15.98:0/1872894646 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 0x7f08fc0c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:23.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.307+0000 7f0903200640 1 -- 172.21.15.98:0/1872894646 >> 172.21.15.98:0/1872894646 conn(0x7f08fc067530 msgr2=0x7f08fc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:23.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 -- 172.21.15.98:0/1872894646 shutdown_connections 2024-05-27T00:32:23.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 -- 172.21.15.98:0/1872894646 wait complete. 2024-05-27T00:32:23.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 Processor -- start 2024-05-27T00:32:23.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 -- start start 2024-05-27T00:32:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 0x7f08fc1f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0735d0 0x7f08fc1e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08fc0c79f0 0x7f08fc142010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc0c8400 con 0x7f08fc0735d0 2024-05-27T00:32:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc0716b0 con 0x7f08fc0c79f0 2024-05-27T00:32:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f08fc0c13f0 con 0x7f08fc0731f0 2024-05-27T00:32:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0735d0 0x7f08fc1e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f3400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0735d0 0x7f08fc1e9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55496/0 (socket says 172.21.15.98:55496) 2024-05-27T00:32:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f3400640 1 -- 172.21.15.98:0/1872883792 learned_addr learned my addr 172.21.15.98:0/1872883792 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08fbe00640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 0x7f08fc1f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0901200640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08fc0c79f0 0x7f08fc142010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f3400640 1 -- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 msgr2=0x7f08fc1f1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f3400640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 0x7f08fc1f1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f3400640 1 -- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08fc0c79f0 msgr2=0x7f08fc142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f3400640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08fc0c79f0 0x7f08fc142010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f3400640 1 -- 172.21.15.98:0/1872883792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08fc1f2490 con 0x7f08fc0735d0 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08fbe00640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 0x7f08fc1f1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f3400640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0735d0 0x7f08fc1e9e40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f08e400c460 tx=0x7f08e400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f8c00640 1 -- 172.21.15.98:0/1872883792 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e40052d0 con 0x7f08fc0735d0 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f8c00640 1 -- 172.21.15.98:0/1872883792 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08e4005470 con 0x7f08fc0735d0 2024-05-27T00:32:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f08f8c00640 1 -- 172.21.15.98:0/1872883792 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4007260 con 0x7f08fc0735d0 2024-05-27T00:32:23.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 -- 172.21.15.98:0/1872883792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08fc0c1d00 con 0x7f08fc0735d0 2024-05-27T00:32:23.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.311+0000 7f0903200640 1 -- 172.21.15.98:0/1872883792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08fc0c3100 con 0x7f08fc0735d0 2024-05-27T00:32:23.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.315+0000 7f0903200640 1 -- 172.21.15.98:0/1872883792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08fc075cb0 con 0x7f08fc0735d0 2024-05-27T00:32:23.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.315+0000 7f08f8c00640 1 -- 172.21.15.98:0/1872883792 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f08e4007400 con 0x7f08fc0735d0 2024-05-27T00:32:23.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.315+0000 7f08f8c00640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f08d80678f0 0x7f08d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:23.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.315+0000 7f08f8c00640 1 -- 172.21.15.98:0/1872883792 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f08e4085620 con 0x7f08fc0735d0 2024-05-27T00:32:23.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.315+0000 7f08fbe00640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f08d80678f0 0x7f08d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:23.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.319+0000 7f08fbe00640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f08d80678f0 0x7f08d8069db0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009bb0 tx=0x7f08ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:23.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.319+0000 7f08f8c00640 1 -- 172.21.15.98:0/1872883792 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f08e4055c40 con 0x7f08fc0735d0 2024-05-27T00:32:23.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 -- 172.21.15.98:0/1872883792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08fc158510 con 0x7f08fc0735d0 2024-05-27T00:32:23.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f08f8c00640 1 -- 172.21.15.98:0/1872883792 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+6010 (secure 0 0 0) 0x7f08e4055540 con 0x7f08fc0735d0 2024-05-27T00:32:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:23.515 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:20.111777+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:32:23.515 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 70 2024-05-27T00:32:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 -- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f08d80678f0 msgr2=0x7f08d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f08d80678f0 0x7f08d8069db0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009bb0 tx=0x7f08ec008040 comp rx=0 tx=0).stop 2024-05-27T00:32:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 -- 172.21.15.98:0/1872883792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0735d0 msgr2=0x7f08fc1e9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0735d0 0x7f08fc1e9e40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f08e400c460 tx=0x7f08e400c920 comp rx=0 tx=0).stop 2024-05-27T00:32:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 -- 172.21.15.98:0/1872883792 shutdown_connections 2024-05-27T00:32:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f08d80678f0 0x7f08d8069db0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08fc0c79f0 0x7f08fc142010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08fc0735d0 0x7f08fc1e9e40 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 --2- 172.21.15.98:0/1872883792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08fc0731f0 0x7f08fc1f1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 -- 172.21.15.98:0/1872883792 >> 172.21.15.98:0/1872883792 conn(0x7f08fc067530 msgr2=0x7f08fc14eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:23.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 -- 172.21.15.98:0/1872883792 shutdown_connections 2024-05-27T00:32:23.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:23.511+0000 7f0903200640 1 -- 172.21.15.98:0/1872883792 wait complete. 2024-05-27T00:32:23.535 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 70, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T00:15:24.956429+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 70, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6396": { "addr": "172.21.15.98:6837/2042067068", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6836", "nonce": 2042067068, "type": "v2" }, { "addr": "172.21.15.98:6837", "nonce": 2042067068, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6396, "incarnation": 69, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T00:32:20.111777+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6396 } } } ], "standbys": [ { "addr": "172.21.15.204:6838/3117088639", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6835", "nonce": 3117088639, "type": "v2" }, { "addr": "172.21.15.204:6838", "nonce": 3117088639, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.98:6835/4095022947", "addrs": { "addrvec": [ { "addr": "172.21.15.98:6833", "nonce": 4095022947, "type": "v2" }, { "addr": "172.21.15.98:6835", "nonce": 4095022947, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6836/1815065053", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1815065053, "type": "v2" }, { "addr": "172.21.15.204:6836", "nonce": 1815065053, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4264, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.204:6839/2745908246", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6837", "nonce": 2745908246, "type": "v2" }, { "addr": "172.21.15.204:6839", "nonce": 2745908246, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-05-27T00:32:24.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:32:24.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:32:25.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:25.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.679+0000 7fe48e800640 1 Processor -- start 2024-05-27T00:32:25.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.679+0000 7fe48e800640 1 -- start start 2024-05-27T00:32:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48e800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4880c7e20 0x7fe48806f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48e800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4880c1150 con 0x7fe4881511b0 2024-05-27T00:32:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48e800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe48806edd0 con 0x7fe4880c8200 2024-05-27T00:32:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48e800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe488072360 con 0x7fe4880c7e20 2024-05-27T00:32:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48de00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe4881511b0 0x7fe488157510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48368/0 (socket says 172.21.15.98:48368) 2024-05-27T00:32:25.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48de00640 1 -- 172.21.15.98:0/2590515785 learned_addr learned my addr 172.21.15.98:0/2590515785 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:32:25.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48d400640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4880c7e20 0x7fe48806f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:25.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690562683 0 0) 0x7fe4880c1150 con 0x7fe4881511b0 2024-05-27T00:32:25.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe470003530 con 0x7fe4881511b0 2024-05-27T00:32:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 545914397 0 0) 0x7fe470003530 con 0x7fe4881511b0 2024-05-27T00:32:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4880c7e20 msgr2=0x7fe48806f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4880c7e20 0x7fe48806f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 >> v1:172.21.15.204:6789/0 conn(0x7fe4880c8200 legacy=0x7fe48806e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe488072640 con 0x7fe4881511b0 2024-05-27T00:32:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fe480001a00 con 0x7fe4881511b0 2024-05-27T00:32:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 >> v1:172.21.15.98:6789/0 conn(0x7fe4881511b0 legacy=0x7fe488157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4700039f0 0x7fe470003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:25.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4880c7e20 0x7fe470004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 0x7fe4700051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe488072360 con 0x7fe470004d50 2024-05-27T00:32:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe47000ba80 con 0x7fe4700039f0 2024-05-27T00:32:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe47000bc80 con 0x7fe4880c7e20 2024-05-27T00:32:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48de00640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 0x7fe4700051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48d400640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4700039f0 0x7fe470003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48ca00640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4880c7e20 0x7fe470004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48de00640 1 -- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4880c7e20 msgr2=0x7fe470004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:25.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48de00640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe4880c7e20 0x7fe470004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:25.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48de00640 1 -- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4700039f0 msgr2=0x7fe470003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:25.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48de00640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4700039f0 0x7fe470003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:25.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48de00640 1 -- 172.21.15.98:0/2590515785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe470003530 con 0x7fe470004d50 2024-05-27T00:32:25.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.683+0000 7fe48d400640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4700039f0 0x7fe470003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:25.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe48de00640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 0x7fe4700051d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe48000a810 tx=0x7fe48001d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9463b7bbecbda7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:25.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48000a460 con 0x7fe470004d50 2024-05-27T00:32:25.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4800ad650 con 0x7fe470004d50 2024-05-27T00:32:25.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe487e00640 1 -- 172.21.15.98:0/2590515785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4800ada40 con 0x7fe470004d50 2024-05-27T00:32:25.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe48e800640 1 -- 172.21.15.98:0/2590515785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 msgr2=0x7fe4700051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:25.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe48e800640 1 --2- 172.21.15.98:0/2590515785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 0x7fe4700051d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe48000a810 tx=0x7fe48001d340 comp rx=0 tx=0).stop 2024-05-27T00:32:25.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe48de00640 1 -- 172.21.15.98:0/2590515785 reap_dead start 2024-05-27T00:32:25.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe48e800640 1 -- 172.21.15.98:0/2590515785 shutdown_connections 2024-05-27T00:32:25.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe48e800640 1 -- 172.21.15.98:0/2590515785 >> 172.21.15.98:0/2590515785 conn(0x7fe4880cbe00 msgr2=0x7fe4880cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:25.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe48e800640 1 -- 172.21.15.98:0/2590515785 shutdown_connections 2024-05-27T00:32:25.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe48e800640 1 -- 172.21.15.98:0/2590515785 wait complete. 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe48e800640 1 Processor -- start 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.687+0000 7fe48e800640 1 -- start start 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48e800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 0x7fe4881f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48e800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe488151cf0 0x7fe488150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe488154160 0x7fe4881545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48e800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe488157c20 con 0x7fe470004d50 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48e800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe488072c00 con 0x7fe488154160 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48e800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe488157da0 con 0x7fe488151cf0 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 0x7fe4881f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 0x7fe4881f3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55502/0 (socket says 172.21.15.98:55502) 2024-05-27T00:32:25.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48d400640 1 -- 172.21.15.98:0/3956980198 learned_addr learned my addr 172.21.15.98:0/3956980198 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48ca00640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe488151cf0 0x7fe488150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48de00640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe488154160 0x7fe4881545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48d400640 1 -- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe488151cf0 msgr2=0x7fe488150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48d400640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe488151cf0 0x7fe488150310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48d400640 1 -- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe488154160 msgr2=0x7fe4881545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48d400640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe488154160 0x7fe4881545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48d400640 1 -- 172.21.15.98:0/3956980198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe488154b20 con 0x7fe470004d50 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48d400640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 0x7fe4881f3890 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe47c011af0 tx=0x7fe47c009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe485600640 1 -- 172.21.15.98:0/3956980198 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c004020 con 0x7fe470004d50 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe485600640 1 -- 172.21.15.98:0/3956980198 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe47c0041c0 con 0x7fe470004d50 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48e800640 1 -- 172.21.15.98:0/3956980198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe488145a10 con 0x7fe470004d50 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe485600640 1 -- 172.21.15.98:0/3956980198 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c003620 con 0x7fe470004d50 2024-05-27T00:32:25.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48e800640 1 -- 172.21.15.98:0/3956980198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4881f4220 con 0x7fe470004d50 2024-05-27T00:32:25.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe48e800640 1 -- 172.21.15.98:0/3956980198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe488075c40 con 0x7fe470004d50 2024-05-27T00:32:25.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.691+0000 7fe485600640 1 -- 172.21.15.98:0/3956980198 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe47c0037c0 con 0x7fe470004d50 2024-05-27T00:32:25.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.695+0000 7fe485600640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4680678c0 0x7fe468069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:25.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.695+0000 7fe485600640 1 -- 172.21.15.98:0/3956980198 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fe47c086690 con 0x7fe470004d50 2024-05-27T00:32:25.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.695+0000 7fe48ca00640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4680678c0 0x7fe468069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:25.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.695+0000 7fe48ca00640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4680678c0 0x7fe468069d80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe488150da0 tx=0x7fe474009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:25.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.699+0000 7fe485600640 1 -- 172.21.15.98:0/3956980198 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fe488154b20 con 0x7fe470004d50 2024-05-27T00:32:25.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 -- 172.21.15.98:0/3956980198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe48806c630 con 0x7fe470004d50 2024-05-27T00:32:25.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe485600640 1 -- 172.21.15.98:0/3956980198 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+6010 (secure 0 0 0) 0x7fe47c00e070 con 0x7fe470004d50 2024-05-27T00:32:25.895 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:25.895 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:20.111777+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:32:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 70 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 -- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4680678c0 msgr2=0x7fe468069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4680678c0 0x7fe468069d80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe488150da0 tx=0x7fe474009040 comp rx=0 tx=0).stop 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 -- 172.21.15.98:0/3956980198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 msgr2=0x7fe4881f3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 0x7fe4881f3890 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe47c011af0 tx=0x7fe47c009f30 comp rx=0 tx=0).stop 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 -- 172.21.15.98:0/3956980198 shutdown_connections 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4680678c0 0x7fe468069d80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe488154160 0x7fe4881545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe488151cf0 0x7fe488150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 --2- 172.21.15.98:0/3956980198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe470004d50 0x7fe4881f3890 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 -- 172.21.15.98:0/3956980198 >> 172.21.15.98:0/3956980198 conn(0x7fe4880cbe00 msgr2=0x7fe4880c5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:25.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 -- 172.21.15.98:0/3956980198 shutdown_connections 2024-05-27T00:32:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:25.891+0000 7fe48e800640 1 -- 172.21.15.98:0/3956980198 wait complete. 2024-05-27T00:32:25.915 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:32:27.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:28.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f484aa00640 1 Processor -- start 2024-05-27T00:32:28.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f484aa00640 1 -- start start 2024-05-27T00:32:28.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f484aa00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f48440cbe10 0x7f48441592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:28.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f484aa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f48440cc1f0 0x7f48440c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f484aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4844073570 0x7f48441559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f484aa00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4844159810 con 0x7f48440cc1f0 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f484aa00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f48440c6dd0 con 0x7f48440cbe10 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f484aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f48440c70b0 con 0x7f4844073570 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f4843400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f48440cc1f0 0x7f48440c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f4843400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f48440cc1f0 0x7f48440c1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55518/0 (socket says 172.21.15.98:55518) 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f4843400640 1 -- 172.21.15.98:0/2539678444 learned_addr learned my addr 172.21.15.98:0/2539678444 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f4848a00640 1 --2- 172.21.15.98:0/2539678444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4844073570 0x7f48441559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f4843e00640 1 --2- 172.21.15.98:0/2539678444 >> v2:172.21.15.204:3300/0 conn(0x7f48440cbe10 0x7f48441592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f4843400640 1 -- 172.21.15.98:0/2539678444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4844073570 msgr2=0x7f48441559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f4843400640 1 --2- 172.21.15.98:0/2539678444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4844073570 0x7f48441559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f4843400640 1 -- 172.21.15.98:0/2539678444 >> v2:172.21.15.204:3300/0 conn(0x7f48440cbe10 msgr2=0x7f48441592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:28.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f4843400640 1 --2- 172.21.15.98:0/2539678444 >> v2:172.21.15.204:3300/0 conn(0x7f48440cbe10 0x7f48441592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.059+0000 7f4843400640 1 -- 172.21.15.98:0/2539678444 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48440c7390 con 0x7f48440cc1f0 2024-05-27T00:32:28.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f4843400640 1 --2- 172.21.15.98:0/2539678444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cc1f0 0x7f48440c1850 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f48440cc760 tx=0x7f483001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd58ca29ab49006 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f4842a00640 1 -- 172.21.15.98:0/2539678444 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48300b5070 con 0x7f48440cc1f0 2024-05-27T00:32:28.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f4842a00640 1 -- 172.21.15.98:0/2539678444 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48300a89d0 con 0x7f48440cc1f0 2024-05-27T00:32:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f4842a00640 1 -- 172.21.15.98:0/2539678444 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48300a8da0 con 0x7f48440cc1f0 2024-05-27T00:32:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f484aa00640 1 -- 172.21.15.98:0/2539678444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cc1f0 msgr2=0x7f48440c1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f484aa00640 1 --2- 172.21.15.98:0/2539678444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cc1f0 0x7f48440c1850 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f48440cc760 tx=0x7f483001ed10 comp rx=0 tx=0).stop 2024-05-27T00:32:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f484aa00640 1 -- 172.21.15.98:0/2539678444 shutdown_connections 2024-05-27T00:32:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f484aa00640 1 --2- 172.21.15.98:0/2539678444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4844073570 0x7f48441559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f484aa00640 1 --2- 172.21.15.98:0/2539678444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cc1f0 0x7f48440c1850 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f484aa00640 1 --2- 172.21.15.98:0/2539678444 >> v2:172.21.15.204:3300/0 conn(0x7f48440cbe10 0x7f48441592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f484aa00640 1 -- 172.21.15.98:0/2539678444 >> 172.21.15.98:0/2539678444 conn(0x7f4844067530 msgr2=0x7f484406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f484aa00640 1 -- 172.21.15.98:0/2539678444 shutdown_connections 2024-05-27T00:32:28.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.063+0000 7f484aa00640 1 -- 172.21.15.98:0/2539678444 wait complete. 2024-05-27T00:32:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f484aa00640 1 Processor -- start 2024-05-27T00:32:28.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f484aa00640 1 -- start start 2024-05-27T00:32:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f484aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4844073570 0x7f4844157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f484aa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cbe10 0x7f48441579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f484aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48440cc1f0 0x7f484414dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f484aa00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48440c79d0 con 0x7f48440cbe10 2024-05-27T00:32:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f484aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f48440c1330 con 0x7f48440cc1f0 2024-05-27T00:32:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f484aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f48440c6990 con 0x7f4844073570 2024-05-27T00:32:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4843400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cbe10 0x7f48441579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4843400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cbe10 0x7f48441579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55526/0 (socket says 172.21.15.98:55526) 2024-05-27T00:32:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4843400640 1 -- 172.21.15.98:0/2852960146 learned_addr learned my addr 172.21.15.98:0/2852960146 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:28.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4843e00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4844073570 0x7f4844157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4848a00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48440cc1f0 0x7f484414dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4843400640 1 -- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4844073570 msgr2=0x7f4844157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4843400640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4844073570 0x7f4844157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4843400640 1 -- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48440cc1f0 msgr2=0x7f484414dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4843400640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48440cc1f0 0x7f484414dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4843400640 1 -- 172.21.15.98:0/2852960146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4844159a50 con 0x7f48440cbe10 2024-05-27T00:32:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4848a00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48440cc1f0 0x7f484414dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:28.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f4843400640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cbe10 0x7f48441579c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f48300b3040 tx=0x7f483001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:28.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f483be00640 1 -- 172.21.15.98:0/2852960146 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48300c0070 con 0x7f48440cbe10 2024-05-27T00:32:28.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f483be00640 1 -- 172.21.15.98:0/2852960146 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48300b0900 con 0x7f48440cbe10 2024-05-27T00:32:28.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f483be00640 1 -- 172.21.15.98:0/2852960146 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48300b5040 con 0x7f48440cbe10 2024-05-27T00:32:28.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f484aa00640 1 -- 172.21.15.98:0/2852960146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48441566a0 con 0x7f48440cbe10 2024-05-27T00:32:28.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.067+0000 7f484aa00640 1 -- 172.21.15.98:0/2852960146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4844159cf0 con 0x7f48440cbe10 2024-05-27T00:32:28.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.071+0000 7f484aa00640 1 -- 172.21.15.98:0/2852960146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4844075cb0 con 0x7f48440cbe10 2024-05-27T00:32:28.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.071+0000 7f483be00640 1 -- 172.21.15.98:0/2852960146 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f48300c3070 con 0x7f48440cbe10 2024-05-27T00:32:28.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.071+0000 7f483be00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 0x7f4824069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:28.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.071+0000 7f483be00640 1 -- 172.21.15.98:0/2852960146 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f4830122f40 con 0x7f48440cbe10 2024-05-27T00:32:28.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.071+0000 7f4843e00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 0x7f4824069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:28.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.071+0000 7f4843e00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 0x7f4824069d80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4834007a60 tx=0x7f4834008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:28.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.075+0000 7f483be00640 1 -- 172.21.15.98:0/2852960146 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4830018870 con 0x7f48440cbe10 2024-05-27T00:32:28.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 -- 172.21.15.98:0/2852960146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48440ca990 con 0x7f48440cbe10 2024-05-27T00:32:28.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f483be00640 1 -- 172.21.15.98:0/2852960146 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6007 (secure 0 0 0) 0x7f48300f2fa0 con 0x7f48440cbe10 2024-05-27T00:32:28.270 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:28.270 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:25.878548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:32:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 71 2024-05-27T00:32:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 -- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 msgr2=0x7f4824069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 0x7f4824069d80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4834007a60 tx=0x7f4834008040 comp rx=0 tx=0).stop 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 -- 172.21.15.98:0/2852960146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cbe10 msgr2=0x7f48441579c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cbe10 0x7f48441579c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f48300b3040 tx=0x7f483001bab0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 -- 172.21.15.98:0/2852960146 shutdown_connections 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f48240678c0 0x7f4824069d80 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48440cc1f0 0x7f484414dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48440cbe10 0x7f48441579c0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 --2- 172.21.15.98:0/2852960146 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4844073570 0x7f4844157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 -- 172.21.15.98:0/2852960146 >> 172.21.15.98:0/2852960146 conn(0x7f4844067530 msgr2=0x7f484406bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 -- 172.21.15.98:0/2852960146 shutdown_connections 2024-05-27T00:32:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:28.267+0000 7f484aa00640 1 -- 172.21.15.98:0/2852960146 wait complete. 2024-05-27T00:32:28.290 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:32:30.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:30.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3e00640 1 Processor -- start 2024-05-27T00:32:30.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3e00640 1 -- start start 2024-05-27T00:32:30.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ed4067390 0x7f4ed4072ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:30.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3e00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f4ed4067770 0x7f4ed4073410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:30.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4ed406edb0 0x7f4ed4152230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:30.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed4072a40 con 0x7f4ed406edb0 2024-05-27T00:32:30.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3e00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed41585c0 con 0x7f4ed4067770 2024-05-27T00:32:30.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed40c78a0 con 0x7f4ed4067390 2024-05-27T00:32:30.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4ed406edb0 0x7f4ed4152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:30.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4ed406edb0 0x7f4ed4152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55538/0 (socket says 172.21.15.98:55538) 2024-05-27T00:32:30.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed3400640 1 -- 172.21.15.98:0/653087989 learned_addr learned my addr 172.21.15.98:0/653087989 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:30.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed2a00640 1 --2- 172.21.15.98:0/653087989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ed4067390 0x7f4ed4072ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:30.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.435+0000 7f4ed2000640 1 --2- 172.21.15.98:0/653087989 >> v2:172.21.15.204:3300/0 conn(0x7f4ed4067770 0x7f4ed4073410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:30.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3400640 1 -- 172.21.15.98:0/653087989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ed4067390 msgr2=0x7f4ed4072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:30.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3400640 1 --2- 172.21.15.98:0/653087989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ed4067390 0x7f4ed4072ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3400640 1 -- 172.21.15.98:0/653087989 >> v2:172.21.15.204:3300/0 conn(0x7f4ed4067770 msgr2=0x7f4ed4073410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:30.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3400640 1 --2- 172.21.15.98:0/653087989 >> v2:172.21.15.204:3300/0 conn(0x7f4ed4067770 0x7f4ed4073410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3400640 1 -- 172.21.15.98:0/653087989 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed4158880 con 0x7f4ed406edb0 2024-05-27T00:32:30.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed2000640 1 --2- 172.21.15.98:0/653087989 >> v2:172.21.15.204:3300/0 conn(0x7f4ed4067770 0x7f4ed4073410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3400640 1 --2- 172.21.15.98:0/653087989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed406edb0 0x7f4ed4152230 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40c4150 tx=0x7f4ecc01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187e8b21d5c6f89b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed1600640 1 -- 172.21.15.98:0/653087989 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc0b5070 con 0x7f4ed406edb0 2024-05-27T00:32:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed1600640 1 -- 172.21.15.98:0/653087989 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ecc0a89d0 con 0x7f4ed406edb0 2024-05-27T00:32:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed1600640 1 -- 172.21.15.98:0/653087989 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc0a8da0 con 0x7f4ed406edb0 2024-05-27T00:32:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3e00640 1 -- 172.21.15.98:0/653087989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed406edb0 msgr2=0x7f4ed4152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/653087989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed406edb0 0x7f4ed4152230 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40c4150 tx=0x7f4ecc01ed10 comp rx=0 tx=0).stop 2024-05-27T00:32:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3e00640 1 -- 172.21.15.98:0/653087989 shutdown_connections 2024-05-27T00:32:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/653087989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed406edb0 0x7f4ed4152230 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/653087989 >> v2:172.21.15.204:3300/0 conn(0x7f4ed4067770 0x7f4ed4073410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/653087989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ed4067390 0x7f4ed4072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3e00640 1 -- 172.21.15.98:0/653087989 >> 172.21.15.98:0/653087989 conn(0x7f4ed406f780 msgr2=0x7f4ed406fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:30.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3e00640 1 -- 172.21.15.98:0/653087989 shutdown_connections 2024-05-27T00:32:30.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.439+0000 7f4ed3e00640 1 -- 172.21.15.98:0/653087989 wait complete. 2024-05-27T00:32:30.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3e00640 1 Processor -- start 2024-05-27T00:32:30.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3e00640 1 -- start start 2024-05-27T00:32:30.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ed4067390 0x7f4ed40c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:30.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4067770 0x7f4ed41f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:30.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ed406edb0 0x7f4ed414d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:30.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed4159120 con 0x7f4ed4067770 2024-05-27T00:32:30.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed4158e40 con 0x7f4ed4067390 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed4159400 con 0x7f4ed406edb0 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed2000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4067770 0x7f4ed41f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed2000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4067770 0x7f4ed41f3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55546/0 (socket says 172.21.15.98:55546) 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed2000640 1 -- 172.21.15.98:0/1276563058 learned_addr learned my addr 172.21.15.98:0/1276563058 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed2a00640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ed4067390 0x7f4ed40c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3400640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ed406edb0 0x7f4ed414d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed2000640 1 -- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ed406edb0 msgr2=0x7f4ed414d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed2000640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ed406edb0 0x7f4ed414d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed2000640 1 -- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ed4067390 msgr2=0x7f4ed40c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed2000640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ed4067390 0x7f4ed40c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed2000640 1 -- 172.21.15.98:0/1276563058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed41513c0 con 0x7f4ed4067770 2024-05-27T00:32:30.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed2000640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4067770 0x7f4ed41f3900 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000e7e0 tx=0x7f4ec000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ecaa00640 1 -- 172.21.15.98:0/1276563058 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0017070 con 0x7f4ed4067770 2024-05-27T00:32:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ecaa00640 1 -- 172.21.15.98:0/1276563058 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec000c6d0 con 0x7f4ed4067770 2024-05-27T00:32:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3e00640 1 -- 172.21.15.98:0/1276563058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed4151a10 con 0x7f4ed4067770 2024-05-27T00:32:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ecaa00640 1 -- 172.21.15.98:0/1276563058 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec000f040 con 0x7f4ed4067770 2024-05-27T00:32:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.443+0000 7f4ed3e00640 1 -- 172.21.15.98:0/1276563058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed4151cb0 con 0x7f4ed4067770 2024-05-27T00:32:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.447+0000 7f4ed3e00640 1 -- 172.21.15.98:0/1276563058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed4151720 con 0x7f4ed4067770 2024-05-27T00:32:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.447+0000 7f4ecaa00640 1 -- 172.21.15.98:0/1276563058 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4ec001a070 con 0x7f4ed4067770 2024-05-27T00:32:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.447+0000 7f4ecaa00640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eb40678f0 0x7f4eb4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.447+0000 7f4ecaa00640 1 -- 172.21.15.98:0/1276563058 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f4ec0085e00 con 0x7f4ed4067770 2024-05-27T00:32:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.447+0000 7f4ed2a00640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eb40678f0 0x7f4eb4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.447+0000 7f4ed2a00640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eb40678f0 0x7f4eb4069db0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40074a0 tx=0x7f4ec400d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.451+0000 7f4ecaa00640 1 -- 172.21.15.98:0/1276563058 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4ec0056420 con 0x7f4ed4067770 2024-05-27T00:32:30.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.639+0000 7f4ed3e00640 1 -- 172.21.15.98:0/1276563058 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ed4157790 con 0x7f4ed4067770 2024-05-27T00:32:30.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.639+0000 7f4ecaa00640 1 -- 172.21.15.98:0/1276563058 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6007 (secure 0 0 0) 0x7f4ec0055d20 con 0x7f4ed4067770 2024-05-27T00:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:25.878548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T00:32:30.646 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 71 2024-05-27T00:32:30.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 -- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eb40678f0 msgr2=0x7f4eb4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:30.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eb40678f0 0x7f4eb4069db0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40074a0 tx=0x7f4ec400d040 comp rx=0 tx=0).stop 2024-05-27T00:32:30.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 -- 172.21.15.98:0/1276563058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4067770 msgr2=0x7f4ed41f3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:30.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4067770 0x7f4ed41f3900 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000e7e0 tx=0x7f4ec000eca0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 -- 172.21.15.98:0/1276563058 shutdown_connections 2024-05-27T00:32:30.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ed406edb0 0x7f4ed414d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ed4067770 0x7f4ed41f3900 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4eb40678f0 0x7f4eb4069db0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 --2- 172.21.15.98:0/1276563058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ed4067390 0x7f4ed40c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 -- 172.21.15.98:0/1276563058 >> 172.21.15.98:0/1276563058 conn(0x7f4ed406f780 msgr2=0x7f4ed406b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 -- 172.21.15.98:0/1276563058 shutdown_connections 2024-05-27T00:32:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:30.643+0000 7f4ed3e00640 1 -- 172.21.15.98:0/1276563058 wait complete. 2024-05-27T00:32:30.666 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T00:32:32.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce72200640 1 Processor -- start 2024-05-27T00:32:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce72200640 1 -- start start 2024-05-27T00:32:32.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce72200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0cbe10 0x7fce6c1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:32.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce72200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce6c0c1f60 con 0x7fce6c073570 2024-05-27T00:32:32.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce72200640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce6c0c6dd0 con 0x7fce6c0cc1f0 2024-05-27T00:32:32.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce72200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fce6c159810 con 0x7fce6c0cbe10 2024-05-27T00:32:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce6be00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fce6c073570 0x7fce6c1559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48382/0 (socket says 172.21.15.98:48382) 2024-05-27T00:32:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce6be00640 1 -- 172.21.15.98:0/2739998369 learned_addr learned my addr 172.21.15.98:0/2739998369 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:32:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce6b400640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0cbe10 0x7fce6c1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3404739833 0 0) 0x7fce6c0c1f60 con 0x7fce6c073570 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce50003530 con 0x7fce6c073570 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119174542 0 0) 0x7fce6c0c6dd0 con 0x7fce6c0cc1f0 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.799+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce6c0c1f60 con 0x7fce6c0cc1f0 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 271960564 0 0) 0x7fce50003530 con 0x7fce6c073570 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0cbe10 msgr2=0x7fce6c1592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0cbe10 0x7fce6c1592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 >> v1:172.21.15.204:6789/0 conn(0x7fce6c0cc1f0 legacy=0x7fce6c0c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6c0c70b0 con 0x7fce6c073570 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fce64001680 con 0x7fce6c073570 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6b400640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0cbe10 0x7fce6c1592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 >> v1:172.21.15.98:6789/0 conn(0x7fce6c073570 legacy=0x7fce6c1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce50003b80 0x7fce50003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0cbe10 0x7fce50004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce50004d50 0x7fce500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c159810 con 0x7fce50004d50 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fce5000ba80 con 0x7fce50003b80 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fce5000bc00 con 0x7fce6c0cbe10 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6be00640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce50004d50 0x7fce500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6aa00640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0cbe10 0x7fce50004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6b400640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce50003b80 0x7fce50003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6be00640 1 -- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0cbe10 msgr2=0x7fce50004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6be00640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0cbe10 0x7fce50004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6be00640 1 -- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce50003b80 msgr2=0x7fce50003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6be00640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce50003b80 0x7fce50003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6be00640 1 -- 172.21.15.98:0/2739998369 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce50003530 con 0x7fce50004d50 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6be00640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce50004d50 0x7fce500051d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c1c0 tx=0x7fce6401cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3ad0dc61e1e5e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64007cd0 con 0x7fce50004d50 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce640a9020 con 0x7fce50004d50 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.803+0000 7fce6a000640 1 -- 172.21.15.98:0/2739998369 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64005ae0 con 0x7fce50004d50 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 -- 172.21.15.98:0/2739998369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce50004d50 msgr2=0x7fce500051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 --2- 172.21.15.98:0/2739998369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce50004d50 0x7fce500051d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c1c0 tx=0x7fce6401cbe0 comp rx=0 tx=0).stop 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6be00640 1 -- 172.21.15.98:0/2739998369 reap_dead start 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 -- 172.21.15.98:0/2739998369 shutdown_connections 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 -- 172.21.15.98:0/2739998369 >> 172.21.15.98:0/2739998369 conn(0x7fce6c067530 msgr2=0x7fce6c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 -- 172.21.15.98:0/2739998369 shutdown_connections 2024-05-27T00:32:32.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 -- 172.21.15.98:0/2739998369 wait complete. 2024-05-27T00:32:32.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 Processor -- start 2024-05-27T00:32:32.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 -- start start 2024-05-27T00:32:32.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce6c0c5470 0x7fce6c06bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:32.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0ae8c0 0x7fce6c1f04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:32.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce6c0c5850 0x7fce6c1f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c0c1330 con 0x7fce6c0c5850 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fce6c0c1050 con 0x7fce6c0c5470 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce72200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fce6c0c79d0 con 0x7fce6c0ae8c0 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce6c0c5850 0x7fce6c1f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce6c0c5850 0x7fce6c1f3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55558/0 (socket says 172.21.15.98:55558) 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6be00640 1 -- 172.21.15.98:0/1859843944 learned_addr learned my addr 172.21.15.98:0/1859843944 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6aa00640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0ae8c0 0x7fce6c1f04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6b400640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce6c0c5470 0x7fce6c06bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6be00640 1 -- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0ae8c0 msgr2=0x7fce6c1f04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6be00640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0ae8c0 0x7fce6c1f04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6be00640 1 -- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce6c0c5470 msgr2=0x7fce6c06bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6be00640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce6c0c5470 0x7fce6c06bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:32.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6be00640 1 -- 172.21.15.98:0/1859843944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6c0aed90 con 0x7fce6c0c5850 2024-05-27T00:32:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.807+0000 7fce6be00640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce6c0c5850 0x7fce6c1f3a80 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c1c0 tx=0x7fce50005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.811+0000 7fce63400640 1 -- 172.21.15.98:0/1859843944 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64007cd0 con 0x7fce6c0c5850 2024-05-27T00:32:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.811+0000 7fce63400640 1 -- 172.21.15.98:0/1859843944 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce640b2030 con 0x7fce6c0c5850 2024-05-27T00:32:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.811+0000 7fce63400640 1 -- 172.21.15.98:0/1859843944 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64001c10 con 0x7fce6c0c5850 2024-05-27T00:32:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.811+0000 7fce72200640 1 -- 172.21.15.98:0/1859843944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce6c0b07f0 con 0x7fce6c0c5850 2024-05-27T00:32:32.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.811+0000 7fce72200640 1 -- 172.21.15.98:0/1859843944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce6c0b0db0 con 0x7fce6c0c5850 2024-05-27T00:32:32.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.811+0000 7fce72200640 1 -- 172.21.15.98:0/1859843944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce6c075cb0 con 0x7fce6c0c5850 2024-05-27T00:32:32.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.811+0000 7fce63400640 1 -- 172.21.15.98:0/1859843944 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fce64004cb0 con 0x7fce6c0c5850 2024-05-27T00:32:32.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.811+0000 7fce63400640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 0x7fce4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:32.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.811+0000 7fce63400640 1 -- 172.21.15.98:0/1859843944 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fce6411f2e0 con 0x7fce6c0c5850 2024-05-27T00:32:32.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.811+0000 7fce6b400640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 0x7fce4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:32.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.815+0000 7fce6b400640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 0x7fce4c069d80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009bd0 tx=0x7fce5c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:32.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:32.815+0000 7fce63400640 1 -- 172.21.15.98:0/1859843944 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fce640168f0 con 0x7fce6c0c5850 2024-05-27T00:32:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.007+0000 7fce72200640 1 -- 172.21.15.98:0/1859843944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce6c06ed80 con 0x7fce6c0c5850 2024-05-27T00:32:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce63400640 1 -- 172.21.15.98:0/1859843944 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6014 (secure 0 0 0) 0x7fce640efa30 con 0x7fce6c0c5850 2024-05-27T00:32:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:33.014 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:32.656788+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6396,"qdb_cluster":[6396]},"id":1}]} 2024-05-27T00:32:33.014 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 72 2024-05-27T00:32:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 -- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 msgr2=0x7fce4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 0x7fce4c069d80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009bd0 tx=0x7fce5c00b040 comp rx=0 tx=0).stop 2024-05-27T00:32:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 -- 172.21.15.98:0/1859843944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce6c0c5850 msgr2=0x7fce6c1f3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:33.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce6c0c5850 0x7fce6c1f3a80 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c1c0 tx=0x7fce50005840 comp rx=0 tx=0).stop 2024-05-27T00:32:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 -- 172.21.15.98:0/1859843944 shutdown_connections 2024-05-27T00:32:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce6c0c5850 0x7fce6c1f3a80 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce6c0ae8c0 0x7fce6c1f04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fce4c0678c0 0x7fce4c069d80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 --2- 172.21.15.98:0/1859843944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce6c0c5470 0x7fce6c06bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 -- 172.21.15.98:0/1859843944 >> 172.21.15.98:0/1859843944 conn(0x7fce6c067530 msgr2=0x7fce6c153430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 -- 172.21.15.98:0/1859843944 shutdown_connections 2024-05-27T00:32:33.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:33.011+0000 7fce72200640 1 -- 172.21.15.98:0/1859843944 wait complete. 2024-05-27T00:32:33.033 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=6240 2024-05-27T00:32:33.033 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.a 2024-05-27T00:32:38.034 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-05-27T00:32:38.034 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-27T00:32:38.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-27T00:32:38.039 INFO:tasks.ceph.mds.a:Started 2024-05-27T00:32:38.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:38.188 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2024-05-27T00:32:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e2de00640 1 Processor -- start 2024-05-27T00:32:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e2de00640 1 -- start start 2024-05-27T00:32:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e2de00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7e280731f0 0x7f7e280c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e2de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280735d0 0x7f7e28151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e2de00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e280c17d0 con 0x7f7e280c79f0 2024-05-27T00:32:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e2de00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7e28071990 con 0x7f7e280731f0 2024-05-27T00:32:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e2de00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7e28070e10 con 0x7f7e280735d0 2024-05-27T00:32:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e26a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7e280731f0 0x7f7e280c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e26a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7e280731f0 0x7f7e280c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52720/0 (socket says 172.21.15.98:52720) 2024-05-27T00:32:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e26a00640 1 -- 172.21.15.98:0/1303496281 learned_addr learned my addr 172.21.15.98:0/1303496281 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e26000640 1 --2- 172.21.15.98:0/1303496281 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280735d0 0x7f7e28151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e26a00640 1 -- 172.21.15.98:0/1303496281 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280735d0 msgr2=0x7f7e28151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e26a00640 1 --2- 172.21.15.98:0/1303496281 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280735d0 0x7f7e28151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:38.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e26a00640 1 -- 172.21.15.98:0/1303496281 >> v1:172.21.15.98:6789/0 conn(0x7f7e280c79f0 legacy=0x7f7e28157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:38.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e25600640 1 -- 172.21.15.98:0/1303496281 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102603084 0 0) 0x7f7e280c17d0 con 0x7f7e280c79f0 2024-05-27T00:32:38.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.247+0000 7f7e26a00640 1 -- 172.21.15.98:0/1303496281 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e280710f0 con 0x7f7e280731f0 2024-05-27T00:32:38.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.251+0000 7f7e26a00640 1 --2- 172.21.15.98:0/1303496281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280731f0 0x7f7e280c0eb0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009e50 tx=0x7f7e1801f230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4204cfd10bbd4fbb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:38.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e25600640 1 -- 172.21.15.98:0/1303496281 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e180b4070 con 0x7f7e280731f0 2024-05-27T00:32:38.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e25600640 1 -- 172.21.15.98:0/1303496281 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e180af400 con 0x7f7e280731f0 2024-05-27T00:32:38.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e25600640 1 -- 172.21.15.98:0/1303496281 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e180af7d0 con 0x7f7e280731f0 2024-05-27T00:32:38.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 -- 172.21.15.98:0/1303496281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280731f0 msgr2=0x7f7e280c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:38.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 --2- 172.21.15.98:0/1303496281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280731f0 0x7f7e280c0eb0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f7e18009e50 tx=0x7f7e1801f230 comp rx=0 tx=0).stop 2024-05-27T00:32:38.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 -- 172.21.15.98:0/1303496281 shutdown_connections 2024-05-27T00:32:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 --2- 172.21.15.98:0/1303496281 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280735d0 0x7f7e28151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 --2- 172.21.15.98:0/1303496281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280731f0 0x7f7e280c0eb0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 -- 172.21.15.98:0/1303496281 >> 172.21.15.98:0/1303496281 conn(0x7f7e28067530 msgr2=0x7f7e2806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 -- 172.21.15.98:0/1303496281 shutdown_connections 2024-05-27T00:32:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 -- 172.21.15.98:0/1303496281 wait complete. 2024-05-27T00:32:38.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 Processor -- start 2024-05-27T00:32:38.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 -- start start 2024-05-27T00:32:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e280731f0 0x7f7e28151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.255+0000 7f7e2de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280735d0 0x7f7e281f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e2de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280c79f0 0x7f7e28146510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e2de00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e280c8400 con 0x7f7e280735d0 2024-05-27T00:32:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e2de00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7e280716b0 con 0x7f7e280731f0 2024-05-27T00:32:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e2de00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7e280c13f0 con 0x7f7e280c79f0 2024-05-27T00:32:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e26000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280735d0 0x7f7e281f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e26000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280735d0 0x7f7e281f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52730/0 (socket says 172.21.15.98:52730) 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e26000640 1 -- 172.21.15.98:0/3879553893 learned_addr learned my addr 172.21.15.98:0/3879553893 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e26a00640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e280731f0 0x7f7e28151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e27400640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280c79f0 0x7f7e28146510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e26000640 1 -- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280c79f0 msgr2=0x7f7e28146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e26000640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280c79f0 0x7f7e28146510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e26000640 1 -- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e280731f0 msgr2=0x7f7e28151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e26000640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e280731f0 0x7f7e28151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e26000640 1 -- 172.21.15.98:0/3879553893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e28155eb0 con 0x7f7e280735d0 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e27400640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280c79f0 0x7f7e28146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e26000640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280735d0 0x7f7e281f43b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400e900 tx=0x7f7e1400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e1f400640 1 -- 172.21.15.98:0/3879553893 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14017070 con 0x7f7e280735d0 2024-05-27T00:32:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e1f400640 1 -- 172.21.15.98:0/3879553893 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e1400ca00 con 0x7f7e280735d0 2024-05-27T00:32:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e1f400640 1 -- 172.21.15.98:0/3879553893 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1400cdf0 con 0x7f7e280735d0 2024-05-27T00:32:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e2de00640 1 -- 172.21.15.98:0/3879553893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e28156960 con 0x7f7e280735d0 2024-05-27T00:32:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e2de00640 1 -- 172.21.15.98:0/3879553893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e280c8740 con 0x7f7e280735d0 2024-05-27T00:32:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.259+0000 7f7e2de00640 1 -- 172.21.15.98:0/3879553893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e28075cb0 con 0x7f7e280735d0 2024-05-27T00:32:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.263+0000 7f7e1f400640 1 -- 172.21.15.98:0/3879553893 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7e1401a070 con 0x7f7e280735d0 2024-05-27T00:32:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.263+0000 7f7e1f400640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7e080678f0 0x7f7e08069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.263+0000 7f7e1f400640 1 -- 172.21.15.98:0/3879553893 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f7e14086070 con 0x7f7e280735d0 2024-05-27T00:32:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.263+0000 7f7e26a00640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7e080678f0 0x7f7e08069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.263+0000 7f7e26a00640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7e080678f0 0x7f7e08069db0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7e180b1000 tx=0x7f7e180aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:38.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.267+0000 7f7e1f400640 1 -- 172.21.15.98:0/3879553893 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f7e140560d0 con 0x7f7e280735d0 2024-05-27T00:32:38.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.651+0000 7f7e2de00640 1 -- 172.21.15.98:0/3879553893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e28158510 con 0x7f7e280735d0 2024-05-27T00:32:38.670 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:38.670 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:32.656788+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6396,"qdb_cluster":[6396]},"id":1}]} 2024-05-27T00:32:38.670 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 72 2024-05-27T00:32:38.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.659+0000 7f7e1f400640 1 -- 172.21.15.98:0/3879553893 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6014 (secure 0 0 0) 0x7f7e1405a3d0 con 0x7f7e280735d0 2024-05-27T00:32:38.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.675+0000 7f7e2de00640 1 -- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7e080678f0 msgr2=0x7f7e08069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:38.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.675+0000 7f7e2de00640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7e080678f0 0x7f7e08069db0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7e180b1000 tx=0x7f7e180aa000 comp rx=0 tx=0).stop 2024-05-27T00:32:38.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.675+0000 7f7e2de00640 1 -- 172.21.15.98:0/3879553893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280735d0 msgr2=0x7f7e281f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:38.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.675+0000 7f7e2de00640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280735d0 0x7f7e281f43b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400e900 tx=0x7f7e1400edc0 comp rx=0 tx=0).stop 2024-05-27T00:32:38.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.699+0000 7f7e2de00640 1 -- 172.21.15.98:0/3879553893 shutdown_connections 2024-05-27T00:32:38.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.699+0000 7f7e2de00640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e280c79f0 0x7f7e28146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:38.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.699+0000 7f7e2de00640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e280735d0 0x7f7e281f43b0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:38.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.699+0000 7f7e2de00640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7e080678f0 0x7f7e08069db0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:38.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.699+0000 7f7e2de00640 1 --2- 172.21.15.98:0/3879553893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e280731f0 0x7f7e28151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:38.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.699+0000 7f7e2de00640 1 -- 172.21.15.98:0/3879553893 >> 172.21.15.98:0/3879553893 conn(0x7f7e28067530 msgr2=0x7f7e2814fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:38.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.703+0000 7f7e2de00640 1 -- 172.21.15.98:0/3879553893 shutdown_connections 2024-05-27T00:32:38.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:38.703+0000 7f7e2de00640 1 -- 172.21.15.98:0/3879553893 wait complete. 2024-05-27T00:32:38.765 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-05-27T00:32:40.554 INFO:tasks.ceph.mds.c.smithi098.stderr:2024-05-27T00:32:40.551+0000 7f221ce00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T00:32:40.727 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 68% (224/329) Updating files: 69% (228/329) Updating files: 70% (231/329) Updating files: 71% (234/329) Updating files: 72% (237/329) Updating files: 72% (238/329) Updating files: 73% (241/329) Updating files: 74% (244/329) Updating files: 75% (247/329) Updating files: 76% (251/329) Updating files: 77% (254/329) Updating files: 78% (257/329) Updating files: 79% (260/329) Updating files: 80% (264/329) Updating files: 81% (267/329) Updating files: 82% (270/329) Updating files: 83% (274/329) Updating files: 84% (277/329) Updating files: 85% (280/329) Updating files: 86% (283/329) Updating files: 87% (287/329) Updating files: 88% (290/329) Updating files: 89% (293/329) Updating files: 90% (297/329) Updating files: 91% (300/329) Updating files: 92% (303/329) Updating files: 93% (306/329) Updating files: 94% (310/329) Updating files: 95% (313/329) Updating files: 96% (316/329) Updating files: 97% (320/329) Updating files: 98% (323/329) Updating files: 99% (326/329) Updating files: 100% (329/329) Updating files: 100% (329/329), done. 2024-05-27T00:32:40.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:32:40.827 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 4c2d73a5095 0.79 2024-05-27T00:32:40.830 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.79 2024-05-27T00:32:40.848 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 80 -eq 48 ] 2024-05-27T00:32:40.848 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.80 2024-05-27T00:32:40.848 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.80 2024-05-27T00:32:40.849 INFO:tasks.workunit.client.0.smithi098.stdout:v0.80 2024-05-27T00:32:40.849 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.80 2024-05-27T00:32:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.915+0000 7fa2c4c00640 1 Processor -- start 2024-05-27T00:32:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2c4c00640 1 -- start start 2024-05-27T00:32:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2c4c00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa2c00cc4f0 0x7fa2c00c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2c4c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa2c00c67c0 0x7fa2c00cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2c4c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00629d0 0x7fa2c015bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2c4c00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa2c00c6330 con 0x7fa2c00cc4f0 2024-05-27T00:32:40.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2c4c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa2c00c6e60 con 0x7fa2c00c67c0 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2c4c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa2c015c0a0 con 0x7fa2c00629d0 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bf400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa2c00cc4f0 0x7fa2c00c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bf400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa2c00cc4f0 0x7fa2c00c7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52746/0 (socket says 172.21.15.98:52746) 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bf400640 1 -- 172.21.15.98:0/2363984128 learned_addr learned my addr 172.21.15.98:0/2363984128 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bea00640 1 --2- 172.21.15.98:0/2363984128 >> v2:172.21.15.204:3300/0 conn(0x7fa2c00c67c0 0x7fa2c00cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bfe00640 1 --2- 172.21.15.98:0/2363984128 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00629d0 0x7fa2c015bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bf400640 1 -- 172.21.15.98:0/2363984128 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00629d0 msgr2=0x7fa2c015bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bf400640 1 --2- 172.21.15.98:0/2363984128 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00629d0 0x7fa2c015bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bf400640 1 -- 172.21.15.98:0/2363984128 >> v2:172.21.15.204:3300/0 conn(0x7fa2c00c67c0 msgr2=0x7fa2c00cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bf400640 1 --2- 172.21.15.98:0/2363984128 >> v2:172.21.15.204:3300/0 conn(0x7fa2c00c67c0 0x7fa2c00cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bf400640 1 -- 172.21.15.98:0/2363984128 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c00c70b0 con 0x7fa2c00cc4f0 2024-05-27T00:32:40.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.919+0000 7fa2bfe00640 1 --2- 172.21.15.98:0/2363984128 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00629d0 0x7fa2c015bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:32:40.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2bf400640 1 --2- 172.21.15.98:0/2363984128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00cc4f0 0x7fa2c00c7720 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00c5cc0 tx=0x7fa2b001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ecf6514e8d4c5d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:40.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2be000640 1 -- 172.21.15.98:0/2363984128 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b00b0070 con 0x7fa2c00cc4f0 2024-05-27T00:32:40.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2be000640 1 -- 172.21.15.98:0/2363984128 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2b00ab600 con 0x7fa2c00cc4f0 2024-05-27T00:32:40.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2be000640 1 -- 172.21.15.98:0/2363984128 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b00ab9d0 con 0x7fa2c00cc4f0 2024-05-27T00:32:40.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 -- 172.21.15.98:0/2363984128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00cc4f0 msgr2=0x7fa2c00c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:40.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 --2- 172.21.15.98:0/2363984128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00cc4f0 0x7fa2c00c7720 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00c5cc0 tx=0x7fa2b001c520 comp rx=0 tx=0).stop 2024-05-27T00:32:40.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 -- 172.21.15.98:0/2363984128 shutdown_connections 2024-05-27T00:32:40.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 --2- 172.21.15.98:0/2363984128 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00629d0 0x7fa2c015bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 --2- 172.21.15.98:0/2363984128 >> v2:172.21.15.204:3300/0 conn(0x7fa2c00c67c0 0x7fa2c00cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 --2- 172.21.15.98:0/2363984128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00cc4f0 0x7fa2c00c7720 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 -- 172.21.15.98:0/2363984128 >> 172.21.15.98:0/2363984128 conn(0x7fa2c006dd00 msgr2=0x7fa2c006b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 -- 172.21.15.98:0/2363984128 shutdown_connections 2024-05-27T00:32:40.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 -- 172.21.15.98:0/2363984128 wait complete. 2024-05-27T00:32:40.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 Processor -- start 2024-05-27T00:32:40.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.923+0000 7fa2c4c00640 1 -- start start 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2c4c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c00629d0 0x7fa2c00c2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2c4c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00c67c0 0x7fa2c01f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2c4c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00cc4f0 0x7fa2c00c1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2c4c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c00c9980 con 0x7fa2c00c67c0 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2c4c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c00c9800 con 0x7fa2c00629d0 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2c4c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa2c00c9b00 con 0x7fa2c00cc4f0 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00c67c0 0x7fa2c01f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00c67c0 0x7fa2c01f3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52760/0 (socket says 172.21.15.98:52760) 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bea00640 1 -- 172.21.15.98:0/408395616 learned_addr learned my addr 172.21.15.98:0/408395616 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bf400640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c00629d0 0x7fa2c00c2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bfe00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00cc4f0 0x7fa2c00c1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:40.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bea00640 1 -- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00cc4f0 msgr2=0x7fa2c00c1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bea00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00cc4f0 0x7fa2c00c1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bea00640 1 -- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c00629d0 msgr2=0x7fa2c00c2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bea00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c00629d0 0x7fa2c00c2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bea00640 1 -- 172.21.15.98:0/408395616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c01f3560 con 0x7fa2c00c67c0 2024-05-27T00:32:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bfe00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00cc4f0 0x7fa2c00c1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:32:40.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2bea00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00c67c0 0x7fa2c01f3fe0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac00d800 tx=0x7fa2ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2b7400640 1 -- 172.21.15.98:0/408395616 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac00b6c0 con 0x7fa2c00c67c0 2024-05-27T00:32:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2b7400640 1 -- 172.21.15.98:0/408395616 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ac019070 con 0x7fa2c00c67c0 2024-05-27T00:32:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2c4c00640 1 -- 172.21.15.98:0/408395616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2c00c2fa0 con 0x7fa2c00c67c0 2024-05-27T00:32:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2b7400640 1 -- 172.21.15.98:0/408395616 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac00ba60 con 0x7fa2c00c67c0 2024-05-27T00:32:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2c4c00640 1 -- 172.21.15.98:0/408395616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2c00c3560 con 0x7fa2c00c67c0 2024-05-27T00:32:40.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.927+0000 7fa2c4c00640 1 -- 172.21.15.98:0/408395616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2c0075c40 con 0x7fa2c00c67c0 2024-05-27T00:32:40.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.931+0000 7fa2b7400640 1 -- 172.21.15.98:0/408395616 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa2ac00bc00 con 0x7fa2c00c67c0 2024-05-27T00:32:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.931+0000 7fa2b7400640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2a00678c0 0x7fa2a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:32:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.931+0000 7fa2b7400640 1 -- 172.21.15.98:0/408395616 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fa2ac0865a0 con 0x7fa2c00c67c0 2024-05-27T00:32:40.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.931+0000 7fa2bf400640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2a00678c0 0x7fa2a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:32:40.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.931+0000 7fa2bf400640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2a00678c0 0x7fa2a0069d80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00ad000 tx=0x7fa2b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:32:40.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:40.939+0000 7fa2b7400640 1 -- 172.21.15.98:0/408395616 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa2c01f3560 con 0x7fa2c00c67c0 2024-05-27T00:32:41.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.139+0000 7fa2c4c00640 1 -- 172.21.15.98:0/408395616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2c00c3730 con 0x7fa2c00c67c0 2024-05-27T00:32:41.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2b7400640 1 -- 172.21.15.98:0/408395616 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6839 (secure 0 0 0) 0x7fa2c00c3730 con 0x7fa2c00c67c0 2024-05-27T00:32:41.145 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:32:41.146 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3675441597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3675441597},{"type":"v1","addr":"172.21.15.98:6834","nonce":3675441597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:32.656788+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6396,"qdb_cluster":[6396]},"id":1}]} 2024-05-27T00:32:41.146 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 73 2024-05-27T00:32:41.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 -- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2a00678c0 msgr2=0x7fa2a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2a00678c0 0x7fa2a0069d80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00ad000 tx=0x7fa2b0008040 comp rx=0 tx=0).stop 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 -- 172.21.15.98:0/408395616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00c67c0 msgr2=0x7fa2c01f3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00c67c0 0x7fa2c01f3fe0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac00d800 tx=0x7fa2ac00dcc0 comp rx=0 tx=0).stop 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 -- 172.21.15.98:0/408395616 shutdown_connections 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2c00cc4f0 0x7fa2c00c1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2c00c67c0 0x7fa2c01f3fe0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa2a00678c0 0x7fa2a0069d80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 --2- 172.21.15.98:0/408395616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2c00629d0 0x7fa2c00c2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 -- 172.21.15.98:0/408395616 >> 172.21.15.98:0/408395616 conn(0x7fa2c006dd00 msgr2=0x7fa2c0161080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 -- 172.21.15.98:0/408395616 shutdown_connections 2024-05-27T00:32:41.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:32:41.143+0000 7fa2c4c00640 1 -- 172.21.15.98:0/408395616 wait complete. 2024-05-27T00:32:41.167 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-05-27T00:32:41.167 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2024-05-27T00:32:43.508 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 84% (148/176) Updating files: 85% (150/176) Updating files: 86% (152/176) Updating files: 87% (154/176) Updating files: 88% (155/176) Updating files: 89% (157/176) Updating files: 90% (159/176) Updating files: 91% (161/176) Updating files: 92% (162/176) Updating files: 93% (164/176) Updating files: 94% (166/176) Updating files: 95% (168/176) Updating files: 96% (169/176) Updating files: 97% (171/176) Updating files: 98% (173/176) Updating files: 99% (175/176) Updating files: 100% (176/176) Updating files: 100% (176/176), done. 2024-05-27T00:32:43.565 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at b78644e7dee 0.80 2024-05-27T00:32:43.571 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.80 2024-05-27T00:32:43.608 INFO:tasks.workunit.client.0.smithi098.stdout:v0.81 2024-05-27T00:32:43.608 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 81 -eq 48 ] 2024-05-27T00:32:43.608 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.81 2024-05-27T00:32:43.609 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.81 2024-05-27T00:32:43.609 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.81 2024-05-27T00:32:51.333 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 67% (210/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 77% (241/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2024-05-27T00:32:51.565 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 8de9501df27 0.81 2024-05-27T00:32:51.577 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.81 2024-05-27T00:32:51.613 INFO:tasks.workunit.client.0.smithi098.stdout:v0.82 2024-05-27T00:32:51.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 82 -eq 48 ] 2024-05-27T00:32:51.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.82 2024-05-27T00:32:51.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.82 2024-05-27T00:32:51.614 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.82 2024-05-27T00:32:52.644 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 14085f42ddd 0.82 2024-05-27T00:32:52.651 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.82 2024-05-27T00:32:52.730 INFO:tasks.workunit.client.0.smithi098.stdout:v0.83 2024-05-27T00:32:52.731 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 83 -eq 48 ] 2024-05-27T00:32:52.731 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.83 2024-05-27T00:32:52.731 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.83 2024-05-27T00:32:52.731 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.83 2024-05-27T00:32:53.932 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 99% (216/218) Updating files: 100% (218/218) Updating files: 100% (218/218), done. 2024-05-27T00:32:54.018 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 78ff1f0a5df 0.83 2024-05-27T00:32:54.023 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.83 2024-05-27T00:32:54.121 INFO:tasks.workunit.client.0.smithi098.stdout:v0.84 2024-05-27T00:32:54.121 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 84 -eq 48 ] 2024-05-27T00:32:54.122 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.84 2024-05-27T00:32:54.122 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.84 2024-05-27T00:32:54.122 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.84 2024-05-27T00:32:54.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:32:54.420 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:32:56.453 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 37% (133/356) Updating files: 38% (136/356) Updating files: 39% (139/356) Updating files: 40% (143/356) Updating files: 41% (146/356) Updating files: 42% (150/356) Updating files: 43% (154/356) Updating files: 44% (157/356) Updating files: 45% (161/356) Updating files: 46% (164/356) Updating files: 47% (168/356) Updating files: 48% (171/356) Updating files: 49% (175/356) Updating files: 50% (178/356) Updating files: 51% (182/356) Updating files: 52% (186/356) Updating files: 53% (189/356) Updating files: 54% (193/356) Updating files: 55% (196/356) Updating files: 56% (200/356) Updating files: 57% (203/356) Updating files: 58% (207/356) Updating files: 59% (211/356) Updating files: 60% (214/356) Updating files: 61% (218/356) Updating files: 62% (221/356) Updating files: 63% (225/356) Updating files: 64% (228/356) Updating files: 65% (232/356) Updating files: 66% (235/356) Updating files: 67% (239/356) Updating files: 68% (243/356) Updating files: 69% (246/356) Updating files: 70% (250/356) Updating files: 71% (253/356) Updating files: 72% (257/356) Updating files: 73% (260/356) Updating files: 74% (264/356) Updating files: 75% (267/356) Updating files: 76% (271/356) Updating files: 77% (275/356) Updating files: 78% (278/356) Updating files: 79% (282/356) Updating files: 80% (285/356) Updating files: 81% (289/356) Updating files: 82% (292/356) Updating files: 83% (296/356) Updating files: 84% (300/356) Updating files: 85% (303/356) Updating files: 86% (307/356) Updating files: 87% (310/356) Updating files: 88% (314/356) Updating files: 89% (317/356) Updating files: 90% (321/356) Updating files: 91% (324/356) Updating files: 92% (328/356) Updating files: 93% (332/356) Updating files: 93% (334/356) Updating files: 94% (335/356) Updating files: 95% (339/356) Updating files: 96% (342/356) Updating files: 97% (346/356) Updating files: 98% (349/356) Updating files: 99% (353/356) Updating files: 100% (356/356) Updating files: 100% (356/356), done. 2024-05-27T00:32:56.526 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 8336f81c5c7 0.84 2024-05-27T00:32:56.543 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.84 2024-05-27T00:32:56.689 INFO:tasks.workunit.client.0.smithi098.stdout:v0.85 2024-05-27T00:32:56.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 85 -eq 48 ] 2024-05-27T00:32:56.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.85 2024-05-27T00:32:56.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.85 2024-05-27T00:32:56.690 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.85 2024-05-27T00:32:58.090 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at a0c22842db9 0.85 2024-05-27T00:32:58.098 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.85 2024-05-27T00:32:58.225 INFO:tasks.workunit.client.0.smithi098.stdout:v0.86 2024-05-27T00:32:58.225 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 86 -eq 48 ] 2024-05-27T00:32:58.225 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.86 2024-05-27T00:32:58.225 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.86 2024-05-27T00:32:58.225 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.86 2024-05-27T00:33:09.329 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 46% (206/447) Updating files: 47% (211/447) Updating files: 48% (215/447) Updating files: 49% (220/447) Updating files: 50% (224/447) Updating files: 51% (228/447) Updating files: 52% (233/447) Updating files: 53% (237/447) Updating files: 54% (242/447) Updating files: 55% (246/447) Updating files: 56% (251/447) Updating files: 57% (255/447) Updating files: 58% (260/447) Updating files: 59% (264/447) Updating files: 60% (269/447) Updating files: 61% (273/447) Updating files: 62% (278/447) Updating files: 63% (282/447) Updating files: 64% (287/447) Updating files: 65% (291/447) Updating files: 66% (296/447) Updating files: 67% (300/447) Updating files: 68% (304/447) Updating files: 69% (309/447) Updating files: 70% (313/447) Updating files: 70% (314/447) Updating files: 71% (318/447) Updating files: 72% (322/447) Updating files: 73% (327/447) Updating files: 74% (331/447) Updating files: 75% (336/447) Updating files: 76% (340/447) Updating files: 77% (345/447) Updating files: 78% (349/447) Updating files: 79% (354/447) Updating files: 80% (358/447) Updating files: 81% (363/447) Updating files: 82% (367/447) Updating files: 83% (372/447) Updating files: 84% (376/447) Updating files: 85% (380/447) Updating files: 86% (385/447) Updating files: 87% (389/447) Updating files: 88% (394/447) Updating files: 89% (398/447) Updating files: 90% (403/447) Updating files: 91% (407/447) Updating files: 92% (412/447) Updating files: 93% (416/447) Updating files: 94% (421/447) Updating files: 95% (425/447) Updating files: 96% (430/447) Updating files: 97% (434/447) Updating files: 98% (439/447) Updating files: 99% (443/447) Updating files: 100% (447/447) Updating files: 100% (447/447), done. 2024-05-27T00:33:09.379 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 97dcc0539df 0.86 2024-05-27T00:33:09.388 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.86 2024-05-27T00:33:09.549 INFO:tasks.workunit.client.0.smithi098.stdout:v0.87 2024-05-27T00:33:09.549 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 87 -eq 48 ] 2024-05-27T00:33:09.549 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.87 2024-05-27T00:33:09.549 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.87 2024-05-27T00:33:09.549 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.87 2024-05-27T00:33:13.846 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 27% (16/59) Updating files: 28% (17/59) Updating files: 30% (18/59) Updating files: 32% (19/59) Updating files: 33% (20/59) Updating files: 35% (21/59) Updating files: 37% (22/59) Updating files: 38% (23/59) Updating files: 40% (24/59) Updating files: 42% (25/59) Updating files: 44% (26/59) Updating files: 45% (27/59) Updating files: 47% (28/59) Updating files: 49% (29/59) Updating files: 50% (30/59) Updating files: 52% (31/59) Updating files: 54% (32/59) Updating files: 55% (33/59) Updating files: 57% (34/59) Updating files: 59% (35/59) Updating files: 61% (36/59) Updating files: 62% (37/59) Updating files: 64% (38/59) Updating files: 66% (39/59) Updating files: 67% (40/59) Updating files: 69% (41/59) Updating files: 71% (42/59) Updating files: 72% (43/59) Updating files: 74% (44/59) Updating files: 76% (45/59) Updating files: 77% (46/59) Updating files: 79% (47/59) Updating files: 81% (48/59) Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2024-05-27T00:33:13.900 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at c51c8f9d80f 0.87 2024-05-27T00:33:13.903 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.87 2024-05-27T00:33:14.039 INFO:tasks.workunit.client.0.smithi098.stdout:v0.88 2024-05-27T00:33:14.039 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 88 -eq 48 ] 2024-05-27T00:33:14.039 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.88 2024-05-27T00:33:14.039 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.88 2024-05-27T00:33:14.039 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.88 2024-05-27T00:33:16.688 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 74% (248/335) Updating files: 75% (252/335) Updating files: 76% (255/335) Updating files: 77% (258/335) Updating files: 78% (262/335) Updating files: 79% (265/335) Updating files: 80% (268/335) Updating files: 81% (272/335) Updating files: 82% (275/335) Updating files: 83% (279/335) Updating files: 84% (282/335) Updating files: 85% (285/335) Updating files: 86% (289/335) Updating files: 87% (292/335) Updating files: 88% (295/335) Updating files: 89% (299/335) Updating files: 90% (302/335) Updating files: 91% (305/335) Updating files: 92% (309/335) Updating files: 93% (312/335) Updating files: 94% (315/335) Updating files: 95% (319/335) Updating files: 96% (322/335) Updating files: 97% (325/335) Updating files: 98% (329/335) Updating files: 99% (332/335) Updating files: 100% (335/335) Updating files: 100% (335/335), done. 2024-05-27T00:33:16.770 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 4be687bf448 0.88 2024-05-27T00:33:16.785 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.88 2024-05-27T00:33:16.887 INFO:tasks.workunit.client.0.smithi098.stdout:v0.89 2024-05-27T00:33:16.887 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 89 -eq 48 ] 2024-05-27T00:33:16.887 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.89 2024-05-27T00:33:16.887 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.89 2024-05-27T00:33:16.888 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.89 2024-05-27T00:33:18.213 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 68fdc0f68e6 0.89 2024-05-27T00:33:18.219 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.89 2024-05-27T00:33:18.346 INFO:tasks.workunit.client.0.smithi098.stdout:v0.90 2024-05-27T00:33:18.346 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 90 -eq 48 ] 2024-05-27T00:33:18.346 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.90 2024-05-27T00:33:18.346 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo v0.90 2024-05-27T00:33:18.346 INFO:tasks.workunit.client.0.smithi098.stderr:+ git reset --hard v0.90 2024-05-27T00:33:25.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:33:25.439 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:33:29.177 INFO:tasks.workunit.client.0.smithi098.stderr:Updating files: 60% (161/265) Updating files: 61% (162/265) Updating files: 62% (165/265) Updating files: 63% (167/265) Updating files: 64% (170/265) Updating files: 65% (173/265) Updating files: 66% (175/265) Updating files: 67% (178/265) Updating files: 68% (181/265) Updating files: 69% (183/265) Updating files: 70% (186/265) Updating files: 71% (189/265) Updating files: 72% (191/265) Updating files: 73% (194/265) Updating files: 74% (197/265) Updating files: 75% (199/265) Updating files: 76% (202/265) Updating files: 77% (205/265) Updating files: 78% (207/265) Updating files: 79% (210/265) Updating files: 80% (212/265) Updating files: 81% (215/265) Updating files: 82% (218/265) Updating files: 83% (220/265) Updating files: 84% (223/265) Updating files: 85% (226/265) Updating files: 86% (228/265) Updating files: 87% (231/265) Updating files: 88% (234/265) Updating files: 89% (236/265) Updating files: 90% (239/265) Updating files: 91% (242/265) Updating files: 92% (244/265) Updating files: 93% (247/265) Updating files: 94% (250/265) Updating files: 95% (252/265) Updating files: 96% (255/265) Updating files: 97% (258/265) Updating files: 98% (260/265) Updating files: 99% (263/265) Updating files: 100% (265/265) Updating files: 100% (265/265), done. 2024-05-27T00:33:29.215 INFO:tasks.workunit.client.0.smithi098.stdout:HEAD is now at 08bd1e1eee3 0.90 2024-05-27T00:33:29.220 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir .snap/v0.90 2024-05-27T00:33:29.301 INFO:tasks.workunit.client.0.smithi098.stderr:+ [ 1 -eq 48 ] 2024-05-27T00:33:29.301 INFO:tasks.workunit.client.0.smithi098.stderr:+ ver=v0.1 2024-05-27T00:33:29.304 INFO:tasks.workunit.client.0.smithi098.stdout:checking v0.1 2024-05-27T00:33:29.304 INFO:tasks.workunit.client.0.smithi098.stderr:+ echo checking v0.1 2024-05-27T00:33:29.304 INFO:tasks.workunit.client.0.smithi098.stderr:+ cd .snap/v0.1 2024-05-27T00:33:29.499 INFO:tasks.workunit.client.0.smithi098.stderr:+ git diff --exit-code 2024-05-27T00:33:31.114 INFO:tasks.workunit.client.0.smithi098.stdout:diff --git a/INSTALL b/INSTALL 2024-05-27T00:33:31.114 INFO:tasks.workunit.client.0.smithi098.stdout:index c4ef142a116..e69de29bb2d 100644 2024-05-27T00:33:31.114 INFO:tasks.workunit.client.0.smithi098.stdout:--- a/INSTALL 2024-05-27T00:33:31.114 INFO:tasks.workunit.client.0.smithi098.stdout:+++ b/INSTALL 2024-05-27T00:33:31.114 INFO:tasks.workunit.client.0.smithi098.stdout:@@ -1,15 +0,0 @@ 2024-05-27T00:33:31.114 INFO:tasks.workunit.client.0.smithi098.stdout:-Installation Instructions 2024-05-27T00:33:31.114 INFO:tasks.workunit.client.0.smithi098.stdout:-************************* 2024-05-27T00:33:31.114 INFO:tasks.workunit.client.0.smithi098.stdout:- 2024-05-27T00:33:31.114 INFO:tasks.workunit.client.0.smithi098.stdout:-If pulling from git, you first need to build the configure script with 2024-05-27T00:33:31.114 INFO:tasks.workunit.client.0.smithi098.stdout:- 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:-$ ./reconf 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:- 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:-Then, 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:- 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:-$ ./configure 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:-$ make 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:- 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:-Note that if the FUSE library is not found, the user-space fuse client 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:-will not be built. 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:- 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:diff --git a/Makefile.am b/Makefile.am 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:index b2edb4d4475..e69de29bb2d 100644 2024-05-27T00:33:31.115 INFO:tasks.workunit.client.0.smithi098.stdout:--- a/Makefile.am 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:+++ b/Makefile.am 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:@@ -1,3 +0,0 @@ 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:-AUTOMAKE_OPTIONS = gnu 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:-EXTRA_DIST = web autogen.sh 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:-SUBDIRS = src 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:\ No newline at end of file 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:diff --git a/README b/README 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:index aa016817ceb..e69de29bb2d 100644 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:--- a/README 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:+++ b/README 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:@@ -1,4 +0,0 @@ 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:-Ceph - a scalable distributed file system 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:------------------------------------------ 2024-05-27T00:33:31.116 INFO:tasks.workunit.client.0.smithi098.stdout:- 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:-Please see http://ceph.sourceforge.net/ for current info. 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:diff --git a/autogen.sh b/autogen.sh 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:index c5d47c5e8b5..e69de29bb2d 100755 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:--- a/autogen.sh 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:+++ b/autogen.sh 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:@@ -1,7 +0,0 @@ 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:-#!/bin/sh 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:-rm -f config.cache 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:-aclocal 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:-autoconf 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:-autoheader 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:-automake -a --add-missing 2024-05-27T00:33:31.117 INFO:tasks.workunit.client.0.smithi098.stdout:-exit 2024-05-27T00:33:31.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:33:31.119 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2024-05-27T00:33:31.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-27T00:33:32.058 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7d7cb32d1d130135a98121a564d709025c2154b6/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_7d7cb32d1d130135a98121a564d709025c2154b6/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7d7cb32d1d130135a98121a564d709025c2154b6/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-05-27T00:33:36.606 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ac664771b3b6445687dcbc93a64d8036 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7d7cb32d1d130135a98121a564d709025c2154b6/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_7d7cb32d1d130135a98121a564d709025c2154b6/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7d7cb32d1d130135a98121a564d709025c2154b6/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-05-27T00:33:36.609 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-05-27T00:33:36.619 INFO:tasks.mds_thrash:joining mds_thrasher 2024-05-27T00:33:36.622 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2024-05-27T00:33:36.622 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2024-05-27T00:33:36.622 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 12 2024-05-27T00:33:36.623 INFO:tasks.mds_thrash:done joining 2024-05-27T00:33:36.623 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-27T00:33:36.651 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-27T00:33:36.651 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:33:36.652 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T00:33:36.668 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-05-27T00:33:36.668 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T00:33:36.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:33:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f847a400640 1 Processor -- start 2024-05-27T00:33:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f847a400640 1 -- start start 2024-05-27T00:33:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f847a400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84740cbe10 0x7f84741592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f847a400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f84740cc1f0 0x7f84740c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f847a400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84740c6dd0 con 0x7f8474073570 2024-05-27T00:33:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f847a400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f8474159810 con 0x7f84740cc1f0 2024-05-27T00:33:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f847a400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f84740c70b0 con 0x7f84740cbe10 2024-05-27T00:33:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8473400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84740cbe10 0x7f84741592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8472a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f84740cc1f0 0x7f84740c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8472a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f84740cc1f0 0x7f84740c1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:53868/0 (socket says 172.21.15.98:53868) 2024-05-27T00:33:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8473400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84740cbe10 0x7f84741592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:59970/0 (socket says 172.21.15.98:59970) 2024-05-27T00:33:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8472a00640 1 -- 172.21.15.98:0/2011120499 learned_addr learned my addr 172.21.15.98:0/2011120499 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8473e00640 1 --1- 172.21.15.98:0/2011120499 >> v1:172.21.15.204:6789/0 conn(0x7f8474073570 0x7f84741559e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:35768/0 (socket says 172.21.15.98:35768) 2024-05-27T00:33:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8473400640 1 -- 172.21.15.98:0/2011120499 >> v2:172.21.15.204:3300/0 conn(0x7f84740cc1f0 msgr2=0x7f84740c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8473400640 1 --2- 172.21.15.98:0/2011120499 >> v2:172.21.15.204:3300/0 conn(0x7f84740cc1f0 0x7f84740c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8473400640 1 -- 172.21.15.98:0/2011120499 >> v1:172.21.15.204:6789/0 conn(0x7f8474073570 legacy=0x7f84741559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8473400640 1 -- 172.21.15.98:0/2011120499 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84740c7390 con 0x7f84740cbe10 2024-05-27T00:33:36.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.906+0000 7f8473400640 1 --2- 172.21.15.98:0/2011120499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84740cbe10 0x7f84741592d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8464009d90 tx=0x7f846401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36ea9b26ed3689e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f8472000640 1 -- 172.21.15.98:0/2011120499 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84640b4070 con 0x7f84740cbe10 2024-05-27T00:33:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f8472000640 1 -- 172.21.15.98:0/2011120499 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84640af600 con 0x7f84740cbe10 2024-05-27T00:33:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f8472000640 1 -- 172.21.15.98:0/2011120499 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84640af9d0 con 0x7f84740cbe10 2024-05-27T00:33:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 -- 172.21.15.98:0/2011120499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84740cbe10 msgr2=0x7f84741592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 --2- 172.21.15.98:0/2011120499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84740cbe10 0x7f84741592d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8464009d90 tx=0x7f846401f150 comp rx=0 tx=0).stop 2024-05-27T00:33:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 -- 172.21.15.98:0/2011120499 shutdown_connections 2024-05-27T00:33:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 --2- 172.21.15.98:0/2011120499 >> v2:172.21.15.204:3300/0 conn(0x7f84740cc1f0 0x7f84740c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 --2- 172.21.15.98:0/2011120499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84740cbe10 0x7f84741592d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 -- 172.21.15.98:0/2011120499 >> 172.21.15.98:0/2011120499 conn(0x7f8474067530 msgr2=0x7f847406ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 -- 172.21.15.98:0/2011120499 shutdown_connections 2024-05-27T00:33:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 -- 172.21.15.98:0/2011120499 wait complete. 2024-05-27T00:33:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 Processor -- start 2024-05-27T00:33:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 -- start start 2024-05-27T00:33:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8474073570 0x7f84740c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84740cbe10 0x7f84740b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84740cc1f0 0x7f84740b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f84740c79d0 con 0x7f84740cc1f0 2024-05-27T00:33:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f84740c1330 con 0x7f84740cbe10 2024-05-27T00:33:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f847a400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f84740c6990 con 0x7f8474073570 2024-05-27T00:33:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f8473e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84740cc1f0 0x7f84740b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f8473e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84740cc1f0 0x7f84740b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59682/0 (socket says 172.21.15.98:59682) 2024-05-27T00:33:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f8473e00640 1 -- 172.21.15.98:0/622080407 learned_addr learned my addr 172.21.15.98:0/622080407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f8473400640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8474073570 0x7f84740c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.910+0000 7f8472a00640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84740cbe10 0x7f84740b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f8473e00640 1 -- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8474073570 msgr2=0x7f84740c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f8473e00640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8474073570 0x7f84740c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f8473e00640 1 -- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84740cbe10 msgr2=0x7f84740b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f8473e00640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84740cbe10 0x7f84740b03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f8473e00640 1 -- 172.21.15.98:0/622080407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84741f46c0 con 0x7f84740cc1f0 2024-05-27T00:33:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f8473e00640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84740cc1f0 0x7f84740b08e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f846c00c3c0 tx=0x7f846c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f846b400640 1 -- 172.21.15.98:0/622080407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c0074e0 con 0x7f84740cc1f0 2024-05-27T00:33:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f846b400640 1 -- 172.21.15.98:0/622080407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f846c007680 con 0x7f84740cc1f0 2024-05-27T00:33:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f846b400640 1 -- 172.21.15.98:0/622080407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c004a10 con 0x7f84740cc1f0 2024-05-27T00:33:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f847a400640 1 -- 172.21.15.98:0/622080407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84740c3d90 con 0x7f84740cc1f0 2024-05-27T00:33:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f847a400640 1 -- 172.21.15.98:0/622080407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84740afba0 con 0x7f84740cc1f0 2024-05-27T00:33:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f847a400640 1 -- 172.21.15.98:0/622080407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8474075cb0 con 0x7f84740cc1f0 2024-05-27T00:33:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f846b400640 1 -- 172.21.15.98:0/622080407 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f846c01a020 con 0x7f84740cc1f0 2024-05-27T00:33:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f846b400640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84540678c0 0x7f8454069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.914+0000 7f846b400640 1 -- 172.21.15.98:0/622080407 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f846c085850 con 0x7f84740cc1f0 2024-05-27T00:33:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.918+0000 7f8473400640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84540678c0 0x7f8454069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.918+0000 7f8473400640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84540678c0 0x7f8454069d80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f84640b1000 tx=0x7f84640aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:36.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:36.922+0000 7f846b400640 1 -- 172.21.15.98:0/622080407 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f846c055790 con 0x7f84740cc1f0 2024-05-27T00:33:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.078+0000 7f847a400640 1 -- 172.21.15.98:0/622080407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8474155170 con 0x7f84740cc1f0 2024-05-27T00:33:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f846b400640 1 -- 172.21.15.98:0/622080407 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+18411 (secure 0 0 0) 0x7f846c059a90 con 0x7f84740cc1f0 2024-05-27T00:33:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:33:37.086 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:32:10.313136+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6834/3070334683":"2024-05-28T00:32:10.313051+0000","172.21.15.98:6836/1802902856":"2024-05-28T00:30:40.295945+0000","172.21.15.98:6834/2148124202":"2024-05-28T00:28:50.279743+0000","172.21.15.98:6832/2148124202":"2024-05-28T00:28:50.279743+0000","172.21.15.98:6837/2940211164":"2024-05-28T00:27:55.271086+0000","172.21.15.98:6836/2940211164":"2024-05-28T00:27:55.271086+0000","172.21.15.98:6834/1223646158":"2024-05-28T00:27:00.259848+0000","172.21.15.98:6837/518550803":"2024-05-28T00:24:50.239991+0000","172.21.15.98:6836/3824013488":"2024-05-28T00:16:25.159349+0000","172.21.15.98:6832/3070334683":"2024-05-28T00:32:10.313051+0000","172.21.15.98:6832/3247908011":"2024-05-28T00:21:25.205263+0000","172.21.15.98:6834/4290077910":"2024-05-28T00:23:40.226716+0000","172.21.15.98:6832/4290077910":"2024-05-28T00:23:40.226716+0000","172.21.15.98:6837/1802902856":"2024-05-28T00:30:40.295945+0000","172.21.15.98:6832/3903574515":"2024-05-28T00:18:25.179026+0000","172.21.15.98:6837/3824013488":"2024-05-28T00:16:25.159349+0000","172.21.15.98:6836/1781395818":"2024-05-28T00:19:35.189766+0000","172.21.15.98:6834/3903574515":"2024-05-28T00:18:25.179026+0000","172.21.15.98:6837/1781395818":"2024-05-28T00:19:35.189766+0000","172.21.15.98:6834/3247908011":"2024-05-28T00:21:25.205263+0000","172.21.15.98:6832/1223646158":"2024-05-28T00:27:00.259848+0000","172.21.15.98:6836/1698253956":"2024-05-28T00:22:20.213891+0000","172.21.15.98:6836/518550803":"2024-05-28T00:24:50.239991+0000","172.21.15.98:6837/1698253956":"2024-05-28T00:22:20.213891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:33:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 -- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84540678c0 msgr2=0x7f8454069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84540678c0 0x7f8454069d80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f84640b1000 tx=0x7f84640aa000 comp rx=0 tx=0).stop 2024-05-27T00:33:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 -- 172.21.15.98:0/622080407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84740cc1f0 msgr2=0x7f84740b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84740cc1f0 0x7f84740b08e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f846c00c3c0 tx=0x7f846c00c880 comp rx=0 tx=0).stop 2024-05-27T00:33:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 -- 172.21.15.98:0/622080407 shutdown_connections 2024-05-27T00:33:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84740cc1f0 0x7f84740b08e0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f84540678c0 0x7f8454069d80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84740cbe10 0x7f84740b03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 --2- 172.21.15.98:0/622080407 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8474073570 0x7f84740c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 -- 172.21.15.98:0/622080407 >> 172.21.15.98:0/622080407 conn(0x7f8474067530 msgr2=0x7f84740c59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 -- 172.21.15.98:0/622080407 shutdown_connections 2024-05-27T00:33:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.082+0000 7f847a400640 1 -- 172.21.15.98:0/622080407 wait complete. 2024-05-27T00:33:37.108 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T00:15:13.211703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T00:15:23.224464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T00:15:23.950896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 35, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T00:33:37.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T00:33:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 Processor -- start 2024-05-27T00:33:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 -- start start 2024-05-27T00:33:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4e6c0c2b10 0x7f4e6c162d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e6c0c8420 con 0x7f4e6c1516b0 2024-05-27T00:33:37.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e6c0c1df0 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4e6c0c3000 con 0x7f4e6c0c2b10 2024-05-27T00:33:37.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e6be00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4e6c1516b0 0x7f4e6c0c1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56872/0 (socket says 172.21.15.98:56872) 2024-05-27T00:33:37.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e71000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4e6c0c2b10 0x7f4e6c162d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e6be00640 1 -- 172.21.15.98:0/3192176295 learned_addr learned my addr 172.21.15.98:0/3192176295 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:37.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e71000640 1 -- 172.21.15.98:0/3192176295 >> v1:172.21.15.204:6789/0 conn(0x7f4e6c0b1e10 legacy=0x7f4e6c06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e71000640 1 -- 172.21.15.98:0/3192176295 >> v1:172.21.15.98:6789/0 conn(0x7f4e6c1516b0 legacy=0x7f4e6c0c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e71000640 1 -- 172.21.15.98:0/3192176295 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e6c0c7e60 con 0x7f4e6c0c2b10 2024-05-27T00:33:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e71000640 1 --2- 172.21.15.98:0/3192176295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0c2b10 0x7f4e6c162d30 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4e640097c0 tx=0x7f4e6401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c03486e4703d9d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e6aa00640 1 -- 172.21.15.98:0/3192176295 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e640b1070 con 0x7f4e6c0c2b10 2024-05-27T00:33:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e6aa00640 1 -- 172.21.15.98:0/3192176295 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e640ac5f0 con 0x7f4e6c0c2b10 2024-05-27T00:33:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e6aa00640 1 -- 172.21.15.98:0/3192176295 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e640ac9c0 con 0x7f4e6c0c2b10 2024-05-27T00:33:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 -- 172.21.15.98:0/3192176295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0c2b10 msgr2=0x7f4e6c162d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 --2- 172.21.15.98:0/3192176295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0c2b10 0x7f4e6c162d30 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f4e640097c0 tx=0x7f4e6401c4a0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 -- 172.21.15.98:0/3192176295 shutdown_connections 2024-05-27T00:33:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 --2- 172.21.15.98:0/3192176295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0c2b10 0x7f4e6c162d30 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 -- 172.21.15.98:0/3192176295 >> 172.21.15.98:0/3192176295 conn(0x7f4e6c0cbf00 msgr2=0x7f4e6c0cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 -- 172.21.15.98:0/3192176295 shutdown_connections 2024-05-27T00:33:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.246+0000 7f4e73000640 1 -- 172.21.15.98:0/3192176295 wait complete. 2024-05-27T00:33:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e73000640 1 Processor -- start 2024-05-27T00:33:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e73000640 1 -- start start 2024-05-27T00:33:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e73000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0b1e10 0x7f4e6c0c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e73000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e6c0c2b10 0x7f4e6c148bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e73000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e6c1516b0 0x7f4e6c0c5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e73000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c151c10 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e73000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c151a90 con 0x7f4e6c0c2b10 2024-05-27T00:33:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e73000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4e6c06ee90 con 0x7f4e6c1516b0 2024-05-27T00:33:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e6be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0b1e10 0x7f4e6c0c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e6be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0b1e10 0x7f4e6c0c4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59700/0 (socket says 172.21.15.98:59700) 2024-05-27T00:33:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e6be00640 1 -- 172.21.15.98:0/1808137984 learned_addr learned my addr 172.21.15.98:0/1808137984 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e71000640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e6c1516b0 0x7f4e6c0c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e6b400640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e6c0c2b10 0x7f4e6c148bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e6be00640 1 -- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e6c1516b0 msgr2=0x7f4e6c0c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e6be00640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e6c1516b0 0x7f4e6c0c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e6be00640 1 -- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e6c0c2b10 msgr2=0x7f4e6c148bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e6be00640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e6c0c2b10 0x7f4e6c148bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e6be00640 1 -- 172.21.15.98:0/1808137984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e6c1f45f0 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e6be00640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0b1e10 0x7f4e6c0c4090 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00cbc0 tx=0x7f4e5c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e63e00640 1 -- 172.21.15.98:0/1808137984 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c017070 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e63e00640 1 -- 172.21.15.98:0/1808137984 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e5c00aca0 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e63e00640 1 -- 172.21.15.98:0/1808137984 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c012600 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e73000640 1 -- 172.21.15.98:0/1808137984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e6c0b0750 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.250+0000 7f4e73000640 1 -- 172.21.15.98:0/1808137984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e6c0c4e60 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.254+0000 7f4e73000640 1 -- 172.21.15.98:0/1808137984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e6c075cb0 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.254+0000 7f4e63e00640 1 -- 172.21.15.98:0/1808137984 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4e5c01a070 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.254+0000 7f4e63e00640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4e4c0678c0 0x7f4e4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.254+0000 7f4e63e00640 1 -- 172.21.15.98:0/1808137984 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f4e5c085d50 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.254+0000 7f4e6b400640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4e4c0678c0 0x7f4e4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.254+0000 7f4e6b400640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4e4c0678c0 0x7f4e4c069d80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c149a10 tx=0x7f4e58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:37.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.258+0000 7f4e63e00640 1 -- 172.21.15.98:0/1808137984 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f4e5c055db0 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.414+0000 7f4e73000640 1 -- 172.21.15.98:0/1808137984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4e6c0c45d0 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.414+0000 7f4e63e00640 1 -- 172.21.15.98:0/1808137984 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7f4e5c05a0b0 con 0x7f4e6c0b1e10 2024-05-27T00:33:37.420 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-05-27T00:33:37.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.414+0000 7f4e73000640 1 -- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4e4c0678c0 msgr2=0x7f4e4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.414+0000 7f4e73000640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4e4c0678c0 0x7f4e4c069d80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c149a10 tx=0x7f4e58009040 comp rx=0 tx=0).stop 2024-05-27T00:33:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.414+0000 7f4e73000640 1 -- 172.21.15.98:0/1808137984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0b1e10 msgr2=0x7f4e6c0c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.414+0000 7f4e73000640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0b1e10 0x7f4e6c0c4090 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00cbc0 tx=0x7f4e5c00a060 comp rx=0 tx=0).stop 2024-05-27T00:33:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.418+0000 7f4e73000640 1 -- 172.21.15.98:0/1808137984 shutdown_connections 2024-05-27T00:33:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.418+0000 7f4e73000640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e6c1516b0 0x7f4e6c0c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.418+0000 7f4e73000640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f4e4c0678c0 0x7f4e4c069d80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.418+0000 7f4e73000640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e6c0c2b10 0x7f4e6c148bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.418+0000 7f4e73000640 1 --2- 172.21.15.98:0/1808137984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e6c0b1e10 0x7f4e6c0c4090 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.418+0000 7f4e73000640 1 -- 172.21.15.98:0/1808137984 >> 172.21.15.98:0/1808137984 conn(0x7f4e6c0cbf00 msgr2=0x7f4e6c0c6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.418+0000 7f4e73000640 1 -- 172.21.15.98:0/1808137984 shutdown_connections 2024-05-27T00:33:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.418+0000 7f4e73000640 1 -- 172.21.15.98:0/1808137984 wait complete. 2024-05-27T00:33:37.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T00:33:37.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 Processor -- start 2024-05-27T00:33:37.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 -- start start 2024-05-27T00:33:37.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1918152990 0x7f191806e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f191806ecc0 0x7f19180c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f191815bc40 con 0x7f19180b1c20 2024-05-27T00:33:37.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f191815c850 con 0x7f1918152990 2024-05-27T00:33:37.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f191815bf20 con 0x7f191806ecc0 2024-05-27T00:33:37.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1916a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1918152990 0x7f191806e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1916a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1918152990 0x7f191806e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59708/0 (socket says 172.21.15.98:59708) 2024-05-27T00:33:37.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1916a00640 1 -- 172.21.15.98:0/119901029 learned_addr learned my addr 172.21.15.98:0/119901029 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:37.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1916000640 1 --2- 172.21.15.98:0/119901029 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f191806ecc0 0x7f19180c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1916a00640 1 -- 172.21.15.98:0/119901029 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f191806ecc0 msgr2=0x7f19180c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:37.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1916a00640 1 --2- 172.21.15.98:0/119901029 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f191806ecc0 0x7f19180c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1916a00640 1 -- 172.21.15.98:0/119901029 >> v1:172.21.15.204:6789/0 conn(0x7f19180b1c20 legacy=0x7f1918158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:37.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1916a00640 1 -- 172.21.15.98:0/119901029 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f191815c200 con 0x7f1918152990 2024-05-27T00:33:37.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1916a00640 1 --2- 172.21.15.98:0/119901029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1918152990 0x7f191806e780 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1908009e50 tx=0x7f190801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf140cb67e631d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1915600640 1 -- 172.21.15.98:0/119901029 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19080a5670 con 0x7f1918152990 2024-05-27T00:33:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1915600640 1 -- 172.21.15.98:0/119901029 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19080a5810 con 0x7f1918152990 2024-05-27T00:33:37.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f1915600640 1 -- 172.21.15.98:0/119901029 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19080a5be0 con 0x7f1918152990 2024-05-27T00:33:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 -- 172.21.15.98:0/119901029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1918152990 msgr2=0x7f191806e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 --2- 172.21.15.98:0/119901029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1918152990 0x7f191806e780 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1908009e50 tx=0x7f190801c9f0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 -- 172.21.15.98:0/119901029 shutdown_connections 2024-05-27T00:33:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 --2- 172.21.15.98:0/119901029 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f191806ecc0 0x7f19180c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 --2- 172.21.15.98:0/119901029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1918152990 0x7f191806e780 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.578+0000 7f191dc00640 1 -- 172.21.15.98:0/119901029 >> 172.21.15.98:0/119901029 conn(0x7f19180cbf00 msgr2=0x7f19180cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:37.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 -- 172.21.15.98:0/119901029 shutdown_connections 2024-05-27T00:33:37.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 -- 172.21.15.98:0/119901029 wait complete. 2024-05-27T00:33:37.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 Processor -- start 2024-05-27T00:33:37.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 -- start start 2024-05-27T00:33:37.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191806ecc0 0x7f191814e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19180b1c20 0x7f19181532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1918152990 0x7f1918153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19181659e0 con 0x7f191806ecc0 2024-05-27T00:33:37.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1918165860 con 0x7f19180b1c20 2024-05-27T00:33:37.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1918165b60 con 0x7f1918152990 2024-05-27T00:33:37.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1916a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191806ecc0 0x7f191814e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1916a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191806ecc0 0x7f191814e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59710/0 (socket says 172.21.15.98:59710) 2024-05-27T00:33:37.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1916a00640 1 -- 172.21.15.98:0/1126280148 learned_addr learned my addr 172.21.15.98:0/1126280148 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1916000640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19180b1c20 0x7f19181532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1917400640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1918152990 0x7f1918153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1916a00640 1 -- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1918152990 msgr2=0x7f1918153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1916a00640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1918152990 0x7f1918153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1916a00640 1 -- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19180b1c20 msgr2=0x7f19181532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1916a00640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19180b1c20 0x7f19181532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1916a00640 1 -- 172.21.15.98:0/1126280148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1918160fe0 con 0x7f191806ecc0 2024-05-27T00:33:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f1916a00640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191806ecc0 0x7f191814e310 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f190801b0a0 tx=0x7f1908019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f190ea00640 1 -- 172.21.15.98:0/1126280148 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19080bc020 con 0x7f191806ecc0 2024-05-27T00:33:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f190ea00640 1 -- 172.21.15.98:0/1126280148 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19080b88a0 con 0x7f191806ecc0 2024-05-27T00:33:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f190ea00640 1 -- 172.21.15.98:0/1126280148 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19080a5670 con 0x7f191806ecc0 2024-05-27T00:33:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 -- 172.21.15.98:0/1126280148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19181f4890 con 0x7f191806ecc0 2024-05-27T00:33:37.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.582+0000 7f191dc00640 1 -- 172.21.15.98:0/1126280148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19180c53f0 con 0x7f191806ecc0 2024-05-27T00:33:37.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.586+0000 7f191dc00640 1 -- 172.21.15.98:0/1126280148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1918075cb0 con 0x7f191806ecc0 2024-05-27T00:33:37.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.586+0000 7f190ea00640 1 -- 172.21.15.98:0/1126280148 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f19080bf020 con 0x7f191806ecc0 2024-05-27T00:33:37.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.586+0000 7f190ea00640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f8067910 0x7f18f8069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.586+0000 7f190ea00640 1 -- 172.21.15.98:0/1126280148 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f1908121250 con 0x7f191806ecc0 2024-05-27T00:33:37.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.586+0000 7f1916000640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f8067910 0x7f18f8069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.586+0000 7f1916000640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f8067910 0x7f18f8069dd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f19180b1960 tx=0x7f1904008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:37.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.590+0000 7f190ea00640 1 -- 172.21.15.98:0/1126280148 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f19080c6030 con 0x7f191806ecc0 2024-05-27T00:33:37.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.750+0000 7f191dc00640 1 -- 172.21.15.98:0/1126280148 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1918157970 con 0x7f191806ecc0 2024-05-27T00:33:37.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.750+0000 7f190ea00640 1 -- 172.21.15.98:0/1126280148 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7f19080165e0 con 0x7f191806ecc0 2024-05-27T00:33:37.757 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-05-27T00:33:37.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 -- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f8067910 msgr2=0x7f18f8069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f8067910 0x7f18f8069dd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f19180b1960 tx=0x7f1904008040 comp rx=0 tx=0).stop 2024-05-27T00:33:37.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 -- 172.21.15.98:0/1126280148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191806ecc0 msgr2=0x7f191814e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191806ecc0 0x7f191814e310 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f190801b0a0 tx=0x7f1908019790 comp rx=0 tx=0).stop 2024-05-27T00:33:37.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 -- 172.21.15.98:0/1126280148 shutdown_connections 2024-05-27T00:33:37.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1918152990 0x7f1918153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f18f8067910 0x7f18f8069dd0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19180b1c20 0x7f19181532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 --2- 172.21.15.98:0/1126280148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f191806ecc0 0x7f191814e310 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 -- 172.21.15.98:0/1126280148 >> 172.21.15.98:0/1126280148 conn(0x7f19180cbf00 msgr2=0x7f19180cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:37.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 -- 172.21.15.98:0/1126280148 shutdown_connections 2024-05-27T00:33:37.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.754+0000 7f191dc00640 1 -- 172.21.15.98:0/1126280148 wait complete. 2024-05-27T00:33:37.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T00:33:37.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb89e200640 1 Processor -- start 2024-05-27T00:33:37.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb89e200640 1 -- start start 2024-05-27T00:33:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb89e200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8981516b0 0x7fb8980c1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb89e200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fb8980c2b10 0x7fb898161d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb89e200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8980c8420 con 0x7fb8980b1e10 2024-05-27T00:33:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb89e200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb8980c1df0 con 0x7fb8981516b0 2024-05-27T00:33:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb89e200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fb8980c3000 con 0x7fb8980c2b10 2024-05-27T00:33:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb897400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8981516b0 0x7fb8980c1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb896a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb8980b1e10 0x7fb89806e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56884/0 (socket says 172.21.15.98:56884) 2024-05-27T00:33:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb896a00640 1 -- 172.21.15.98:0/4253844764 learned_addr learned my addr 172.21.15.98:0/4253844764 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb897e00640 1 --2- 172.21.15.98:0/4253844764 >> v2:172.21.15.204:3300/0 conn(0x7fb8980c2b10 0x7fb898161d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb897400640 1 -- 172.21.15.98:0/4253844764 >> v2:172.21.15.204:3300/0 conn(0x7fb8980c2b10 msgr2=0x7fb898161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb897400640 1 --2- 172.21.15.98:0/4253844764 >> v2:172.21.15.204:3300/0 conn(0x7fb8980c2b10 0x7fb898161d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb897400640 1 -- 172.21.15.98:0/4253844764 >> v1:172.21.15.98:6789/0 conn(0x7fb8980b1e10 legacy=0x7fb89806e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb897400640 1 -- 172.21.15.98:0/4253844764 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8980c7e60 con 0x7fb8981516b0 2024-05-27T00:33:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.914+0000 7fb896000640 1 -- 172.21.15.98:0/4253844764 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4040791993 0 0) 0x7fb8980c8420 con 0x7fb8980b1e10 2024-05-27T00:33:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb897400640 1 --2- 172.21.15.98:0/4253844764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8981516b0 0x7fb8980c1300 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb898158590 tx=0x7fb88801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8263e36c4e425282 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb896000640 1 -- 172.21.15.98:0/4253844764 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880b0070 con 0x7fb8981516b0 2024-05-27T00:33:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb896000640 1 -- 172.21.15.98:0/4253844764 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8880ab600 con 0x7fb8981516b0 2024-05-27T00:33:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb896000640 1 -- 172.21.15.98:0/4253844764 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880ab9d0 con 0x7fb8981516b0 2024-05-27T00:33:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb89e200640 1 -- 172.21.15.98:0/4253844764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8981516b0 msgr2=0x7fb8980c1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb89e200640 1 --2- 172.21.15.98:0/4253844764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8981516b0 0x7fb8980c1300 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb898158590 tx=0x7fb88801c480 comp rx=0 tx=0).stop 2024-05-27T00:33:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb89e200640 1 -- 172.21.15.98:0/4253844764 shutdown_connections 2024-05-27T00:33:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb89e200640 1 --2- 172.21.15.98:0/4253844764 >> v2:172.21.15.204:3300/0 conn(0x7fb8980c2b10 0x7fb898161d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb89e200640 1 --2- 172.21.15.98:0/4253844764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8981516b0 0x7fb8980c1300 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb89e200640 1 -- 172.21.15.98:0/4253844764 >> 172.21.15.98:0/4253844764 conn(0x7fb8980cbf00 msgr2=0x7fb8980cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb89e200640 1 -- 172.21.15.98:0/4253844764 shutdown_connections 2024-05-27T00:33:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb89e200640 1 -- 172.21.15.98:0/4253844764 wait complete. 2024-05-27T00:33:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb89e200640 1 Processor -- start 2024-05-27T00:33:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.918+0000 7fb89e200640 1 -- start start 2024-05-27T00:33:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb89e200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980b1e10 0x7fb89806d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb89e200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8980c2b10 0x7fb8981f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb89e200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8981516b0 0x7fb898145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb89e200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb898073030 con 0x7fb8980b1e10 2024-05-27T00:33:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb89e200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb898072eb0 con 0x7fb8981516b0 2024-05-27T00:33:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb89e200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb898151a90 con 0x7fb8980c2b10 2024-05-27T00:33:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980b1e10 0x7fb89806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980b1e10 0x7fb89806d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59728/0 (socket says 172.21.15.98:59728) 2024-05-27T00:33:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897400640 1 -- 172.21.15.98:0/3060234011 learned_addr learned my addr 172.21.15.98:0/3060234011 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897e00640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8981516b0 0x7fb898145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb896a00640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8980c2b10 0x7fb8981f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897400640 1 -- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8980c2b10 msgr2=0x7fb8981f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897400640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8980c2b10 0x7fb8981f4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897400640 1 -- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8981516b0 msgr2=0x7fb898145640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897400640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8981516b0 0x7fb898145640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897400640 1 -- 172.21.15.98:0/3060234011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8981f72c0 con 0x7fb8980b1e10 2024-05-27T00:33:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897e00640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8981516b0 0x7fb898145640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:33:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb897400640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980b1e10 0x7fb89806d240 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb88801a0b0 tx=0x7fb888019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb88f400640 1 -- 172.21.15.98:0/3060234011 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880b0040 con 0x7fb8980b1e10 2024-05-27T00:33:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb88f400640 1 -- 172.21.15.98:0/3060234011 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8880b9070 con 0x7fb8980b1e10 2024-05-27T00:33:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb89e200640 1 -- 172.21.15.98:0/3060234011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8981f75a0 con 0x7fb8980b1e10 2024-05-27T00:33:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb88f400640 1 -- 172.21.15.98:0/3060234011 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880b4650 con 0x7fb8980b1e10 2024-05-27T00:33:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb89e200640 1 -- 172.21.15.98:0/3060234011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8981f7b60 con 0x7fb8980b1e10 2024-05-27T00:33:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.922+0000 7fb89e200640 1 -- 172.21.15.98:0/3060234011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8981f7da0 con 0x7fb8980b1e10 2024-05-27T00:33:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.926+0000 7fb88f400640 1 -- 172.21.15.98:0/3060234011 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb8880b47f0 con 0x7fb8980b1e10 2024-05-27T00:33:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.926+0000 7fb88f400640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb8780678f0 0x7fb878069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.926+0000 7fb88f400640 1 -- 172.21.15.98:0/3060234011 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fb88811ff00 con 0x7fb8980b1e10 2024-05-27T00:33:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.926+0000 7fb896a00640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb8780678f0 0x7fb878069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.926+0000 7fb896a00640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb8780678f0 0x7fb878069db0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb898162180 tx=0x7fb884009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:37.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:37.930+0000 7fb88f400640 1 -- 172.21.15.98:0/3060234011 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fb888016070 con 0x7fb8980b1e10 2024-05-27T00:33:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 -- 172.21.15.98:0/3060234011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb8981f3930 con 0x7fb8980b1e10 2024-05-27T00:33:38.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb88f400640 1 -- 172.21.15.98:0/3060234011 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7fb8880f05b0 con 0x7fb8980b1e10 2024-05-27T00:33:38.092 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-05-27T00:33:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 -- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb8780678f0 msgr2=0x7fb878069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb8780678f0 0x7fb878069db0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb898162180 tx=0x7fb884009040 comp rx=0 tx=0).stop 2024-05-27T00:33:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 -- 172.21.15.98:0/3060234011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980b1e10 msgr2=0x7fb89806d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980b1e10 0x7fb89806d240 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb88801a0b0 tx=0x7fb888019220 comp rx=0 tx=0).stop 2024-05-27T00:33:38.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 -- 172.21.15.98:0/3060234011 shutdown_connections 2024-05-27T00:33:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fb8780678f0 0x7fb878069db0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8981516b0 0x7fb898145640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8980c2b10 0x7fb8981f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 --2- 172.21.15.98:0/3060234011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8980b1e10 0x7fb89806d240 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 -- 172.21.15.98:0/3060234011 >> 172.21.15.98:0/3060234011 conn(0x7fb8980cbf00 msgr2=0x7fb89814f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 -- 172.21.15.98:0/3060234011 shutdown_connections 2024-05-27T00:33:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.086+0000 7fb89e200640 1 -- 172.21.15.98:0/3060234011 wait complete. 2024-05-27T00:33:38.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-27T00:33:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b8600640 1 Processor -- start 2024-05-27T00:33:38.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b8600640 1 -- start start 2024-05-27T00:33:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b8600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b01519d0 0x7fd3b00c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b8600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fd3b00c15c0 0x7fd3b01578b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b8600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3b00c84a0 con 0x7fd3b00c11a0 2024-05-27T00:33:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b8600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fd3b0070d80 con 0x7fd3b00c15c0 2024-05-27T00:33:38.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b8600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b0071900 con 0x7fd3b01519d0 2024-05-27T00:33:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b5200640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd3b00c11a0 0x7fd3b006ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56892/0 (socket says 172.21.15.98:56892) 2024-05-27T00:33:38.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b5200640 1 -- 172.21.15.98:0/857983156 learned_addr learned my addr 172.21.15.98:0/857983156 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:38.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b6600640 1 --2- 172.21.15.98:0/857983156 >> v2:172.21.15.204:3300/0 conn(0x7fd3b00c15c0 0x7fd3b01578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:38.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b5c00640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b01519d0 0x7fd3b00c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:38.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132774430 0 0) 0x7fd3b00c84a0 con 0x7fd3b00c11a0 2024-05-27T00:33:38.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd398003530 con 0x7fd3b00c11a0 2024-05-27T00:33:38.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1549295097 0 0) 0x7fd398003530 con 0x7fd3b00c11a0 2024-05-27T00:33:38.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b01519d0 msgr2=0x7fd3b00c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:38.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b01519d0 0x7fd3b00c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 >> v2:172.21.15.204:3300/0 conn(0x7fd3b00c15c0 msgr2=0x7fd3b01578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:38.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 --2- 172.21.15.98:0/857983156 >> v2:172.21.15.204:3300/0 conn(0x7fd3b00c15c0 0x7fd3b01578b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b0070fd0 con 0x7fd3b00c11a0 2024-05-27T00:33:38.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fd39c001680 con 0x7fd3b00c11a0 2024-05-27T00:33:38.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 >> v1:172.21.15.98:6789/0 conn(0x7fd3b00c11a0 legacy=0x7fd3b006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:38.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3980039c0 0x7fd398003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:38.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd398004810 0x7fd398004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:38.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b01519d0 0x7fd3980051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:38.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b0070d80 con 0x7fd3980039c0 2024-05-27T00:33:38.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b0071900 con 0x7fd398004810 2024-05-27T00:33:38.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd39800b9d0 con 0x7fd3b01519d0 2024-05-27T00:33:38.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b5c00640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3980039c0 0x7fd398003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:38.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b6600640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b01519d0 0x7fd3980051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:38.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b5200640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd398004810 0x7fd398004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:38.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b5c00640 1 -- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b01519d0 msgr2=0x7fd3980051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:38.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b5c00640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b01519d0 0x7fd3980051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b5c00640 1 -- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd398004810 msgr2=0x7fd398004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:38.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b5c00640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd398004810 0x7fd398004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b5c00640 1 -- 172.21.15.98:0/857983156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd398003530 con 0x7fd3980039c0 2024-05-27T00:33:38.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.246+0000 7fd3b6600640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b01519d0 0x7fd3980051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:38.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3b5c00640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3980039c0 0x7fd398003de0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009ae0 tx=0x7fd3a401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3039ba5b7b406dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:38.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40096b0 con 0x7fd3980039c0 2024-05-27T00:33:38.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a40a6480 con 0x7fd3980039c0 2024-05-27T00:33:38.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3afe00640 1 -- 172.21.15.98:0/857983156 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40a6870 con 0x7fd3980039c0 2024-05-27T00:33:38.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3b8600640 1 -- 172.21.15.98:0/857983156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3980039c0 msgr2=0x7fd398003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3b8600640 1 --2- 172.21.15.98:0/857983156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3980039c0 0x7fd398003de0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009ae0 tx=0x7fd3a401ca70 comp rx=0 tx=0).stop 2024-05-27T00:33:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3b6600640 1 -- 172.21.15.98:0/857983156 reap_dead start 2024-05-27T00:33:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3b8600640 1 -- 172.21.15.98:0/857983156 shutdown_connections 2024-05-27T00:33:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3b8600640 1 -- 172.21.15.98:0/857983156 >> 172.21.15.98:0/857983156 conn(0x7fd3b00cc200 msgr2=0x7fd3b00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3b8600640 1 -- 172.21.15.98:0/857983156 shutdown_connections 2024-05-27T00:33:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.250+0000 7fd3b8600640 1 -- 172.21.15.98:0/857983156 wait complete. 2024-05-27T00:33:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b8600640 1 Processor -- start 2024-05-27T00:33:38.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b8600640 1 -- start start 2024-05-27T00:33:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b8600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3980039c0 0x7fd3b01552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b8600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b014f460 0x7fd3b00c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:38.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b8600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b014f860 0x7fd3b00c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b8600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b006f110 con 0x7fd3b014f860 2024-05-27T00:33:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b8600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b00c8220 con 0x7fd3980039c0 2024-05-27T00:33:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b8600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b006f290 con 0x7fd3b014f460 2024-05-27T00:33:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b6600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b014f860 0x7fd3b00c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b6600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b014f860 0x7fd3b00c46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59740/0 (socket says 172.21.15.98:59740) 2024-05-27T00:33:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b6600640 1 -- 172.21.15.98:0/224321933 learned_addr learned my addr 172.21.15.98:0/224321933 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:38.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b5200640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b014f460 0x7fd3b00c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b5c00640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3980039c0 0x7fd3b01552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b6600640 1 -- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b014f460 msgr2=0x7fd3b00c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b6600640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b014f460 0x7fd3b00c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b6600640 1 -- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3980039c0 msgr2=0x7fd3b01552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b6600640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3980039c0 0x7fd3b01552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b6600640 1 -- 172.21.15.98:0/224321933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b01450b0 con 0x7fd3b014f860 2024-05-27T00:33:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b6600640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b014f860 0x7fd3b00c46f0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd3b006dff0 tx=0x7fd3a8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:38.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3ad600640 1 -- 172.21.15.98:0/224321933 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a801a030 con 0x7fd3b014f860 2024-05-27T00:33:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3ad600640 1 -- 172.21.15.98:0/224321933 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a8010da0 con 0x7fd3b014f860 2024-05-27T00:33:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b8600640 1 -- 172.21.15.98:0/224321933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b01508d0 con 0x7fd3b014f860 2024-05-27T00:33:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3ad600640 1 -- 172.21.15.98:0/224321933 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8003be0 con 0x7fd3b014f860 2024-05-27T00:33:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.254+0000 7fd3b8600640 1 -- 172.21.15.98:0/224321933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b0150b70 con 0x7fd3b014f860 2024-05-27T00:33:38.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.258+0000 7fd3b8600640 1 -- 172.21.15.98:0/224321933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b0075cb0 con 0x7fd3b014f860 2024-05-27T00:33:38.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.258+0000 7fd3ad600640 1 -- 172.21.15.98:0/224321933 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd3a801d070 con 0x7fd3b014f860 2024-05-27T00:33:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.258+0000 7fd3ad600640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3900678f0 0x7fd390069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.258+0000 7fd3ad600640 1 -- 172.21.15.98:0/224321933 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fd3a80866d0 con 0x7fd3b014f860 2024-05-27T00:33:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.258+0000 7fd3b5c00640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3900678f0 0x7fd390069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.258+0000 7fd3b5c00640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3900678f0 0x7fd390069db0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40ad000 tx=0x7fd3a40b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.262+0000 7fd3ad600640 1 -- 172.21.15.98:0/224321933 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fd3a8056d90 con 0x7fd3b014f860 2024-05-27T00:33:38.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 -- 172.21.15.98:0/224321933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd3b0145630 con 0x7fd3b014f860 2024-05-27T00:33:38.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3ad600640 1 -- 172.21.15.98:0/224321933 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 24 entries v48) ==== 81+0+1462 (secure 0 0 0) 0x7fd3b0145630 con 0x7fd3b014f860 2024-05-27T00:33:38.450 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6834/3070334683 2024-05-28T00:32:10.313051+0000 2024-05-27T00:33:38.450 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6836/1802902856 2024-05-28T00:30:40.295945+0000 2024-05-27T00:33:38.451 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6834/2148124202 2024-05-28T00:28:50.279743+0000 2024-05-27T00:33:38.451 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6832/2148124202 2024-05-28T00:28:50.279743+0000 2024-05-27T00:33:38.451 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6837/2940211164 2024-05-28T00:27:55.271086+0000 2024-05-27T00:33:38.451 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6836/2940211164 2024-05-28T00:27:55.271086+0000 2024-05-27T00:33:38.451 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6834/1223646158 2024-05-28T00:27:00.259848+0000 2024-05-27T00:33:38.451 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6837/518550803 2024-05-28T00:24:50.239991+0000 2024-05-27T00:33:38.451 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6836/3824013488 2024-05-28T00:16:25.159349+0000 2024-05-27T00:33:38.451 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6832/3070334683 2024-05-28T00:32:10.313051+0000 2024-05-27T00:33:38.452 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6832/3247908011 2024-05-28T00:21:25.205263+0000 2024-05-27T00:33:38.452 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6834/4290077910 2024-05-28T00:23:40.226716+0000 2024-05-27T00:33:38.452 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6832/4290077910 2024-05-28T00:23:40.226716+0000 2024-05-27T00:33:38.452 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6837/1802902856 2024-05-28T00:30:40.295945+0000 2024-05-27T00:33:38.452 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6832/3903574515 2024-05-28T00:18:25.179026+0000 2024-05-27T00:33:38.452 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6837/3824013488 2024-05-28T00:16:25.159349+0000 2024-05-27T00:33:38.452 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6836/1781395818 2024-05-28T00:19:35.189766+0000 2024-05-27T00:33:38.453 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6834/3903574515 2024-05-28T00:18:25.179026+0000 2024-05-27T00:33:38.453 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6837/1781395818 2024-05-28T00:19:35.189766+0000 2024-05-27T00:33:38.453 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6834/3247908011 2024-05-28T00:21:25.205263+0000 2024-05-27T00:33:38.453 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6832/1223646158 2024-05-28T00:27:00.259848+0000 2024-05-27T00:33:38.453 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6836/1698253956 2024-05-28T00:22:20.213891+0000 2024-05-27T00:33:38.453 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6836/518550803 2024-05-28T00:24:50.239991+0000 2024-05-27T00:33:38.453 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6837/1698253956 2024-05-28T00:22:20.213891+0000 2024-05-27T00:33:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:listed 24 entries 2024-05-27T00:33:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 -- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3900678f0 msgr2=0x7fd390069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3900678f0 0x7fd390069db0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40ad000 tx=0x7fd3a40b2040 comp rx=0 tx=0).stop 2024-05-27T00:33:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 -- 172.21.15.98:0/224321933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b014f860 msgr2=0x7fd3b00c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b014f860 0x7fd3b00c46f0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd3b006dff0 tx=0x7fd3a8010590 comp rx=0 tx=0).stop 2024-05-27T00:33:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 -- 172.21.15.98:0/224321933 shutdown_connections 2024-05-27T00:33:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3b014f860 0x7fd3b00c46f0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3b014f460 0x7fd3b00c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fd3900678f0 0x7fd390069db0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 --2- 172.21.15.98:0/224321933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3980039c0 0x7fd3b01552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 -- 172.21.15.98:0/224321933 >> 172.21.15.98:0/224321933 conn(0x7fd3b00cc200 msgr2=0x7fd3b00c5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 -- 172.21.15.98:0/224321933 shutdown_connections 2024-05-27T00:33:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:38.446+0000 7fd3b8600640 1 -- 172.21.15.98:0/224321933 wait complete. 2024-05-27T00:33:38.472 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-27T00:33:38.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:33:38.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-27T00:33:40.128 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:33:40.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:33:40.129 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-27T00:33:40.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:33:40.149 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-05-27T00:33:40.205 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-27T00:33:40.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:33:40.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T00:33:40.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T00:33:40.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-05-27T00:33:40.464 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-27T00:33:40.474 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-27T00:33:40.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:33:40.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.610+0000 7f0b34e00640 1 Processor -- start 2024-05-27T00:33:40.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.610+0000 7f0b34e00640 1 -- start start 2024-05-27T00:33:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.614+0000 7f0b34e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b300731f0 0x7f0b300c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.614+0000 7f0b34e00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b300c17d0 con 0x7f0b300735d0 2024-05-27T00:33:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.614+0000 7f0b34e00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b30070e10 con 0x7f0b300c79f0 2024-05-27T00:33:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.614+0000 7f0b34e00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0b30071990 con 0x7f0b300731f0 2024-05-27T00:33:40.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.614+0000 7f0b2de00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b300731f0 0x7f0b300c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.614+0000 7f0b25400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0b300735d0 0x7f0b30151430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56900/0 (socket says 172.21.15.98:56900) 2024-05-27T00:33:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.614+0000 7f0b25400640 1 -- 172.21.15.98:0/1810217878 learned_addr learned my addr 172.21.15.98:0/1810217878 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.614+0000 7f0b2de00640 1 -- 172.21.15.98:0/1810217878 >> v1:172.21.15.204:6789/0 conn(0x7f0b300c79f0 legacy=0x7f0b30157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:40.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.614+0000 7f0b2de00640 1 -- 172.21.15.98:0/1810217878 >> v1:172.21.15.98:6789/0 conn(0x7f0b300735d0 legacy=0x7f0b30151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:40.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.614+0000 7f0b2de00640 1 -- 172.21.15.98:0/1810217878 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b300710f0 con 0x7f0b300731f0 2024-05-27T00:33:40.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b2de00640 1 --2- 172.21.15.98:0/1810217878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300731f0 0x7f0b300c0eb0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0b30071b10 tx=0x7f0b2001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cbe11da3248354 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:40.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b2d400640 1 -- 172.21.15.98:0/1810217878 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b200b2070 con 0x7f0b300731f0 2024-05-27T00:33:40.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b2d400640 1 -- 172.21.15.98:0/1810217878 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b200a59e0 con 0x7f0b300731f0 2024-05-27T00:33:40.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b2d400640 1 -- 172.21.15.98:0/1810217878 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b200a5db0 con 0x7f0b300731f0 2024-05-27T00:33:40.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 -- 172.21.15.98:0/1810217878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300731f0 msgr2=0x7f0b300c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:40.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 --2- 172.21.15.98:0/1810217878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300731f0 0x7f0b300c0eb0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0b30071b10 tx=0x7f0b2001c920 comp rx=0 tx=0).stop 2024-05-27T00:33:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 -- 172.21.15.98:0/1810217878 shutdown_connections 2024-05-27T00:33:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 --2- 172.21.15.98:0/1810217878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300731f0 0x7f0b300c0eb0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 -- 172.21.15.98:0/1810217878 >> 172.21.15.98:0/1810217878 conn(0x7f0b30067530 msgr2=0x7f0b3006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 -- 172.21.15.98:0/1810217878 shutdown_connections 2024-05-27T00:33:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 -- 172.21.15.98:0/1810217878 wait complete. 2024-05-27T00:33:40.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 Processor -- start 2024-05-27T00:33:40.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 -- start start 2024-05-27T00:33:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b300731f0 0x7f0b3014beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b300735d0 0x7f0b30146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300c79f0 0x7f0b300c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b300c1570 con 0x7f0b300c79f0 2024-05-27T00:33:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0b300c13f0 con 0x7f0b300731f0 2024-05-27T00:33:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b34e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0b30151b40 con 0x7f0b300735d0 2024-05-27T00:33:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b2e800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300c79f0 0x7f0b300c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b2e800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300c79f0 0x7f0b300c9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59762/0 (socket says 172.21.15.98:59762) 2024-05-27T00:33:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b2e800640 1 -- 172.21.15.98:0/3613320317 learned_addr learned my addr 172.21.15.98:0/3613320317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.618+0000 7f0b2de00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b300731f0 0x7f0b3014beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b25400640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b300735d0 0x7f0b30146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b2e800640 1 -- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b300735d0 msgr2=0x7f0b30146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b2e800640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b300735d0 0x7f0b30146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b2e800640 1 -- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b300731f0 msgr2=0x7f0b3014beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b2e800640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b300731f0 0x7f0b3014beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b2e800640 1 -- 172.21.15.98:0/3613320317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3014c3f0 con 0x7f0b300c79f0 2024-05-27T00:33:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b2de00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b300731f0 0x7f0b3014beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b2e800640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300c79f0 0x7f0b300c9a40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800c340 tx=0x7f0b2800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b26a00640 1 -- 172.21.15.98:0/3613320317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28005220 con 0x7f0b300c79f0 2024-05-27T00:33:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b26a00640 1 -- 172.21.15.98:0/3613320317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b280053c0 con 0x7f0b300c79f0 2024-05-27T00:33:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b26a00640 1 -- 172.21.15.98:0/3613320317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28007270 con 0x7f0b300c79f0 2024-05-27T00:33:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b34e00640 1 -- 172.21.15.98:0/3613320317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b3014dc60 con 0x7f0b300c79f0 2024-05-27T00:33:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b34e00640 1 -- 172.21.15.98:0/3613320317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b301f3e70 con 0x7f0b300c79f0 2024-05-27T00:33:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b34e00640 1 -- 172.21.15.98:0/3613320317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b30075cb0 con 0x7f0b300c79f0 2024-05-27T00:33:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b26a00640 1 -- 172.21.15.98:0/3613320317 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0b28019020 con 0x7f0b300c79f0 2024-05-27T00:33:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b26a00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b0c0678c0 0x7f0b0c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b26a00640 1 -- 172.21.15.98:0/3613320317 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f0b280854d0 con 0x7f0b300c79f0 2024-05-27T00:33:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.622+0000 7f0b2de00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b0c0678c0 0x7f0b0c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.626+0000 7f0b2de00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b0c0678c0 0x7f0b0c069d80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0b200af000 tx=0x7f0b200a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.630+0000 7f0b26a00640 1 -- 172.21.15.98:0/3613320317 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f0b28055490 con 0x7f0b300c79f0 2024-05-27T00:33:40.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.786+0000 7f0b34e00640 1 -- 172.21.15.98:0/3613320317 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0b3014f5e0 con 0x7f0b0c0678c0 2024-05-27T00:33:40.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.806+0000 7f0b26a00640 1 -- 172.21.15.98:0/3613320317 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257325 (secure 0 0 0) 0x7f0b3014f5e0 con 0x7f0b0c0678c0 2024-05-27T00:33:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:33:40.810 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:33:40.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.806+0000 7f0b34e00640 1 -- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b0c0678c0 msgr2=0x7f0b0c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:40.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.806+0000 7f0b34e00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b0c0678c0 0x7f0b0c069d80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0b200af000 tx=0x7f0b200a7070 comp rx=0 tx=0).stop 2024-05-27T00:33:40.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.806+0000 7f0b34e00640 1 -- 172.21.15.98:0/3613320317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300c79f0 msgr2=0x7f0b300c9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:40.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.806+0000 7f0b34e00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300c79f0 0x7f0b300c9a40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800c340 tx=0x7f0b2800c800 comp rx=0 tx=0).stop 2024-05-27T00:33:40.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.810+0000 7f0b34e00640 1 -- 172.21.15.98:0/3613320317 shutdown_connections 2024-05-27T00:33:40.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.810+0000 7f0b34e00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b300c79f0 0x7f0b300c9a40 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:40.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.810+0000 7f0b34e00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b300735d0 0x7f0b30146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:40.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.810+0000 7f0b34e00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f0b0c0678c0 0x7f0b0c069d80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:40.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.810+0000 7f0b34e00640 1 --2- 172.21.15.98:0/3613320317 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b300731f0 0x7f0b3014beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:40.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.810+0000 7f0b34e00640 1 -- 172.21.15.98:0/3613320317 >> 172.21.15.98:0/3613320317 conn(0x7f0b30067530 msgr2=0x7f0b30157970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:40.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.810+0000 7f0b34e00640 1 -- 172.21.15.98:0/3613320317 shutdown_connections 2024-05-27T00:33:40.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:40.810+0000 7f0b34e00640 1 -- 172.21.15.98:0/3613320317 wait complete. 2024-05-27T00:33:40.828 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":595,"stamp":"2024-05-27T00:33:39.542504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1956708577,"num_objects":41115,"num_object_clones":18298,"num_object_copies":82230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41115,"num_whiteouts":12489,"num_read":13891,"num_read_kb":6221406,"num_write":126437,"num_write_kb":3703202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1834,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130934,"ondisk_log_size":130934,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5016024,"kb_used_data":3916120,"kb_used_omap":12,"kb_used_meta":1099827,"kb_avail":749958696,"statfs":{"total":773094113280,"available":767957704704,"internally_reserved":0,"allocated":4010106880,"data_stored":3917368218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1126223410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":63167789,"num_objects":639,"num_object_clones":225,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":203,"num_read":456,"num_read_kb":49335,"num_write":1016,"num_write_kb":62825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004786"},"pg_stats":[{"pgid":"3.3f","version":"48'1517","reported_seq":3393,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.925854+0000","last_change":"2024-05-27T00:17:54.336701+0000","last_active":"2024-05-27T00:33:33.925854+0000","last_peered":"2024-05-27T00:33:33.925854+0000","last_clean":"2024-05-27T00:33:33.925854+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:33:33.925854+0000","last_undegraded":"2024-05-27T00:33:33.925854+0000","last_fullsized":"2024-05-27T00:33:33.925854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:39.250119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027619028,"stat_sum":{"num_bytes":17373324,"num_objects":575,"num_object_clones":268,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":196,"num_read":49,"num_read_kb":31548,"num_write":1364,"num_write_kb":18943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"48'1504","reported_seq":3403,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.895060+0000","last_change":"2024-05-27T00:17:54.490184+0000","last_active":"2024-05-27T00:33:30.895060+0000","last_peered":"2024-05-27T00:33:30.895060+0000","last_clean":"2024-05-27T00:33:30.895060+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:33:30.895060+0000","last_undegraded":"2024-05-27T00:33:30.895060+0000","last_fullsized":"2024-05-27T00:33:30.895060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1504,"log_dups_size":0,"ondisk_log_size":1504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:05:48.629948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015469564,"stat_sum":{"num_bytes":18053131,"num_objects":546,"num_object_clones":255,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":169,"num_read":59,"num_read_kb":35851,"num_write":1378,"num_write_kb":19996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"48'1616","reported_seq":3466,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.917802+0000","last_change":"2024-05-27T00:17:54.493058+0000","last_active":"2024-05-27T00:33:33.917802+0000","last_peered":"2024-05-27T00:33:33.917802+0000","last_clean":"2024-05-27T00:33:33.917802+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:33:33.917802+0000","last_undegraded":"2024-05-27T00:33:33.917802+0000","last_fullsized":"2024-05-27T00:33:33.917802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:59:53.965584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018507847000000001,"stat_sum":{"num_bytes":15200236,"num_objects":629,"num_object_clones":298,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":212,"num_read":53,"num_read_kb":23374,"num_write":1443,"num_write_kb":25758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"48'1615","reported_seq":3479,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.883919+0000","last_change":"2024-05-27T00:17:54.510484+0000","last_active":"2024-05-27T00:33:30.883919+0000","last_peered":"2024-05-27T00:33:30.883919+0000","last_clean":"2024-05-27T00:33:30.883919+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:33:30.883919+0000","last_undegraded":"2024-05-27T00:33:30.883919+0000","last_fullsized":"2024-05-27T00:33:30.883919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:39.250119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034338709000000002,"stat_sum":{"num_bytes":10173730,"num_objects":618,"num_object_clones":288,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":202,"num_read":56,"num_read_kb":12442,"num_write":1451,"num_write_kb":16900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"48'1701","reported_seq":3542,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.395661+0000","last_change":"2024-05-27T00:17:54.331579+0000","last_active":"2024-05-27T00:33:33.395661+0000","last_peered":"2024-05-27T00:33:33.395661+0000","last_clean":"2024-05-27T00:33:33.395661+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:33:33.395661+0000","last_undegraded":"2024-05-27T00:33:33.395661+0000","last_fullsized":"2024-05-27T00:33:33.395661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1701,"log_dups_size":0,"ondisk_log_size":1701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:35:37.983576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026116615999999999,"stat_sum":{"num_bytes":20047775,"num_objects":666,"num_object_clones":311,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":219,"num_read":58,"num_read_kb":29919,"num_write":1529,"num_write_kb":21884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"48'1457","reported_seq":3333,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.926396+0000","last_change":"2024-05-27T00:17:54.507830+0000","last_active":"2024-05-27T00:33:33.926396+0000","last_peered":"2024-05-27T00:33:33.926396+0000","last_clean":"2024-05-27T00:33:33.926396+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:33:33.926396+0000","last_undegraded":"2024-05-27T00:33:33.926396+0000","last_fullsized":"2024-05-27T00:33:33.926396+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1457,"log_dups_size":0,"ondisk_log_size":1457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:48.582600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032595461999999999,"stat_sum":{"num_bytes":14545166,"num_objects":545,"num_object_clones":256,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":181,"num_read":38,"num_read_kb":8264,"num_write":1295,"num_write_kb":15901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"48'1575","reported_seq":3443,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.871914+0000","last_change":"2024-05-27T00:17:54.335843+0000","last_active":"2024-05-27T00:33:29.871914+0000","last_peered":"2024-05-27T00:33:29.871914+0000","last_clean":"2024-05-27T00:33:29.871914+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:33:29.871914+0000","last_undegraded":"2024-05-27T00:33:29.871914+0000","last_fullsized":"2024-05-27T00:33:29.871914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1575,"log_dups_size":0,"ondisk_log_size":1575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:27:21.913063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029344160000000001,"stat_sum":{"num_bytes":19339722,"num_objects":602,"num_object_clones":277,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":190,"num_read":51,"num_read_kb":28747,"num_write":1431,"num_write_kb":22126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"48'1515","reported_seq":3386,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.468342+0000","last_change":"2024-05-27T00:17:54.485261+0000","last_active":"2024-05-27T00:33:30.468342+0000","last_peered":"2024-05-27T00:33:30.468342+0000","last_clean":"2024-05-27T00:33:30.468342+0000","last_became_active":"2024-05-27T00:16:06.280009+0000","last_became_peered":"2024-05-27T00:16:06.280009+0000","last_unstale":"2024-05-27T00:33:30.468342+0000","last_undegraded":"2024-05-27T00:33:30.468342+0000","last_fullsized":"2024-05-27T00:33:30.468342+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:14:46.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012011287000000001,"stat_sum":{"num_bytes":35835659,"num_objects":577,"num_object_clones":269,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":179,"num_read":52,"num_read_kb":60147,"num_write":1378,"num_write_kb":38796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.37","version":"48'1620","reported_seq":3476,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.395963+0000","last_change":"2024-05-27T00:17:54.485867+0000","last_active":"2024-05-27T00:33:33.395963+0000","last_peered":"2024-05-27T00:33:33.395963+0000","last_clean":"2024-05-27T00:33:33.395963+0000","last_became_active":"2024-05-27T00:16:06.280544+0000","last_became_peered":"2024-05-27T00:16:06.280544+0000","last_unstale":"2024-05-27T00:33:33.395963+0000","last_undegraded":"2024-05-27T00:33:33.395963+0000","last_fullsized":"2024-05-27T00:33:33.395963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:20:13.063312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096681779999999995,"stat_sum":{"num_bytes":20135711,"num_objects":629,"num_object_clones":292,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":198,"num_read":60,"num_read_kb":29133,"num_write":1469,"num_write_kb":25955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"48'1514","reported_seq":3391,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.450019+0000","last_change":"2024-05-27T00:17:54.337723+0000","last_active":"2024-05-27T00:33:30.450019+0000","last_peered":"2024-05-27T00:33:30.450019+0000","last_clean":"2024-05-27T00:33:30.450019+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:33:30.450019+0000","last_undegraded":"2024-05-27T00:33:30.450019+0000","last_fullsized":"2024-05-27T00:33:30.450019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1514,"log_dups_size":0,"ondisk_log_size":1514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:55:01.773614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027993297,"stat_sum":{"num_bytes":14778263,"num_objects":589,"num_object_clones":272,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":186,"num_read":55,"num_read_kb":22038,"num_write":1380,"num_write_kb":16492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"48'1627","reported_seq":3458,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.895588+0000","last_change":"2024-05-27T00:17:54.335171+0000","last_active":"2024-05-27T00:33:33.895588+0000","last_peered":"2024-05-27T00:33:33.895588+0000","last_clean":"2024-05-27T00:33:33.895588+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:33:33.895588+0000","last_undegraded":"2024-05-27T00:33:33.895588+0000","last_fullsized":"2024-05-27T00:33:33.895588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1627,"log_dups_size":0,"ondisk_log_size":1627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:11:26.796581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027428447000000002,"stat_sum":{"num_bytes":10658234,"num_objects":659,"num_object_clones":308,"num_object_copies":1318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":204,"num_read":45,"num_read_kb":12410,"num_write":1475,"num_write_kb":11557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"48'1609","reported_seq":3434,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.363658+0000","last_change":"2024-05-27T00:17:54.504751+0000","last_active":"2024-05-27T00:33:33.363658+0000","last_peered":"2024-05-27T00:33:33.363658+0000","last_clean":"2024-05-27T00:33:33.363658+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:33:33.363658+0000","last_undegraded":"2024-05-27T00:33:33.363658+0000","last_fullsized":"2024-05-27T00:33:33.363658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:51:55.125064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021724307000000002,"stat_sum":{"num_bytes":19374715,"num_objects":636,"num_object_clones":300,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":200,"num_read":31,"num_read_kb":14211,"num_write":1467,"num_write_kb":31877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"48'1547","reported_seq":3401,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.417979+0000","last_change":"2024-05-27T00:17:54.309597+0000","last_active":"2024-05-27T00:33:30.417979+0000","last_peered":"2024-05-27T00:33:30.417979+0000","last_clean":"2024-05-27T00:33:30.417979+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:33:30.417979+0000","last_undegraded":"2024-05-27T00:33:30.417979+0000","last_fullsized":"2024-05-27T00:33:30.417979+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1547,"log_dups_size":0,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:01:46.216541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089222539999999993,"stat_sum":{"num_bytes":26519812,"num_objects":619,"num_object_clones":291,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":212,"num_read":51,"num_read_kb":32873,"num_write":1364,"num_write_kb":36569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"48'1587","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.412049+0000","last_change":"2024-05-27T00:17:54.491142+0000","last_active":"2024-05-27T00:33:30.412049+0000","last_peered":"2024-05-27T00:33:30.412049+0000","last_clean":"2024-05-27T00:33:30.412049+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:33:30.412049+0000","last_undegraded":"2024-05-27T00:33:30.412049+0000","last_fullsized":"2024-05-27T00:33:30.412049+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1587,"log_dups_size":0,"ondisk_log_size":1587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:43:35.726386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016842227000000001,"stat_sum":{"num_bytes":10308290,"num_objects":608,"num_object_clones":288,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":204,"num_read":44,"num_read_kb":16487,"num_write":1426,"num_write_kb":16803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"48'1620","reported_seq":3487,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.955725+0000","last_change":"2024-05-27T00:17:54.491503+0000","last_active":"2024-05-27T00:33:30.955725+0000","last_peered":"2024-05-27T00:33:30.955725+0000","last_clean":"2024-05-27T00:33:30.955725+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:33:30.955725+0000","last_undegraded":"2024-05-27T00:33:30.955725+0000","last_fullsized":"2024-05-27T00:33:30.955725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:33:06.746690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016821815,"stat_sum":{"num_bytes":22996149,"num_objects":605,"num_object_clones":283,"num_object_copies":1210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":605,"num_whiteouts":212,"num_read":56,"num_read_kb":39458,"num_write":1456,"num_write_kb":27752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"48'1589","reported_seq":3458,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.899422+0000","last_change":"2024-05-27T00:17:54.506224+0000","last_active":"2024-05-27T00:33:30.899422+0000","last_peered":"2024-05-27T00:33:30.899422+0000","last_clean":"2024-05-27T00:33:30.899422+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:33:30.899422+0000","last_undegraded":"2024-05-27T00:33:30.899422+0000","last_fullsized":"2024-05-27T00:33:30.899422+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1589,"log_dups_size":0,"ondisk_log_size":1589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:44.778493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.031092022,"stat_sum":{"num_bytes":14532615,"num_objects":603,"num_object_clones":288,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":201,"num_read":63,"num_read_kb":16619,"num_write":1426,"num_write_kb":20413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"48'1709","reported_seq":3577,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.368113+0000","last_change":"2024-05-27T00:17:54.511207+0000","last_active":"2024-05-27T00:33:33.368113+0000","last_peered":"2024-05-27T00:33:33.368113+0000","last_clean":"2024-05-27T00:33:33.368113+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:33:33.368113+0000","last_undegraded":"2024-05-27T00:33:33.368113+0000","last_fullsized":"2024-05-27T00:33:33.368113+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1709,"log_dups_size":0,"ondisk_log_size":1709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:03:55.531180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034311531999999999,"stat_sum":{"num_bytes":28217639,"num_objects":653,"num_object_clones":304,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":203,"num_read":78,"num_read_kb":63526,"num_write":1561,"num_write_kb":34080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"48'1512","reported_seq":3382,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.394396+0000","last_change":"2024-05-27T00:17:54.487572+0000","last_active":"2024-05-27T00:33:33.394396+0000","last_peered":"2024-05-27T00:33:33.394396+0000","last_clean":"2024-05-27T00:33:33.394396+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:33:33.394396+0000","last_undegraded":"2024-05-27T00:33:33.394396+0000","last_fullsized":"2024-05-27T00:33:33.394396+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:07:28.224179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010480461999999999,"stat_sum":{"num_bytes":6654877,"num_objects":588,"num_object_clones":271,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":191,"num_read":47,"num_read_kb":139,"num_write":1368,"num_write_kb":8104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"48'1671","reported_seq":3510,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.397671+0000","last_change":"2024-05-27T00:17:54.337799+0000","last_active":"2024-05-27T00:33:33.397671+0000","last_peered":"2024-05-27T00:33:33.397671+0000","last_clean":"2024-05-27T00:33:33.397671+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:33:33.397671+0000","last_undegraded":"2024-05-27T00:33:33.397671+0000","last_fullsized":"2024-05-27T00:33:33.397671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:13.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029916169999999999,"stat_sum":{"num_bytes":16039418,"num_objects":654,"num_object_clones":310,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":208,"num_read":54,"num_read_kb":28815,"num_write":1504,"num_write_kb":20996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"48'1603","reported_seq":3487,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.440010+0000","last_change":"2024-05-27T00:17:54.335854+0000","last_active":"2024-05-27T00:33:30.440010+0000","last_peered":"2024-05-27T00:33:30.440010+0000","last_clean":"2024-05-27T00:33:30.440010+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:33:30.440010+0000","last_undegraded":"2024-05-27T00:33:30.440010+0000","last_fullsized":"2024-05-27T00:33:30.440010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:13:02.845565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027334256000000001,"stat_sum":{"num_bytes":31314512,"num_objects":608,"num_object_clones":281,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":187,"num_read":70,"num_read_kb":56922,"num_write":1458,"num_write_kb":40262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"48'1614","reported_seq":3479,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.930688+0000","last_change":"2024-05-27T00:17:54.493382+0000","last_active":"2024-05-27T00:33:33.930688+0000","last_peered":"2024-05-27T00:33:33.930688+0000","last_clean":"2024-05-27T00:33:33.930688+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:33:33.930688+0000","last_undegraded":"2024-05-27T00:33:33.930688+0000","last_fullsized":"2024-05-27T00:33:33.930688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1614,"log_dups_size":0,"ondisk_log_size":1614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:11:52.944101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017108005999999999,"stat_sum":{"num_bytes":14791903,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":193,"num_read":59,"num_read_kb":12448,"num_write":1482,"num_write_kb":20672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"48'1678","reported_seq":3512,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.367645+0000","last_change":"2024-05-27T00:17:54.509854+0000","last_active":"2024-05-27T00:33:33.367645+0000","last_peered":"2024-05-27T00:33:33.367645+0000","last_clean":"2024-05-27T00:33:33.367645+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:33:33.367645+0000","last_undegraded":"2024-05-27T00:33:33.367645+0000","last_fullsized":"2024-05-27T00:33:33.367645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1678,"log_dups_size":0,"ondisk_log_size":1678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:13.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033862588999999998,"stat_sum":{"num_bytes":18976923,"num_objects":661,"num_object_clones":315,"num_object_copies":1322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":222,"num_read":54,"num_read_kb":29811,"num_write":1493,"num_write_kb":24680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"48'1529","reported_seq":3405,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.389080+0000","last_change":"2024-05-27T00:17:54.309499+0000","last_active":"2024-05-27T00:33:33.389080+0000","last_peered":"2024-05-27T00:33:33.389080+0000","last_clean":"2024-05-27T00:33:33.389080+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:33:33.389080+0000","last_undegraded":"2024-05-27T00:33:33.389080+0000","last_fullsized":"2024-05-27T00:33:33.389080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1529,"log_dups_size":0,"ondisk_log_size":1529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:32:55.895752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088470660000000007,"stat_sum":{"num_bytes":9565297,"num_objects":574,"num_object_clones":267,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":178,"num_read":48,"num_read_kb":13982,"num_write":1418,"num_write_kb":15615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"48'1569","reported_seq":3463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.893810+0000","last_change":"2024-05-27T00:17:54.509276+0000","last_active":"2024-05-27T00:33:29.893810+0000","last_peered":"2024-05-27T00:33:29.893810+0000","last_clean":"2024-05-27T00:33:29.893810+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:33:29.893810+0000","last_undegraded":"2024-05-27T00:33:29.893810+0000","last_fullsized":"2024-05-27T00:33:29.893810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:55:24.602133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033457270999999997,"stat_sum":{"num_bytes":30464560,"num_objects":585,"num_object_clones":269,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":175,"num_read":68,"num_read_kb":58155,"num_write":1439,"num_write_kb":31950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"48'1566","reported_seq":3425,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.854148+0000","last_change":"2024-05-27T00:17:54.310134+0000","last_active":"2024-05-27T00:33:29.854148+0000","last_peered":"2024-05-27T00:33:29.854148+0000","last_clean":"2024-05-27T00:33:29.854148+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:33:29.854148+0000","last_undegraded":"2024-05-27T00:33:29.854148+0000","last_fullsized":"2024-05-27T00:33:29.854148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1566,"log_dups_size":0,"ondisk_log_size":1566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:37:27.481263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089651499999999999,"stat_sum":{"num_bytes":14362147,"num_objects":603,"num_object_clones":285,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":194,"num_read":49,"num_read_kb":20561,"num_write":1422,"num_write_kb":17027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"48'1667","reported_seq":3529,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.893411+0000","last_change":"2024-05-27T00:17:54.337554+0000","last_active":"2024-05-27T00:33:33.893411+0000","last_peered":"2024-05-27T00:33:33.893411+0000","last_clean":"2024-05-27T00:33:33.893411+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:33:33.893411+0000","last_undegraded":"2024-05-27T00:33:33.893411+0000","last_fullsized":"2024-05-27T00:33:33.893411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1667,"log_dups_size":0,"ondisk_log_size":1667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:02:46.037448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024583906999999999,"stat_sum":{"num_bytes":23325444,"num_objects":624,"num_object_clones":288,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":196,"num_read":55,"num_read_kb":32562,"num_write":1514,"num_write_kb":26070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"48'1526","reported_seq":3399,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.912844+0000","last_change":"2024-05-27T00:17:54.491723+0000","last_active":"2024-05-27T00:33:33.912844+0000","last_peered":"2024-05-27T00:33:33.912844+0000","last_clean":"2024-05-27T00:33:33.912844+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:33:33.912844+0000","last_undegraded":"2024-05-27T00:33:33.912844+0000","last_fullsized":"2024-05-27T00:33:33.912844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:41:31.328434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016722006000000001,"stat_sum":{"num_bytes":10472871,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":180,"num_read":47,"num_read_kb":12352,"num_write":1385,"num_write_kb":12462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"48'1571","reported_seq":3430,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.392636+0000","last_change":"2024-05-27T00:17:54.309792+0000","last_active":"2024-05-27T00:33:33.392636+0000","last_peered":"2024-05-27T00:33:33.392636+0000","last_clean":"2024-05-27T00:33:33.392636+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:33:33.392636+0000","last_undegraded":"2024-05-27T00:33:33.392636+0000","last_fullsized":"2024-05-27T00:33:33.392636+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:03:52.656303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089478649999999993,"stat_sum":{"num_bytes":10328890,"num_objects":604,"num_object_clones":285,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":194,"num_read":49,"num_read_kb":15706,"num_write":1422,"num_write_kb":15432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"48'1703","reported_seq":3521,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.956715+0000","last_change":"2024-05-27T00:17:54.494236+0000","last_active":"2024-05-27T00:33:30.956715+0000","last_peered":"2024-05-27T00:33:30.956715+0000","last_clean":"2024-05-27T00:33:30.956715+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:33:30.956715+0000","last_undegraded":"2024-05-27T00:33:30.956715+0000","last_fullsized":"2024-05-27T00:33:30.956715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1703,"log_dups_size":0,"ondisk_log_size":1703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:17:24.056537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017911486000000001,"stat_sum":{"num_bytes":9846144,"num_objects":668,"num_object_clones":317,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":212,"num_read":40,"num_read_kb":1060,"num_write":1530,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"48'1604","reported_seq":3472,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.396278+0000","last_change":"2024-05-27T00:17:54.510031+0000","last_active":"2024-05-27T00:33:33.396278+0000","last_peered":"2024-05-27T00:33:33.396278+0000","last_clean":"2024-05-27T00:33:33.396278+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:33:33.396278+0000","last_undegraded":"2024-05-27T00:33:33.396278+0000","last_fullsized":"2024-05-27T00:33:33.396278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1604,"log_dups_size":0,"ondisk_log_size":1604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:09.263437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033387499000000001,"stat_sum":{"num_bytes":18975931,"num_objects":601,"num_object_clones":281,"num_object_copies":1202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":187,"num_read":54,"num_read_kb":24794,"num_write":1457,"num_write_kb":20478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"48'1651","reported_seq":3529,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.347644+0000","last_change":"2024-05-27T00:17:54.484908+0000","last_active":"2024-05-27T00:33:30.347644+0000","last_peered":"2024-05-27T00:33:30.347644+0000","last_clean":"2024-05-27T00:33:30.347644+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:33:30.347644+0000","last_undegraded":"2024-05-27T00:33:30.347644+0000","last_fullsized":"2024-05-27T00:33:30.347644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1651,"log_dups_size":0,"ondisk_log_size":1651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:10:39.525108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009527878,"stat_sum":{"num_bytes":27932857,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":194,"num_read":62,"num_read_kb":41489,"num_write":1506,"num_write_kb":37195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"48'1571","reported_seq":3467,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.389518+0000","last_change":"2024-05-27T00:17:54.331060+0000","last_active":"2024-05-27T00:33:33.389518+0000","last_peered":"2024-05-27T00:33:33.389518+0000","last_clean":"2024-05-27T00:33:33.389518+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:33:33.389518+0000","last_undegraded":"2024-05-27T00:33:33.389518+0000","last_fullsized":"2024-05-27T00:33:33.389518+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:05:48.629948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025435675000000001,"stat_sum":{"num_bytes":17932144,"num_objects":570,"num_object_clones":264,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":185,"num_read":65,"num_read_kb":34529,"num_write":1438,"num_write_kb":20974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"48'1632","reported_seq":3479,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.383849+0000","last_change":"2024-05-27T00:17:54.501574+0000","last_active":"2024-05-27T00:33:33.383849+0000","last_peered":"2024-05-27T00:33:33.383849+0000","last_clean":"2024-05-27T00:33:33.383849+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:33:33.383849+0000","last_undegraded":"2024-05-27T00:33:33.383849+0000","last_fullsized":"2024-05-27T00:33:33.383849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1632,"log_dups_size":0,"ondisk_log_size":1632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:16:03.345753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019320113,"stat_sum":{"num_bytes":26864770,"num_objects":637,"num_object_clones":297,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":204,"num_read":50,"num_read_kb":43416,"num_write":1495,"num_write_kb":28253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"48'1097","reported_seq":1312,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.639463+0000","last_change":"2024-05-27T00:15:24.611108+0000","last_active":"2024-05-27T00:33:34.639463+0000","last_peered":"2024-05-27T00:33:34.639463+0000","last_clean":"2024-05-27T00:33:34.639463+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:33:34.639463+0000","last_undegraded":"2024-05-27T00:33:34.639463+0000","last_fullsized":"2024-05-27T00:33:34.639463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:11.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":240,"num_read_kb":174989,"num_write":1187,"num_write_kb":92068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"48'1572","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.366164+0000","last_change":"2024-05-27T00:17:54.493715+0000","last_active":"2024-05-27T00:33:33.366164+0000","last_peered":"2024-05-27T00:33:33.366164+0000","last_clean":"2024-05-27T00:33:33.366164+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:33:33.366164+0000","last_undegraded":"2024-05-27T00:33:33.366164+0000","last_fullsized":"2024-05-27T00:33:33.366164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1572,"log_dups_size":0,"ondisk_log_size":1572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:50:38.769021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017111721,"stat_sum":{"num_bytes":21962235,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":198,"num_read":57,"num_read_kb":45093,"num_write":1410,"num_write_kb":24294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"48'843","reported_seq":1087,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.869212+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:33:33.869212+0000","last_peered":"2024-05-27T00:33:33.869212+0000","last_clean":"2024-05-27T00:33:33.869212+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:33:33.869212+0000","last_undegraded":"2024-05-27T00:33:33.869212+0000","last_fullsized":"2024-05-27T00:33:33.869212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":305,"num_read_kb":205128,"num_write":983,"num_write_kb":76341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":67,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"48'594","reported_seq":837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.992961+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:33:29.992961+0000","last_peered":"2024-05-27T00:33:29.992961+0000","last_clean":"2024-05-27T00:33:29.992961+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:33:29.992961+0000","last_undegraded":"2024-05-27T00:33:29.992961+0000","last_fullsized":"2024-05-27T00:33:29.992961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:46:08.645927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":317,"num_read_kb":129314,"num_write":740,"num_write_kb":59170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"48'1791","reported_seq":3623,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.885090+0000","last_change":"2024-05-27T00:17:54.309270+0000","last_active":"2024-05-27T00:33:33.885090+0000","last_peered":"2024-05-27T00:33:33.885090+0000","last_clean":"2024-05-27T00:33:33.885090+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:33:33.885090+0000","last_undegraded":"2024-05-27T00:33:33.885090+0000","last_fullsized":"2024-05-27T00:33:33.885090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1791,"log_dups_size":0,"ondisk_log_size":1791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:49:00.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087892630000000003,"stat_sum":{"num_bytes":15177983,"num_objects":665,"num_object_clones":314,"num_object_copies":1330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":214,"num_read":52,"num_read_kb":17293,"num_write":1633,"num_write_kb":25745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"48'1911","reported_seq":2158,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.499719+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:33:33.499719+0000","last_peered":"2024-05-27T00:33:33.499719+0000","last_clean":"2024-05-27T00:33:33.499719+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:33:33.499719+0000","last_undegraded":"2024-05-27T00:33:33.499719+0000","last_fullsized":"2024-05-27T00:33:33.499719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1911,"log_dups_size":0,"ondisk_log_size":1911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":310,"num_read_kb":210919,"num_write":2080,"num_write_kb":80661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"48'1555","reported_seq":3431,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.839742+0000","last_change":"2024-05-27T00:17:54.489271+0000","last_active":"2024-05-27T00:33:29.839742+0000","last_peered":"2024-05-27T00:33:29.839742+0000","last_clean":"2024-05-27T00:33:29.839742+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:33:29.839742+0000","last_undegraded":"2024-05-27T00:33:29.839742+0000","last_fullsized":"2024-05-27T00:33:29.839742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:49:06.355571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014633893,"stat_sum":{"num_bytes":23977264,"num_objects":602,"num_object_clones":282,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":198,"num_read":64,"num_read_kb":52608,"num_write":1398,"num_write_kb":30195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"48'592","reported_seq":854,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.900929+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:33:33.900929+0000","last_peered":"2024-05-27T00:33:33.900929+0000","last_clean":"2024-05-27T00:33:33.900929+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:33:33.900929+0000","last_undegraded":"2024-05-27T00:33:33.900929+0000","last_fullsized":"2024-05-27T00:33:33.900929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21338252,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":289,"num_read_kb":124309,"num_write":674,"num_write_kb":63278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"48'1500","reported_seq":3389,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.928969+0000","last_change":"2024-05-27T00:17:54.512612+0000","last_active":"2024-05-27T00:33:30.928969+0000","last_peered":"2024-05-27T00:33:30.928969+0000","last_clean":"2024-05-27T00:33:30.928969+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:33:30.928969+0000","last_undegraded":"2024-05-27T00:33:30.928969+0000","last_fullsized":"2024-05-27T00:33:30.928969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:07:00.308291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034220156000000002,"stat_sum":{"num_bytes":22845870,"num_objects":565,"num_object_clones":264,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":188,"num_read":59,"num_read_kb":32908,"num_write":1344,"num_write_kb":23966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"48'1486","reported_seq":1692,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.894653+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:33:33.894653+0000","last_peered":"2024-05-27T00:33:33.894653+0000","last_clean":"2024-05-27T00:33:33.894653+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:33:33.894653+0000","last_undegraded":"2024-05-27T00:33:33.894653+0000","last_fullsized":"2024-05-27T00:33:33.894653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":244,"num_read_kb":161522,"num_write":1589,"num_write_kb":88109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"48'1752","reported_seq":3524,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.367187+0000","last_change":"2024-05-27T00:17:54.493629+0000","last_active":"2024-05-27T00:33:33.367187+0000","last_peered":"2024-05-27T00:33:33.367187+0000","last_clean":"2024-05-27T00:33:33.367187+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:33:33.367187+0000","last_undegraded":"2024-05-27T00:33:33.367187+0000","last_fullsized":"2024-05-27T00:33:33.367187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1752,"log_dups_size":0,"ondisk_log_size":1752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:26:59.993852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017358472,"stat_sum":{"num_bytes":15481537,"num_objects":707,"num_object_clones":381,"num_object_copies":1414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":707,"num_whiteouts":205,"num_read":59,"num_read_kb":17601,"num_write":1504,"num_write_kb":17792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"48'1524","reported_seq":1820,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.936597+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:33:33.936597+0000","last_peered":"2024-05-27T00:33:33.936597+0000","last_clean":"2024-05-27T00:33:33.936597+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:33:33.936597+0000","last_undegraded":"2024-05-27T00:33:33.936597+0000","last_fullsized":"2024-05-27T00:33:33.936597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":428,"num_read_kb":130260,"num_write":1739,"num_write_kb":60710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"48'1583","reported_seq":3446,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.901089+0000","last_change":"2024-05-27T00:17:54.506220+0000","last_active":"2024-05-27T00:33:30.901089+0000","last_peered":"2024-05-27T00:33:30.901089+0000","last_clean":"2024-05-27T00:33:30.901089+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:33:30.901089+0000","last_undegraded":"2024-05-27T00:33:30.901089+0000","last_fullsized":"2024-05-27T00:33:30.901089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1583,"log_dups_size":0,"ondisk_log_size":1583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:32:01.052698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022583427,"stat_sum":{"num_bytes":13475539,"num_objects":608,"num_object_clones":285,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":200,"num_read":54,"num_read_kb":12130,"num_write":1429,"num_write_kb":19813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"48'1064","reported_seq":1346,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.872488+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:33:33.872488+0000","last_peered":"2024-05-27T00:33:33.872488+0000","last_clean":"2024-05-27T00:33:33.872488+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:33:33.872488+0000","last_undegraded":"2024-05-27T00:33:33.872488+0000","last_fullsized":"2024-05-27T00:33:33.872488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555104,"num_objects":77,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":365,"num_read_kb":194738,"num_write":1216,"num_write_kb":100979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"48'1567","reported_seq":3423,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.931447+0000","last_change":"2024-05-27T00:17:54.336019+0000","last_active":"2024-05-27T00:33:33.931447+0000","last_peered":"2024-05-27T00:33:33.931447+0000","last_clean":"2024-05-27T00:33:33.931447+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:33:33.931447+0000","last_undegraded":"2024-05-27T00:33:33.931447+0000","last_fullsized":"2024-05-27T00:33:33.931447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:48.582600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.030558409000000002,"stat_sum":{"num_bytes":10200444,"num_objects":601,"num_object_clones":280,"num_object_copies":1202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":188,"num_read":42,"num_read_kb":12368,"num_write":1435,"num_write_kb":13032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"48'830","reported_seq":1084,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.892943+0000","last_change":"2024-05-27T00:15:24.605986+0000","last_active":"2024-05-27T00:33:33.892943+0000","last_peered":"2024-05-27T00:33:33.892943+0000","last_clean":"2024-05-27T00:33:33.892943+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:33:33.892943+0000","last_undegraded":"2024-05-27T00:33:33.892943+0000","last_fullsized":"2024-05-27T00:33:33.892943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971554,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":336,"num_read_kb":131285,"num_write":974,"num_write_kb":63316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":56,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"48'1706","reported_seq":3516,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.390563+0000","last_change":"2024-05-27T00:17:54.493027+0000","last_active":"2024-05-27T00:33:33.390563+0000","last_peered":"2024-05-27T00:33:33.390563+0000","last_clean":"2024-05-27T00:33:33.390563+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:33:33.390563+0000","last_undegraded":"2024-05-27T00:33:33.390563+0000","last_fullsized":"2024-05-27T00:33:33.390563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1706,"log_dups_size":0,"ondisk_log_size":1706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:38:11.155792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018271039999999999,"stat_sum":{"num_bytes":20173932,"num_objects":652,"num_object_clones":356,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":187,"num_read":72,"num_read_kb":36313,"num_write":1471,"num_write_kb":31527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"48'502","reported_seq":699,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.529271+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:33:34.529271+0000","last_peered":"2024-05-27T00:33:34.529271+0000","last_clean":"2024-05-27T00:33:34.529271+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:33:34.529271+0000","last_undegraded":"2024-05-27T00:33:34.529271+0000","last_fullsized":"2024-05-27T00:33:34.529271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":232,"num_read_kb":71610,"num_write":625,"num_write_kb":47247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"48'1677","reported_seq":3546,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.955606+0000","last_change":"2024-05-27T00:17:54.334890+0000","last_active":"2024-05-27T00:33:30.955606+0000","last_peered":"2024-05-27T00:33:30.955606+0000","last_clean":"2024-05-27T00:33:30.955606+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:33:30.955606+0000","last_undegraded":"2024-05-27T00:33:30.955606+0000","last_fullsized":"2024-05-27T00:33:30.955606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1677,"log_dups_size":0,"ondisk_log_size":1677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:55.040370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028694234999999998,"stat_sum":{"num_bytes":27706000,"num_objects":637,"num_object_clones":296,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":195,"num_read":71,"num_read_kb":46547,"num_write":1532,"num_write_kb":29714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"48'364","reported_seq":562,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.779855+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:33:29.779855+0000","last_peered":"2024-05-27T00:33:29.779855+0000","last_clean":"2024-05-27T00:33:29.779855+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:33:29.779855+0000","last_undegraded":"2024-05-27T00:33:29.779855+0000","last_fullsized":"2024-05-27T00:33:29.779855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":244,"num_read_kb":99922,"num_write":491,"num_write_kb":51118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"48'1573","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.391959+0000","last_change":"2024-05-27T00:17:54.332802+0000","last_active":"2024-05-27T00:33:33.391959+0000","last_peered":"2024-05-27T00:33:33.391959+0000","last_clean":"2024-05-27T00:33:33.391959+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:33:33.391959+0000","last_undegraded":"2024-05-27T00:33:33.391959+0000","last_fullsized":"2024-05-27T00:33:33.391959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:41:31.328434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025579069999999999,"stat_sum":{"num_bytes":19680366,"num_objects":592,"num_object_clones":272,"num_object_copies":1184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":592,"num_whiteouts":185,"num_read":43,"num_read_kb":14554,"num_write":1424,"num_write_kb":24215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"48'749","reported_seq":1035,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.864326+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:33:33.864326+0000","last_peered":"2024-05-27T00:33:33.864326+0000","last_clean":"2024-05-27T00:33:33.864326+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:33:33.864326+0000","last_undegraded":"2024-05-27T00:33:33.864326+0000","last_fullsized":"2024-05-27T00:33:33.864326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":749,"log_dups_size":0,"ondisk_log_size":749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":479,"num_read_kb":120710,"num_write":984,"num_write_kb":72758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"48'1617","reported_seq":3492,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.972082+0000","last_change":"2024-05-27T00:17:54.334829+0000","last_active":"2024-05-27T00:33:30.972082+0000","last_peered":"2024-05-27T00:33:30.972082+0000","last_clean":"2024-05-27T00:33:30.972082+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:33:30.972082+0000","last_undegraded":"2024-05-27T00:33:30.972082+0000","last_fullsized":"2024-05-27T00:33:30.972082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1617,"log_dups_size":0,"ondisk_log_size":1617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:00:47.756042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028937418999999999,"stat_sum":{"num_bytes":19454539,"num_objects":599,"num_object_clones":279,"num_object_copies":1198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":599,"num_whiteouts":201,"num_read":60,"num_read_kb":39320,"num_write":1444,"num_write_kb":29954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"48'926","reported_seq":1176,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.639917+0000","last_change":"2024-05-27T00:15:24.601924+0000","last_active":"2024-05-27T00:33:34.639917+0000","last_peered":"2024-05-27T00:33:34.639917+0000","last_clean":"2024-05-27T00:33:34.639917+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:33:34.639917+0000","last_undegraded":"2024-05-27T00:33:34.639917+0000","last_fullsized":"2024-05-27T00:33:34.639917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":926,"log_dups_size":0,"ondisk_log_size":926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:32.185839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971542,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":329,"num_read_kb":104768,"num_write":1076,"num_write_kb":59606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"48'1603","reported_seq":3463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.427204+0000","last_change":"2024-05-27T00:17:54.335438+0000","last_active":"2024-05-27T00:33:30.427204+0000","last_peered":"2024-05-27T00:33:30.427204+0000","last_clean":"2024-05-27T00:33:30.427204+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:33:30.427204+0000","last_undegraded":"2024-05-27T00:33:30.427204+0000","last_fullsized":"2024-05-27T00:33:30.427204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:44.778493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029056843999999998,"stat_sum":{"num_bytes":27639049,"num_objects":630,"num_object_clones":290,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":189,"num_read":55,"num_read_kb":39571,"num_write":1460,"num_write_kb":33520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"48'850","reported_seq":1063,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:28.880655+0000","last_change":"2024-05-27T00:15:24.600878+0000","last_active":"2024-05-27T00:33:28.880655+0000","last_peered":"2024-05-27T00:33:28.880655+0000","last_clean":"2024-05-27T00:33:28.880655+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:33:28.880655+0000","last_undegraded":"2024-05-27T00:33:28.880655+0000","last_fullsized":"2024-05-27T00:33:28.880655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":241,"num_read_kb":194399,"num_write":970,"num_write_kb":84408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"48'1539","reported_seq":3419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.936151+0000","last_change":"2024-05-27T00:17:54.337688+0000","last_active":"2024-05-27T00:33:30.936151+0000","last_peered":"2024-05-27T00:33:30.936151+0000","last_clean":"2024-05-27T00:33:30.936151+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:33:30.936151+0000","last_undegraded":"2024-05-27T00:33:30.936151+0000","last_fullsized":"2024-05-27T00:33:30.936151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1539,"log_dups_size":0,"ondisk_log_size":1539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:09.263437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028414170999999998,"stat_sum":{"num_bytes":22590275,"num_objects":587,"num_object_clones":272,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":186,"num_read":57,"num_read_kb":45093,"num_write":1409,"num_write_kb":28119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"48'1073","reported_seq":1310,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.705095+0000","last_change":"2024-05-27T00:15:24.603452+0000","last_active":"2024-05-27T00:33:34.705095+0000","last_peered":"2024-05-27T00:33:34.705095+0000","last_clean":"2024-05-27T00:33:34.705095+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:33:34.705095+0000","last_undegraded":"2024-05-27T00:33:34.705095+0000","last_fullsized":"2024-05-27T00:33:34.705095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1073,"log_dups_size":0,"ondisk_log_size":1073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34922452,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":293,"num_read_kb":166112,"num_write":1227,"num_write_kb":85323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"48'1558","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.931551+0000","last_change":"2024-05-27T00:17:54.512510+0000","last_active":"2024-05-27T00:33:30.931551+0000","last_peered":"2024-05-27T00:33:30.931551+0000","last_clean":"2024-05-27T00:33:30.931551+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:33:30.931551+0000","last_undegraded":"2024-05-27T00:33:30.931551+0000","last_fullsized":"2024-05-27T00:33:30.931551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1558,"log_dups_size":0,"ondisk_log_size":1558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:45.104337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034804740000000001,"stat_sum":{"num_bytes":14041732,"num_objects":561,"num_object_clones":257,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":186,"num_read":41,"num_read_kb":16491,"num_write":1423,"num_write_kb":17133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"48'755","reported_seq":983,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.895276+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:33:33.895276+0000","last_peered":"2024-05-27T00:33:33.895276+0000","last_clean":"2024-05-27T00:33:33.895276+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:33:33.895276+0000","last_undegraded":"2024-05-27T00:33:33.895276+0000","last_fullsized":"2024-05-27T00:33:33.895276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":292,"num_read_kb":141638,"num_write":926,"num_write_kb":64367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"48'1568","reported_seq":3422,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.895827+0000","last_change":"2024-05-27T00:17:54.333811+0000","last_active":"2024-05-27T00:33:33.895827+0000","last_peered":"2024-05-27T00:33:33.895827+0000","last_clean":"2024-05-27T00:33:33.895827+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:33:33.895827+0000","last_undegraded":"2024-05-27T00:33:33.895827+0000","last_fullsized":"2024-05-27T00:33:33.895827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:14.148909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028015244000000002,"stat_sum":{"num_bytes":15735198,"num_objects":613,"num_object_clones":289,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":202,"num_read":48,"num_read_kb":16478,"num_write":1397,"num_write_kb":22754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":231,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:32:10.519872+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:32:10.519872+0000","last_peered":"2024-05-27T00:32:10.519872+0000","last_clean":"2024-05-27T00:32:10.519872+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:32:10.519872+0000","last_undegraded":"2024-05-27T00:32:10.519872+0000","last_fullsized":"2024-05-27T00:32:10.519872+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"48'875","reported_seq":1072,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.499832+0000","last_change":"2024-05-27T00:15:24.603781+0000","last_active":"2024-05-27T00:33:33.499832+0000","last_peered":"2024-05-27T00:33:33.499832+0000","last_clean":"2024-05-27T00:33:33.499832+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:33:33.499832+0000","last_undegraded":"2024-05-27T00:33:33.499832+0000","last_fullsized":"2024-05-27T00:33:33.499832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":219,"num_read_kb":131843,"num_write":965,"num_write_kb":67539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"48'1616","reported_seq":3478,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.386594+0000","last_change":"2024-05-27T00:17:54.309203+0000","last_active":"2024-05-27T00:33:33.386594+0000","last_peered":"2024-05-27T00:33:33.386594+0000","last_clean":"2024-05-27T00:33:33.386594+0000","last_became_active":"2024-05-27T00:16:06.517816+0000","last_became_peered":"2024-05-27T00:16:06.517816+0000","last_unstale":"2024-05-27T00:33:33.386594+0000","last_undegraded":"2024-05-27T00:33:33.386594+0000","last_fullsized":"2024-05-27T00:33:33.386594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:08.300384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085828740000000008,"stat_sum":{"num_bytes":27686043,"num_objects":633,"num_object_clones":292,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":198,"num_read":54,"num_read_kb":34546,"num_write":1470,"num_write_kb":32830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"48'429","reported_seq":666,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.891792+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:33:33.891792+0000","last_peered":"2024-05-27T00:33:33.891792+0000","last_clean":"2024-05-27T00:33:33.891792+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:33:33.891792+0000","last_undegraded":"2024-05-27T00:33:33.891792+0000","last_fullsized":"2024-05-27T00:33:33.891792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":299,"num_read_kb":174228,"num_write":552,"num_write_kb":63235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"48'1520","reported_seq":3388,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.382336+0000","last_change":"2024-05-27T00:17:54.490594+0000","last_active":"2024-05-27T00:33:33.382336+0000","last_peered":"2024-05-27T00:33:33.382336+0000","last_clean":"2024-05-27T00:33:33.382336+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:33:33.382336+0000","last_undegraded":"2024-05-27T00:33:33.382336+0000","last_fullsized":"2024-05-27T00:33:33.382336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:45:47.096794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016778365999999999,"stat_sum":{"num_bytes":6046316,"num_objects":564,"num_object_clones":261,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":182,"num_read":34,"num_read_kb":53,"num_write":1394,"num_write_kb":7687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'1654","reported_seq":2540,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.495949+0000","last_change":"2024-05-27T00:15:24.598182+0000","last_active":"2024-05-27T00:33:33.495949+0000","last_peered":"2024-05-27T00:33:33.495949+0000","last_clean":"2024-05-27T00:33:33.495949+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:33:33.495949+0000","last_undegraded":"2024-05-27T00:33:33.495949+0000","last_fullsized":"2024-05-27T00:33:33.495949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1654,"log_dups_size":0,"ondisk_log_size":1654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":85,"num_object_clones":0,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":1008,"num_read_kb":169683,"num_write":1827,"num_write_kb":85277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"48'1615","reported_seq":3473,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.909399+0000","last_change":"2024-05-27T00:17:54.491784+0000","last_active":"2024-05-27T00:33:33.909399+0000","last_peered":"2024-05-27T00:33:33.909399+0000","last_clean":"2024-05-27T00:33:33.909399+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:33:33.909399+0000","last_undegraded":"2024-05-27T00:33:33.909399+0000","last_fullsized":"2024-05-27T00:33:33.909399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:22.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016042048999999999,"stat_sum":{"num_bytes":11249311,"num_objects":630,"num_object_clones":289,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":203,"num_read":51,"num_read_kb":12401,"num_write":1463,"num_write_kb":13659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"48'708","reported_seq":967,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.358425+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:33:30.358425+0000","last_peered":"2024-05-27T00:33:30.358425+0000","last_clean":"2024-05-27T00:33:30.358425+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:33:30.358425+0000","last_undegraded":"2024-05-27T00:33:30.358425+0000","last_fullsized":"2024-05-27T00:33:30.358425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":344,"num_read_kb":168323,"num_write":926,"num_write_kb":68366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"48'1642","reported_seq":3512,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.350257+0000","last_change":"2024-05-27T00:17:54.505548+0000","last_active":"2024-05-27T00:33:30.350257+0000","last_peered":"2024-05-27T00:33:30.350257+0000","last_clean":"2024-05-27T00:33:30.350257+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:33:30.350257+0000","last_undegraded":"2024-05-27T00:33:30.350257+0000","last_fullsized":"2024-05-27T00:33:30.350257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:08.630844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021175818999999999,"stat_sum":{"num_bytes":27156672,"num_objects":598,"num_object_clones":274,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":182,"num_read":49,"num_read_kb":37771,"num_write":1519,"num_write_kb":29445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"48'814","reported_seq":1035,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.885329+0000","last_change":"2024-05-27T00:15:24.603518+0000","last_active":"2024-05-27T00:33:33.885329+0000","last_peered":"2024-05-27T00:33:33.885329+0000","last_clean":"2024-05-27T00:33:33.885329+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:33:33.885329+0000","last_undegraded":"2024-05-27T00:33:33.885329+0000","last_fullsized":"2024-05-27T00:33:33.885329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":262,"num_read_kb":165581,"num_write":935,"num_write_kb":83695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"48'1593","reported_seq":3458,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.920096+0000","last_change":"2024-05-27T00:17:54.511904+0000","last_active":"2024-05-27T00:33:30.920096+0000","last_peered":"2024-05-27T00:33:30.920096+0000","last_clean":"2024-05-27T00:33:30.920096+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:33:30.920096+0000","last_undegraded":"2024-05-27T00:33:30.920096+0000","last_fullsized":"2024-05-27T00:33:30.920096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1593,"log_dups_size":0,"ondisk_log_size":1593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:58:08.710145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034092800999999999,"stat_sum":{"num_bytes":22484461,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":211,"num_read":50,"num_read_kb":38442,"num_write":1413,"num_write_kb":28463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"48'1412","reported_seq":1652,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.537498+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:33:34.537498+0000","last_peered":"2024-05-27T00:33:34.537498+0000","last_clean":"2024-05-27T00:33:34.537498+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:33:34.537498+0000","last_undegraded":"2024-05-27T00:33:34.537498+0000","last_fullsized":"2024-05-27T00:33:34.537498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1412,"log_dups_size":0,"ondisk_log_size":1412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":299,"num_read_kb":161725,"num_write":1529,"num_write_kb":67545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"48'1569","reported_seq":3448,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.363078+0000","last_change":"2024-05-27T00:17:54.488788+0000","last_active":"2024-05-27T00:33:33.363078+0000","last_peered":"2024-05-27T00:33:33.363078+0000","last_clean":"2024-05-27T00:33:33.363078+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:33:33.363078+0000","last_undegraded":"2024-05-27T00:33:33.363078+0000","last_fullsized":"2024-05-27T00:33:33.363078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:42:30.595550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014512239,"stat_sum":{"num_bytes":23092289,"num_objects":584,"num_object_clones":267,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":187,"num_read":51,"num_read_kb":31023,"num_write":1448,"num_write_kb":28334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'833","reported_seq":1070,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.637756+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:33:34.637756+0000","last_peered":"2024-05-27T00:33:34.637756+0000","last_clean":"2024-05-27T00:33:34.637756+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:33:34.637756+0000","last_undegraded":"2024-05-27T00:33:34.637756+0000","last_fullsized":"2024-05-27T00:33:34.637756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":833,"log_dups_size":0,"ondisk_log_size":833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":312,"num_read_kb":117165,"num_write":1596,"num_write_kb":59298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"48'1511","reported_seq":3386,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.500418+0000","last_change":"2024-05-27T00:17:54.310241+0000","last_active":"2024-05-27T00:33:30.500418+0000","last_peered":"2024-05-27T00:33:30.500418+0000","last_clean":"2024-05-27T00:33:30.500418+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:33:30.500418+0000","last_undegraded":"2024-05-27T00:33:30.500418+0000","last_fullsized":"2024-05-27T00:33:30.500418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:51.525130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089509419999999999,"stat_sum":{"num_bytes":13250576,"num_objects":583,"num_object_clones":269,"num_object_copies":1166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":181,"num_read":49,"num_read_kb":2453,"num_write":1381,"num_write_kb":19589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"48'643","reported_seq":895,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.706802+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:33:34.706802+0000","last_peered":"2024-05-27T00:33:34.706802+0000","last_clean":"2024-05-27T00:33:34.706802+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:33:34.706802+0000","last_undegraded":"2024-05-27T00:33:34.706802+0000","last_fullsized":"2024-05-27T00:33:34.706802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":352,"num_read_kb":95868,"num_write":810,"num_write_kb":47576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"48'1573","reported_seq":3446,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.501372+0000","last_change":"2024-05-27T00:17:54.492705+0000","last_active":"2024-05-27T00:33:30.501372+0000","last_peered":"2024-05-27T00:33:30.501372+0000","last_clean":"2024-05-27T00:33:30.501372+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:33:30.501372+0000","last_undegraded":"2024-05-27T00:33:30.501372+0000","last_fullsized":"2024-05-27T00:33:30.501372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:40:53.350258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018225775999999999,"stat_sum":{"num_bytes":9758467,"num_objects":579,"num_object_clones":270,"num_object_copies":1158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":186,"num_read":49,"num_read_kb":4155,"num_write":1423,"num_write_kb":16751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'471","reported_seq":692,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.886300+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:33:30.886300+0000","last_peered":"2024-05-27T00:33:30.886300+0000","last_clean":"2024-05-27T00:33:30.886300+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:33:30.886300+0000","last_undegraded":"2024-05-27T00:33:30.886300+0000","last_fullsized":"2024-05-27T00:33:30.886300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":297,"num_read_kb":59216,"num_write":633,"num_write_kb":42916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":70,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"48'1624","reported_seq":3487,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.893727+0000","last_change":"2024-05-27T00:17:54.504734+0000","last_active":"2024-05-27T00:33:33.893727+0000","last_peered":"2024-05-27T00:33:33.893727+0000","last_clean":"2024-05-27T00:33:33.893727+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:33:33.893727+0000","last_undegraded":"2024-05-27T00:33:33.893727+0000","last_fullsized":"2024-05-27T00:33:33.893727+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:34:00.119528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022802596000000001,"stat_sum":{"num_bytes":19569863,"num_objects":620,"num_object_clones":284,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":196,"num_read":52,"num_read_kb":30708,"num_write":1470,"num_write_kb":21076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"48'783","reported_seq":953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.177344+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:33:30.177344+0000","last_peered":"2024-05-27T00:33:30.177344+0000","last_clean":"2024-05-27T00:33:30.177344+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:33:30.177344+0000","last_undegraded":"2024-05-27T00:33:30.177344+0000","last_fullsized":"2024-05-27T00:33:30.177344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":783,"log_dups_size":0,"ondisk_log_size":783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":176,"num_read_kb":80391,"num_write":878,"num_write_kb":51066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"48'1663","reported_seq":3525,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.385750+0000","last_change":"2024-05-27T00:17:54.308939+0000","last_active":"2024-05-27T00:33:33.385750+0000","last_peered":"2024-05-27T00:33:33.385750+0000","last_clean":"2024-05-27T00:33:33.385750+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:33:33.385750+0000","last_undegraded":"2024-05-27T00:33:33.385750+0000","last_fullsized":"2024-05-27T00:33:33.385750+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:17:28.826160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084477519999999993,"stat_sum":{"num_bytes":19800736,"num_objects":636,"num_object_clones":295,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":205,"num_read":62,"num_read_kb":32956,"num_write":1508,"num_write_kb":26318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"48'1047","reported_seq":1271,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.532477+0000","last_change":"2024-05-27T00:15:24.598238+0000","last_active":"2024-05-27T00:33:34.532477+0000","last_peered":"2024-05-27T00:33:34.532477+0000","last_clean":"2024-05-27T00:33:34.532477+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:33:34.532477+0000","last_undegraded":"2024-05-27T00:33:34.532477+0000","last_fullsized":"2024-05-27T00:33:34.532477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1047,"log_dups_size":0,"ondisk_log_size":1047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":269,"num_read_kb":153672,"num_write":1132,"num_write_kb":75680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"48'1615","reported_seq":3456,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.386744+0000","last_change":"2024-05-27T00:17:54.310021+0000","last_active":"2024-05-27T00:33:33.386744+0000","last_peered":"2024-05-27T00:33:33.386744+0000","last_clean":"2024-05-27T00:33:33.386744+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:33:33.386744+0000","last_undegraded":"2024-05-27T00:33:33.386744+0000","last_fullsized":"2024-05-27T00:33:33.386744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:07:28.174929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091147599999999995,"stat_sum":{"num_bytes":15827867,"num_objects":624,"num_object_clones":289,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":199,"num_read":35,"num_read_kb":8228,"num_write":1456,"num_write_kb":20842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"48'689","reported_seq":942,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.707187+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:33:34.707187+0000","last_peered":"2024-05-27T00:33:34.707187+0000","last_clean":"2024-05-27T00:33:34.707187+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:33:34.707187+0000","last_undegraded":"2024-05-27T00:33:34.707187+0000","last_fullsized":"2024-05-27T00:33:34.707187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":689,"log_dups_size":0,"ondisk_log_size":689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":331,"num_read_kb":161875,"num_write":831,"num_write_kb":80242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"48'1587","reported_seq":3445,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.396200+0000","last_change":"2024-05-27T00:17:54.333938+0000","last_active":"2024-05-27T00:33:30.396200+0000","last_peered":"2024-05-27T00:33:30.396200+0000","last_clean":"2024-05-27T00:33:30.396200+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:33:30.396200+0000","last_undegraded":"2024-05-27T00:33:30.396200+0000","last_fullsized":"2024-05-27T00:33:30.396200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1587,"log_dups_size":0,"ondisk_log_size":1587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:22.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026544363000000001,"stat_sum":{"num_bytes":11103766,"num_objects":576,"num_object_clones":273,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":173,"num_read":36,"num_read_kb":3645,"num_write":1460,"num_write_kb":12414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"48'1524","reported_seq":3388,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.922267+0000","last_change":"2024-05-27T00:17:54.485932+0000","last_active":"2024-05-27T00:33:33.922267+0000","last_peered":"2024-05-27T00:33:33.922267+0000","last_clean":"2024-05-27T00:33:33.922267+0000","last_became_active":"2024-05-27T00:16:06.811922+0000","last_became_peered":"2024-05-27T00:16:06.811922+0000","last_unstale":"2024-05-27T00:33:33.922267+0000","last_undegraded":"2024-05-27T00:33:33.922267+0000","last_fullsized":"2024-05-27T00:33:33.922267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:29:39.727324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011085174,"stat_sum":{"num_bytes":14346371,"num_objects":587,"num_object_clones":272,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":198,"num_read":48,"num_read_kb":4199,"num_write":1374,"num_write_kb":25831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"48'490","reported_seq":703,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.794312+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:33:34.794312+0000","last_peered":"2024-05-27T00:33:34.794312+0000","last_clean":"2024-05-27T00:33:34.794312+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:33:34.794312+0000","last_undegraded":"2024-05-27T00:33:34.794312+0000","last_fullsized":"2024-05-27T00:33:34.794312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":267,"num_read_kb":124158,"num_write":632,"num_write_kb":63683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"48'1663","reported_seq":3491,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.392971+0000","last_change":"2024-05-27T00:17:54.511007+0000","last_active":"2024-05-27T00:33:33.392971+0000","last_peered":"2024-05-27T00:33:33.392971+0000","last_clean":"2024-05-27T00:33:33.392971+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:33:33.392971+0000","last_undegraded":"2024-05-27T00:33:33.392971+0000","last_fullsized":"2024-05-27T00:33:33.392971+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:43:33.746539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034279472999999998,"stat_sum":{"num_bytes":10536557,"num_objects":647,"num_object_clones":309,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":224,"num_read":43,"num_read_kb":11171,"num_write":1484,"num_write_kb":16947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"48'860","reported_seq":1117,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.539304+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:33:34.539304+0000","last_peered":"2024-05-27T00:33:34.539304+0000","last_clean":"2024-05-27T00:33:34.539304+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:33:34.539304+0000","last_undegraded":"2024-05-27T00:33:34.539304+0000","last_fullsized":"2024-05-27T00:33:34.539304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":860,"log_dups_size":0,"ondisk_log_size":860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":337,"num_read_kb":206604,"num_write":984,"num_write_kb":108700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"48'1538","reported_seq":3409,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.343921+0000","last_change":"2024-05-27T00:17:54.493359+0000","last_active":"2024-05-27T00:33:30.343921+0000","last_peered":"2024-05-27T00:33:30.343921+0000","last_clean":"2024-05-27T00:33:30.343921+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:33:30.343921+0000","last_undegraded":"2024-05-27T00:33:30.343921+0000","last_fullsized":"2024-05-27T00:33:30.343921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1538,"log_dups_size":0,"ondisk_log_size":1538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:32:56.589612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017325917,"stat_sum":{"num_bytes":22746478,"num_objects":600,"num_object_clones":283,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":186,"num_read":59,"num_read_kb":40021,"num_write":1401,"num_write_kb":24139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"48'1273","reported_seq":1530,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.702528+0000","last_change":"2024-05-27T00:15:24.600571+0000","last_active":"2024-05-27T00:33:34.702528+0000","last_peered":"2024-05-27T00:33:34.702528+0000","last_clean":"2024-05-27T00:33:34.702528+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:33:34.702528+0000","last_undegraded":"2024-05-27T00:33:34.702528+0000","last_fullsized":"2024-05-27T00:33:34.702528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1273,"log_dups_size":0,"ondisk_log_size":1273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:44:21.645223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":353,"num_read_kb":145683,"num_write":1459,"num_write_kb":63890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"48'1559","reported_seq":3435,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.394563+0000","last_change":"2024-05-27T00:17:54.309879+0000","last_active":"2024-05-27T00:33:33.394563+0000","last_peered":"2024-05-27T00:33:33.394563+0000","last_clean":"2024-05-27T00:33:33.394563+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:33:33.394563+0000","last_undegraded":"2024-05-27T00:33:33.394563+0000","last_fullsized":"2024-05-27T00:33:33.394563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1559,"log_dups_size":0,"ondisk_log_size":1559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:20:48.824140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089714449999999998,"stat_sum":{"num_bytes":23504918,"num_objects":585,"num_object_clones":273,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":188,"num_read":53,"num_read_kb":39809,"num_write":1415,"num_write_kb":24852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"48'514","reported_seq":753,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.528633+0000","last_change":"2024-05-27T00:15:24.615066+0000","last_active":"2024-05-27T00:33:34.528633+0000","last_peered":"2024-05-27T00:33:34.528633+0000","last_clean":"2024-05-27T00:33:34.528633+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:33:34.528633+0000","last_undegraded":"2024-05-27T00:33:34.528633+0000","last_fullsized":"2024-05-27T00:33:34.528633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":327,"num_read_kb":63931,"num_write":692,"num_write_kb":47409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1174241513,"num_objects":39009,"num_object_clones":18298,"num_object_copies":78018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39009,"num_whiteouts":12489,"num_read":3379,"num_read_kb":1679737,"num_write":92416,"num_write_kb":1475210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2440749056,"data_stored":2350449106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102162,"ondisk_log_size":102162,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":781876696,"num_objects":2104,"num_object_clones":0,"num_object_copies":4208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":10396,"num_read_kb":4541569,"num_write":33862,"num_write_kb":2225576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1834,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1563811840,"data_stored":1563753392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28655,"ondisk_log_size":28655,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542368,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746260,"kb_used_data":606860,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93625580,"statfs":{"total":96636764160,"available":95872593920,"internally_reserved":0,"allocated":621424640,"data_stored":608366707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142735818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542369,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573040,"kb_used_data":443368,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93798800,"statfs":{"total":96636764160,"available":96049971200,"internally_reserved":0,"allocated":454008832,"data_stored":445134014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542370,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741064,"kb_used_data":583616,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93630776,"statfs":{"total":96636764160,"available":95877914624,"internally_reserved":0,"allocated":597622784,"data_stored":584115090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161216965},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542368,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662452,"kb_used_data":518892,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93709388,"statfs":{"total":96636764160,"available":95958413312,"internally_reserved":0,"allocated":531345408,"data_stored":517393413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542369,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":656408,"kb_used_data":521168,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93715432,"statfs":{"total":96636764160,"available":95964602368,"internally_reserved":0,"allocated":533676032,"data_stored":519848185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138475975},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542368,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439708,"kb_used_data":336404,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93932132,"statfs":{"total":96636764160,"available":96186503168,"internally_reserved":0,"allocated":344477696,"data_stored":336181610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542369,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573740,"kb_used_data":426020,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93798100,"statfs":{"total":96636764160,"available":96049254400,"internally_reserved":0,"allocated":436244480,"data_stored":428173666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151255495},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542368,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623352,"kb_used_data":479792,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93748488,"statfs":{"total":96636764160,"available":95998451712,"internally_reserved":0,"allocated":491307008,"data_stored":478155533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167784448,"data_stored":167772798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":223674368,"data_stored":223666222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125829120,"data_stored":125829120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192946176,"data_stored":192938656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":160763904,"data_stored":160751628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":214282240,"data_stored":214276326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":222674944,"data_stored":222665516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":255856640,"data_stored":255853126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":322383872,"data_stored":309544244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212025344,"data_stored":204259321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218103808,"data_stored":210104367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340185088,"data_stored":326661406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":370036736,"data_stored":356393662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":382795776,"data_stored":369590641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":230789120,"data_stored":222220375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":364429312,"data_stored":351675090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:33:40.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:33:41.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.002+0000 7f05f1800640 1 Processor -- start 2024-05-27T00:33:41.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.002+0000 7f05f1800640 1 -- start start 2024-05-27T00:33:41.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05f1800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 0x7f05ec0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05f1800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05ec0735d0 0x7f05ec151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05f1800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05ec0c17d0 con 0x7f05ec0c79f0 2024-05-27T00:33:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05f1800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f05ec071990 con 0x7f05ec0735d0 2024-05-27T00:33:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05f1800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f05ec070e10 con 0x7f05ec0731f0 2024-05-27T00:33:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05e2000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f05ec0735d0 0x7f05ec151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05eb400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f05ec0c79f0 0x7f05ec157970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56916/0 (socket says 172.21.15.98:56916) 2024-05-27T00:33:41.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05eb400640 1 -- 172.21.15.98:0/1475257541 learned_addr learned my addr 172.21.15.98:0/1475257541 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05eaa00640 1 --2- 172.21.15.98:0/1475257541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 0x7f05ec0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05e2000640 1 -- 172.21.15.98:0/1475257541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 msgr2=0x7f05ec0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05e2000640 1 --2- 172.21.15.98:0/1475257541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 0x7f05ec0c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05e2000640 1 -- 172.21.15.98:0/1475257541 >> v1:172.21.15.98:6789/0 conn(0x7f05ec0c79f0 legacy=0x7f05ec157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.006+0000 7f05e2000640 1 -- 172.21.15.98:0/1475257541 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ec0710f0 con 0x7f05ec0735d0 2024-05-27T00:33:41.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05e2000640 1 --2- 172.21.15.98:0/1475257541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0735d0 0x7f05ec151430 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f05d40097c0 tx=0x7f05d401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6b6448175f86fe server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:41.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05ea000640 1 -- 172.21.15.98:0/1475257541 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d40b5070 con 0x7f05ec0735d0 2024-05-27T00:33:41.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05ea000640 1 -- 172.21.15.98:0/1475257541 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d40a8990 con 0x7f05ec0735d0 2024-05-27T00:33:41.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05ea000640 1 -- 172.21.15.98:0/1475257541 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d40a8d60 con 0x7f05ec0735d0 2024-05-27T00:33:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 -- 172.21.15.98:0/1475257541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0735d0 msgr2=0x7f05ec151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 --2- 172.21.15.98:0/1475257541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0735d0 0x7f05ec151430 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f05d40097c0 tx=0x7f05d401ec90 comp rx=0 tx=0).stop 2024-05-27T00:33:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 -- 172.21.15.98:0/1475257541 shutdown_connections 2024-05-27T00:33:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 --2- 172.21.15.98:0/1475257541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0735d0 0x7f05ec151430 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 --2- 172.21.15.98:0/1475257541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 0x7f05ec0c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 -- 172.21.15.98:0/1475257541 >> 172.21.15.98:0/1475257541 conn(0x7f05ec067530 msgr2=0x7f05ec06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:41.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 -- 172.21.15.98:0/1475257541 shutdown_connections 2024-05-27T00:33:41.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 -- 172.21.15.98:0/1475257541 wait complete. 2024-05-27T00:33:41.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 Processor -- start 2024-05-27T00:33:41.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 -- start start 2024-05-27T00:33:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 0x7f05ec0c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05ec0735d0 0x7f05ec14e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0c79f0 0x7f05ec1f0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec0c8400 con 0x7f05ec0c79f0 2024-05-27T00:33:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec0716b0 con 0x7f05ec0735d0 2024-05-27T00:33:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.010+0000 7f05f1800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f05ec0c13f0 con 0x7f05ec0731f0 2024-05-27T00:33:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0c79f0 0x7f05ec1f0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eb400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0c79f0 0x7f05ec1f0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59780/0 (socket says 172.21.15.98:59780) 2024-05-27T00:33:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eb400640 1 -- 172.21.15.98:0/2312296152 learned_addr learned my addr 172.21.15.98:0/2312296152 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eaa00640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 0x7f05ec0c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05e2000640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05ec0735d0 0x7f05ec14e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eb400640 1 -- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 msgr2=0x7f05ec0c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eb400640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 0x7f05ec0c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eb400640 1 -- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05ec0735d0 msgr2=0x7f05ec14e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eb400640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05ec0735d0 0x7f05ec14e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eb400640 1 -- 172.21.15.98:0/2312296152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ec150f10 con 0x7f05ec0c79f0 2024-05-27T00:33:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eaa00640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 0x7f05ec0c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05e2000640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05ec0735d0 0x7f05ec14e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:41.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05eb400640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0c79f0 0x7f05ec1f0450 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f05e400ca80 tx=0x7f05e400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05e3400640 1 -- 172.21.15.98:0/2312296152 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4017070 con 0x7f05ec0c79f0 2024-05-27T00:33:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05e3400640 1 -- 172.21.15.98:0/2312296152 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05e400ac30 con 0x7f05ec0c79f0 2024-05-27T00:33:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05f1800640 1 -- 172.21.15.98:0/2312296152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05ec15b950 con 0x7f05ec0c79f0 2024-05-27T00:33:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05e3400640 1 -- 172.21.15.98:0/2312296152 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4012600 con 0x7f05ec0c79f0 2024-05-27T00:33:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05f1800640 1 -- 172.21.15.98:0/2312296152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05ec15b670 con 0x7f05ec0c79f0 2024-05-27T00:33:41.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05f1800640 1 -- 172.21.15.98:0/2312296152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05ec144be0 con 0x7f05ec0c79f0 2024-05-27T00:33:41.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.014+0000 7f05e3400640 1 -- 172.21.15.98:0/2312296152 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f05e401a070 con 0x7f05ec0c79f0 2024-05-27T00:33:41.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.018+0000 7f05e3400640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f05c80678c0 0x7f05c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.018+0000 7f05e3400640 1 -- 172.21.15.98:0/2312296152 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f05e4085d20 con 0x7f05ec0c79f0 2024-05-27T00:33:41.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.018+0000 7f05eaa00640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f05c80678c0 0x7f05c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.018+0000 7f05eaa00640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f05c80678c0 0x7f05c8069d80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f05dc007a70 tx=0x7f05dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.022+0000 7f05e3400640 1 -- 172.21.15.98:0/2312296152 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f05e400ae20 con 0x7f05ec0c79f0 2024-05-27T00:33:41.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.174+0000 7f05f1800640 1 -- 172.21.15.98:0/2312296152 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f05ec158510 con 0x7f05c80678c0 2024-05-27T00:33:41.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.194+0000 7f05e3400640 1 -- 172.21.15.98:0/2312296152 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257325 (secure 0 0 0) 0x7f05ec158510 con 0x7f05c80678c0 2024-05-27T00:33:41.197 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:33:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:33:41.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 -- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f05c80678c0 msgr2=0x7f05c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f05c80678c0 0x7f05c8069d80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f05dc007a70 tx=0x7f05dc008040 comp rx=0 tx=0).stop 2024-05-27T00:33:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 -- 172.21.15.98:0/2312296152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0c79f0 msgr2=0x7f05ec1f0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0c79f0 0x7f05ec1f0450 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f05e400ca80 tx=0x7f05e400cf40 comp rx=0 tx=0).stop 2024-05-27T00:33:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 -- 172.21.15.98:0/2312296152 shutdown_connections 2024-05-27T00:33:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05ec0c79f0 0x7f05ec1f0450 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f05c80678c0 0x7f05c8069d80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05ec0735d0 0x7f05ec14e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 --2- 172.21.15.98:0/2312296152 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05ec0731f0 0x7f05ec0c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 -- 172.21.15.98:0/2312296152 >> 172.21.15.98:0/2312296152 conn(0x7f05ec067530 msgr2=0x7f05ec14ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:41.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 -- 172.21.15.98:0/2312296152 shutdown_connections 2024-05-27T00:33:41.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.198+0000 7f05f1800640 1 -- 172.21.15.98:0/2312296152 wait complete. 2024-05-27T00:33:41.218 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":595,"stamp":"2024-05-27T00:33:39.542504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1956708577,"num_objects":41115,"num_object_clones":18298,"num_object_copies":82230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41115,"num_whiteouts":12489,"num_read":13891,"num_read_kb":6221406,"num_write":126437,"num_write_kb":3703202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1834,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130934,"ondisk_log_size":130934,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5016024,"kb_used_data":3916120,"kb_used_omap":12,"kb_used_meta":1099827,"kb_avail":749958696,"statfs":{"total":773094113280,"available":767957704704,"internally_reserved":0,"allocated":4010106880,"data_stored":3917368218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1126223410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":63167789,"num_objects":639,"num_object_clones":225,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":203,"num_read":456,"num_read_kb":49335,"num_write":1016,"num_write_kb":62825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004786"},"pg_stats":[{"pgid":"3.3f","version":"48'1517","reported_seq":3393,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.925854+0000","last_change":"2024-05-27T00:17:54.336701+0000","last_active":"2024-05-27T00:33:33.925854+0000","last_peered":"2024-05-27T00:33:33.925854+0000","last_clean":"2024-05-27T00:33:33.925854+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:33:33.925854+0000","last_undegraded":"2024-05-27T00:33:33.925854+0000","last_fullsized":"2024-05-27T00:33:33.925854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:39.250119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027619028,"stat_sum":{"num_bytes":17373324,"num_objects":575,"num_object_clones":268,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":196,"num_read":49,"num_read_kb":31548,"num_write":1364,"num_write_kb":18943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"48'1504","reported_seq":3403,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.895060+0000","last_change":"2024-05-27T00:17:54.490184+0000","last_active":"2024-05-27T00:33:30.895060+0000","last_peered":"2024-05-27T00:33:30.895060+0000","last_clean":"2024-05-27T00:33:30.895060+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:33:30.895060+0000","last_undegraded":"2024-05-27T00:33:30.895060+0000","last_fullsized":"2024-05-27T00:33:30.895060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1504,"log_dups_size":0,"ondisk_log_size":1504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:05:48.629948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015469564,"stat_sum":{"num_bytes":18053131,"num_objects":546,"num_object_clones":255,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":169,"num_read":59,"num_read_kb":35851,"num_write":1378,"num_write_kb":19996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"48'1616","reported_seq":3466,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.917802+0000","last_change":"2024-05-27T00:17:54.493058+0000","last_active":"2024-05-27T00:33:33.917802+0000","last_peered":"2024-05-27T00:33:33.917802+0000","last_clean":"2024-05-27T00:33:33.917802+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:33:33.917802+0000","last_undegraded":"2024-05-27T00:33:33.917802+0000","last_fullsized":"2024-05-27T00:33:33.917802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:59:53.965584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018507847000000001,"stat_sum":{"num_bytes":15200236,"num_objects":629,"num_object_clones":298,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":212,"num_read":53,"num_read_kb":23374,"num_write":1443,"num_write_kb":25758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"48'1615","reported_seq":3479,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.883919+0000","last_change":"2024-05-27T00:17:54.510484+0000","last_active":"2024-05-27T00:33:30.883919+0000","last_peered":"2024-05-27T00:33:30.883919+0000","last_clean":"2024-05-27T00:33:30.883919+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:33:30.883919+0000","last_undegraded":"2024-05-27T00:33:30.883919+0000","last_fullsized":"2024-05-27T00:33:30.883919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:39.250119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034338709000000002,"stat_sum":{"num_bytes":10173730,"num_objects":618,"num_object_clones":288,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":202,"num_read":56,"num_read_kb":12442,"num_write":1451,"num_write_kb":16900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"48'1701","reported_seq":3542,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.395661+0000","last_change":"2024-05-27T00:17:54.331579+0000","last_active":"2024-05-27T00:33:33.395661+0000","last_peered":"2024-05-27T00:33:33.395661+0000","last_clean":"2024-05-27T00:33:33.395661+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:33:33.395661+0000","last_undegraded":"2024-05-27T00:33:33.395661+0000","last_fullsized":"2024-05-27T00:33:33.395661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1701,"log_dups_size":0,"ondisk_log_size":1701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:35:37.983576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026116615999999999,"stat_sum":{"num_bytes":20047775,"num_objects":666,"num_object_clones":311,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":219,"num_read":58,"num_read_kb":29919,"num_write":1529,"num_write_kb":21884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"48'1457","reported_seq":3333,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.926396+0000","last_change":"2024-05-27T00:17:54.507830+0000","last_active":"2024-05-27T00:33:33.926396+0000","last_peered":"2024-05-27T00:33:33.926396+0000","last_clean":"2024-05-27T00:33:33.926396+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:33:33.926396+0000","last_undegraded":"2024-05-27T00:33:33.926396+0000","last_fullsized":"2024-05-27T00:33:33.926396+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1457,"log_dups_size":0,"ondisk_log_size":1457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:48.582600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032595461999999999,"stat_sum":{"num_bytes":14545166,"num_objects":545,"num_object_clones":256,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":181,"num_read":38,"num_read_kb":8264,"num_write":1295,"num_write_kb":15901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"48'1575","reported_seq":3443,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.871914+0000","last_change":"2024-05-27T00:17:54.335843+0000","last_active":"2024-05-27T00:33:29.871914+0000","last_peered":"2024-05-27T00:33:29.871914+0000","last_clean":"2024-05-27T00:33:29.871914+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:33:29.871914+0000","last_undegraded":"2024-05-27T00:33:29.871914+0000","last_fullsized":"2024-05-27T00:33:29.871914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1575,"log_dups_size":0,"ondisk_log_size":1575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:27:21.913063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029344160000000001,"stat_sum":{"num_bytes":19339722,"num_objects":602,"num_object_clones":277,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":190,"num_read":51,"num_read_kb":28747,"num_write":1431,"num_write_kb":22126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"48'1515","reported_seq":3386,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.468342+0000","last_change":"2024-05-27T00:17:54.485261+0000","last_active":"2024-05-27T00:33:30.468342+0000","last_peered":"2024-05-27T00:33:30.468342+0000","last_clean":"2024-05-27T00:33:30.468342+0000","last_became_active":"2024-05-27T00:16:06.280009+0000","last_became_peered":"2024-05-27T00:16:06.280009+0000","last_unstale":"2024-05-27T00:33:30.468342+0000","last_undegraded":"2024-05-27T00:33:30.468342+0000","last_fullsized":"2024-05-27T00:33:30.468342+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:14:46.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012011287000000001,"stat_sum":{"num_bytes":35835659,"num_objects":577,"num_object_clones":269,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":179,"num_read":52,"num_read_kb":60147,"num_write":1378,"num_write_kb":38796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.37","version":"48'1620","reported_seq":3476,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.395963+0000","last_change":"2024-05-27T00:17:54.485867+0000","last_active":"2024-05-27T00:33:33.395963+0000","last_peered":"2024-05-27T00:33:33.395963+0000","last_clean":"2024-05-27T00:33:33.395963+0000","last_became_active":"2024-05-27T00:16:06.280544+0000","last_became_peered":"2024-05-27T00:16:06.280544+0000","last_unstale":"2024-05-27T00:33:33.395963+0000","last_undegraded":"2024-05-27T00:33:33.395963+0000","last_fullsized":"2024-05-27T00:33:33.395963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:20:13.063312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096681779999999995,"stat_sum":{"num_bytes":20135711,"num_objects":629,"num_object_clones":292,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":198,"num_read":60,"num_read_kb":29133,"num_write":1469,"num_write_kb":25955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"48'1514","reported_seq":3391,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.450019+0000","last_change":"2024-05-27T00:17:54.337723+0000","last_active":"2024-05-27T00:33:30.450019+0000","last_peered":"2024-05-27T00:33:30.450019+0000","last_clean":"2024-05-27T00:33:30.450019+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:33:30.450019+0000","last_undegraded":"2024-05-27T00:33:30.450019+0000","last_fullsized":"2024-05-27T00:33:30.450019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1514,"log_dups_size":0,"ondisk_log_size":1514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:55:01.773614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027993297,"stat_sum":{"num_bytes":14778263,"num_objects":589,"num_object_clones":272,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":186,"num_read":55,"num_read_kb":22038,"num_write":1380,"num_write_kb":16492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"48'1627","reported_seq":3458,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.895588+0000","last_change":"2024-05-27T00:17:54.335171+0000","last_active":"2024-05-27T00:33:33.895588+0000","last_peered":"2024-05-27T00:33:33.895588+0000","last_clean":"2024-05-27T00:33:33.895588+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:33:33.895588+0000","last_undegraded":"2024-05-27T00:33:33.895588+0000","last_fullsized":"2024-05-27T00:33:33.895588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1627,"log_dups_size":0,"ondisk_log_size":1627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:11:26.796581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027428447000000002,"stat_sum":{"num_bytes":10658234,"num_objects":659,"num_object_clones":308,"num_object_copies":1318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":204,"num_read":45,"num_read_kb":12410,"num_write":1475,"num_write_kb":11557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"48'1609","reported_seq":3434,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.363658+0000","last_change":"2024-05-27T00:17:54.504751+0000","last_active":"2024-05-27T00:33:33.363658+0000","last_peered":"2024-05-27T00:33:33.363658+0000","last_clean":"2024-05-27T00:33:33.363658+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:33:33.363658+0000","last_undegraded":"2024-05-27T00:33:33.363658+0000","last_fullsized":"2024-05-27T00:33:33.363658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:51:55.125064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021724307000000002,"stat_sum":{"num_bytes":19374715,"num_objects":636,"num_object_clones":300,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":200,"num_read":31,"num_read_kb":14211,"num_write":1467,"num_write_kb":31877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"48'1547","reported_seq":3401,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.417979+0000","last_change":"2024-05-27T00:17:54.309597+0000","last_active":"2024-05-27T00:33:30.417979+0000","last_peered":"2024-05-27T00:33:30.417979+0000","last_clean":"2024-05-27T00:33:30.417979+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:33:30.417979+0000","last_undegraded":"2024-05-27T00:33:30.417979+0000","last_fullsized":"2024-05-27T00:33:30.417979+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1547,"log_dups_size":0,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:01:46.216541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089222539999999993,"stat_sum":{"num_bytes":26519812,"num_objects":619,"num_object_clones":291,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":212,"num_read":51,"num_read_kb":32873,"num_write":1364,"num_write_kb":36569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"48'1587","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.412049+0000","last_change":"2024-05-27T00:17:54.491142+0000","last_active":"2024-05-27T00:33:30.412049+0000","last_peered":"2024-05-27T00:33:30.412049+0000","last_clean":"2024-05-27T00:33:30.412049+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:33:30.412049+0000","last_undegraded":"2024-05-27T00:33:30.412049+0000","last_fullsized":"2024-05-27T00:33:30.412049+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1587,"log_dups_size":0,"ondisk_log_size":1587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:43:35.726386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016842227000000001,"stat_sum":{"num_bytes":10308290,"num_objects":608,"num_object_clones":288,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":204,"num_read":44,"num_read_kb":16487,"num_write":1426,"num_write_kb":16803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"48'1620","reported_seq":3487,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.955725+0000","last_change":"2024-05-27T00:17:54.491503+0000","last_active":"2024-05-27T00:33:30.955725+0000","last_peered":"2024-05-27T00:33:30.955725+0000","last_clean":"2024-05-27T00:33:30.955725+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:33:30.955725+0000","last_undegraded":"2024-05-27T00:33:30.955725+0000","last_fullsized":"2024-05-27T00:33:30.955725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:33:06.746690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016821815,"stat_sum":{"num_bytes":22996149,"num_objects":605,"num_object_clones":283,"num_object_copies":1210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":605,"num_whiteouts":212,"num_read":56,"num_read_kb":39458,"num_write":1456,"num_write_kb":27752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"48'1589","reported_seq":3458,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.899422+0000","last_change":"2024-05-27T00:17:54.506224+0000","last_active":"2024-05-27T00:33:30.899422+0000","last_peered":"2024-05-27T00:33:30.899422+0000","last_clean":"2024-05-27T00:33:30.899422+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:33:30.899422+0000","last_undegraded":"2024-05-27T00:33:30.899422+0000","last_fullsized":"2024-05-27T00:33:30.899422+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1589,"log_dups_size":0,"ondisk_log_size":1589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:44.778493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.031092022,"stat_sum":{"num_bytes":14532615,"num_objects":603,"num_object_clones":288,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":201,"num_read":63,"num_read_kb":16619,"num_write":1426,"num_write_kb":20413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"48'1709","reported_seq":3577,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.368113+0000","last_change":"2024-05-27T00:17:54.511207+0000","last_active":"2024-05-27T00:33:33.368113+0000","last_peered":"2024-05-27T00:33:33.368113+0000","last_clean":"2024-05-27T00:33:33.368113+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:33:33.368113+0000","last_undegraded":"2024-05-27T00:33:33.368113+0000","last_fullsized":"2024-05-27T00:33:33.368113+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1709,"log_dups_size":0,"ondisk_log_size":1709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:03:55.531180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034311531999999999,"stat_sum":{"num_bytes":28217639,"num_objects":653,"num_object_clones":304,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":203,"num_read":78,"num_read_kb":63526,"num_write":1561,"num_write_kb":34080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"48'1512","reported_seq":3382,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.394396+0000","last_change":"2024-05-27T00:17:54.487572+0000","last_active":"2024-05-27T00:33:33.394396+0000","last_peered":"2024-05-27T00:33:33.394396+0000","last_clean":"2024-05-27T00:33:33.394396+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:33:33.394396+0000","last_undegraded":"2024-05-27T00:33:33.394396+0000","last_fullsized":"2024-05-27T00:33:33.394396+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:07:28.224179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010480461999999999,"stat_sum":{"num_bytes":6654877,"num_objects":588,"num_object_clones":271,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":191,"num_read":47,"num_read_kb":139,"num_write":1368,"num_write_kb":8104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"48'1671","reported_seq":3510,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.397671+0000","last_change":"2024-05-27T00:17:54.337799+0000","last_active":"2024-05-27T00:33:33.397671+0000","last_peered":"2024-05-27T00:33:33.397671+0000","last_clean":"2024-05-27T00:33:33.397671+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:33:33.397671+0000","last_undegraded":"2024-05-27T00:33:33.397671+0000","last_fullsized":"2024-05-27T00:33:33.397671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:13.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029916169999999999,"stat_sum":{"num_bytes":16039418,"num_objects":654,"num_object_clones":310,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":208,"num_read":54,"num_read_kb":28815,"num_write":1504,"num_write_kb":20996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"48'1603","reported_seq":3487,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.440010+0000","last_change":"2024-05-27T00:17:54.335854+0000","last_active":"2024-05-27T00:33:30.440010+0000","last_peered":"2024-05-27T00:33:30.440010+0000","last_clean":"2024-05-27T00:33:30.440010+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:33:30.440010+0000","last_undegraded":"2024-05-27T00:33:30.440010+0000","last_fullsized":"2024-05-27T00:33:30.440010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:13:02.845565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027334256000000001,"stat_sum":{"num_bytes":31314512,"num_objects":608,"num_object_clones":281,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":187,"num_read":70,"num_read_kb":56922,"num_write":1458,"num_write_kb":40262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"48'1614","reported_seq":3479,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.930688+0000","last_change":"2024-05-27T00:17:54.493382+0000","last_active":"2024-05-27T00:33:33.930688+0000","last_peered":"2024-05-27T00:33:33.930688+0000","last_clean":"2024-05-27T00:33:33.930688+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:33:33.930688+0000","last_undegraded":"2024-05-27T00:33:33.930688+0000","last_fullsized":"2024-05-27T00:33:33.930688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1614,"log_dups_size":0,"ondisk_log_size":1614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:11:52.944101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017108005999999999,"stat_sum":{"num_bytes":14791903,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":193,"num_read":59,"num_read_kb":12448,"num_write":1482,"num_write_kb":20672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"48'1678","reported_seq":3512,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.367645+0000","last_change":"2024-05-27T00:17:54.509854+0000","last_active":"2024-05-27T00:33:33.367645+0000","last_peered":"2024-05-27T00:33:33.367645+0000","last_clean":"2024-05-27T00:33:33.367645+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:33:33.367645+0000","last_undegraded":"2024-05-27T00:33:33.367645+0000","last_fullsized":"2024-05-27T00:33:33.367645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1678,"log_dups_size":0,"ondisk_log_size":1678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:13.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033862588999999998,"stat_sum":{"num_bytes":18976923,"num_objects":661,"num_object_clones":315,"num_object_copies":1322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":222,"num_read":54,"num_read_kb":29811,"num_write":1493,"num_write_kb":24680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"48'1529","reported_seq":3405,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.389080+0000","last_change":"2024-05-27T00:17:54.309499+0000","last_active":"2024-05-27T00:33:33.389080+0000","last_peered":"2024-05-27T00:33:33.389080+0000","last_clean":"2024-05-27T00:33:33.389080+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:33:33.389080+0000","last_undegraded":"2024-05-27T00:33:33.389080+0000","last_fullsized":"2024-05-27T00:33:33.389080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1529,"log_dups_size":0,"ondisk_log_size":1529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:32:55.895752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088470660000000007,"stat_sum":{"num_bytes":9565297,"num_objects":574,"num_object_clones":267,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":178,"num_read":48,"num_read_kb":13982,"num_write":1418,"num_write_kb":15615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"48'1569","reported_seq":3463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.893810+0000","last_change":"2024-05-27T00:17:54.509276+0000","last_active":"2024-05-27T00:33:29.893810+0000","last_peered":"2024-05-27T00:33:29.893810+0000","last_clean":"2024-05-27T00:33:29.893810+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:33:29.893810+0000","last_undegraded":"2024-05-27T00:33:29.893810+0000","last_fullsized":"2024-05-27T00:33:29.893810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:55:24.602133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033457270999999997,"stat_sum":{"num_bytes":30464560,"num_objects":585,"num_object_clones":269,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":175,"num_read":68,"num_read_kb":58155,"num_write":1439,"num_write_kb":31950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"48'1566","reported_seq":3425,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.854148+0000","last_change":"2024-05-27T00:17:54.310134+0000","last_active":"2024-05-27T00:33:29.854148+0000","last_peered":"2024-05-27T00:33:29.854148+0000","last_clean":"2024-05-27T00:33:29.854148+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:33:29.854148+0000","last_undegraded":"2024-05-27T00:33:29.854148+0000","last_fullsized":"2024-05-27T00:33:29.854148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1566,"log_dups_size":0,"ondisk_log_size":1566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:37:27.481263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089651499999999999,"stat_sum":{"num_bytes":14362147,"num_objects":603,"num_object_clones":285,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":194,"num_read":49,"num_read_kb":20561,"num_write":1422,"num_write_kb":17027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"48'1667","reported_seq":3529,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.893411+0000","last_change":"2024-05-27T00:17:54.337554+0000","last_active":"2024-05-27T00:33:33.893411+0000","last_peered":"2024-05-27T00:33:33.893411+0000","last_clean":"2024-05-27T00:33:33.893411+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:33:33.893411+0000","last_undegraded":"2024-05-27T00:33:33.893411+0000","last_fullsized":"2024-05-27T00:33:33.893411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1667,"log_dups_size":0,"ondisk_log_size":1667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:02:46.037448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024583906999999999,"stat_sum":{"num_bytes":23325444,"num_objects":624,"num_object_clones":288,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":196,"num_read":55,"num_read_kb":32562,"num_write":1514,"num_write_kb":26070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"48'1526","reported_seq":3399,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.912844+0000","last_change":"2024-05-27T00:17:54.491723+0000","last_active":"2024-05-27T00:33:33.912844+0000","last_peered":"2024-05-27T00:33:33.912844+0000","last_clean":"2024-05-27T00:33:33.912844+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:33:33.912844+0000","last_undegraded":"2024-05-27T00:33:33.912844+0000","last_fullsized":"2024-05-27T00:33:33.912844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:41:31.328434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016722006000000001,"stat_sum":{"num_bytes":10472871,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":180,"num_read":47,"num_read_kb":12352,"num_write":1385,"num_write_kb":12462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"48'1571","reported_seq":3430,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.392636+0000","last_change":"2024-05-27T00:17:54.309792+0000","last_active":"2024-05-27T00:33:33.392636+0000","last_peered":"2024-05-27T00:33:33.392636+0000","last_clean":"2024-05-27T00:33:33.392636+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:33:33.392636+0000","last_undegraded":"2024-05-27T00:33:33.392636+0000","last_fullsized":"2024-05-27T00:33:33.392636+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:03:52.656303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089478649999999993,"stat_sum":{"num_bytes":10328890,"num_objects":604,"num_object_clones":285,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":194,"num_read":49,"num_read_kb":15706,"num_write":1422,"num_write_kb":15432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"48'1703","reported_seq":3521,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.956715+0000","last_change":"2024-05-27T00:17:54.494236+0000","last_active":"2024-05-27T00:33:30.956715+0000","last_peered":"2024-05-27T00:33:30.956715+0000","last_clean":"2024-05-27T00:33:30.956715+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:33:30.956715+0000","last_undegraded":"2024-05-27T00:33:30.956715+0000","last_fullsized":"2024-05-27T00:33:30.956715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1703,"log_dups_size":0,"ondisk_log_size":1703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:17:24.056537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017911486000000001,"stat_sum":{"num_bytes":9846144,"num_objects":668,"num_object_clones":317,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":212,"num_read":40,"num_read_kb":1060,"num_write":1530,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"48'1604","reported_seq":3472,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.396278+0000","last_change":"2024-05-27T00:17:54.510031+0000","last_active":"2024-05-27T00:33:33.396278+0000","last_peered":"2024-05-27T00:33:33.396278+0000","last_clean":"2024-05-27T00:33:33.396278+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:33:33.396278+0000","last_undegraded":"2024-05-27T00:33:33.396278+0000","last_fullsized":"2024-05-27T00:33:33.396278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1604,"log_dups_size":0,"ondisk_log_size":1604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:09.263437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033387499000000001,"stat_sum":{"num_bytes":18975931,"num_objects":601,"num_object_clones":281,"num_object_copies":1202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":187,"num_read":54,"num_read_kb":24794,"num_write":1457,"num_write_kb":20478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"48'1651","reported_seq":3529,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.347644+0000","last_change":"2024-05-27T00:17:54.484908+0000","last_active":"2024-05-27T00:33:30.347644+0000","last_peered":"2024-05-27T00:33:30.347644+0000","last_clean":"2024-05-27T00:33:30.347644+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:33:30.347644+0000","last_undegraded":"2024-05-27T00:33:30.347644+0000","last_fullsized":"2024-05-27T00:33:30.347644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1651,"log_dups_size":0,"ondisk_log_size":1651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:10:39.525108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009527878,"stat_sum":{"num_bytes":27932857,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":194,"num_read":62,"num_read_kb":41489,"num_write":1506,"num_write_kb":37195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"48'1571","reported_seq":3467,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.389518+0000","last_change":"2024-05-27T00:17:54.331060+0000","last_active":"2024-05-27T00:33:33.389518+0000","last_peered":"2024-05-27T00:33:33.389518+0000","last_clean":"2024-05-27T00:33:33.389518+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:33:33.389518+0000","last_undegraded":"2024-05-27T00:33:33.389518+0000","last_fullsized":"2024-05-27T00:33:33.389518+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:05:48.629948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025435675000000001,"stat_sum":{"num_bytes":17932144,"num_objects":570,"num_object_clones":264,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":185,"num_read":65,"num_read_kb":34529,"num_write":1438,"num_write_kb":20974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"48'1632","reported_seq":3479,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.383849+0000","last_change":"2024-05-27T00:17:54.501574+0000","last_active":"2024-05-27T00:33:33.383849+0000","last_peered":"2024-05-27T00:33:33.383849+0000","last_clean":"2024-05-27T00:33:33.383849+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:33:33.383849+0000","last_undegraded":"2024-05-27T00:33:33.383849+0000","last_fullsized":"2024-05-27T00:33:33.383849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1632,"log_dups_size":0,"ondisk_log_size":1632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:16:03.345753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019320113,"stat_sum":{"num_bytes":26864770,"num_objects":637,"num_object_clones":297,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":204,"num_read":50,"num_read_kb":43416,"num_write":1495,"num_write_kb":28253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"48'1097","reported_seq":1312,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.639463+0000","last_change":"2024-05-27T00:15:24.611108+0000","last_active":"2024-05-27T00:33:34.639463+0000","last_peered":"2024-05-27T00:33:34.639463+0000","last_clean":"2024-05-27T00:33:34.639463+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:33:34.639463+0000","last_undegraded":"2024-05-27T00:33:34.639463+0000","last_fullsized":"2024-05-27T00:33:34.639463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:11.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":240,"num_read_kb":174989,"num_write":1187,"num_write_kb":92068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"48'1572","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.366164+0000","last_change":"2024-05-27T00:17:54.493715+0000","last_active":"2024-05-27T00:33:33.366164+0000","last_peered":"2024-05-27T00:33:33.366164+0000","last_clean":"2024-05-27T00:33:33.366164+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:33:33.366164+0000","last_undegraded":"2024-05-27T00:33:33.366164+0000","last_fullsized":"2024-05-27T00:33:33.366164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1572,"log_dups_size":0,"ondisk_log_size":1572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:50:38.769021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017111721,"stat_sum":{"num_bytes":21962235,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":198,"num_read":57,"num_read_kb":45093,"num_write":1410,"num_write_kb":24294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"48'843","reported_seq":1087,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.869212+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:33:33.869212+0000","last_peered":"2024-05-27T00:33:33.869212+0000","last_clean":"2024-05-27T00:33:33.869212+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:33:33.869212+0000","last_undegraded":"2024-05-27T00:33:33.869212+0000","last_fullsized":"2024-05-27T00:33:33.869212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":305,"num_read_kb":205128,"num_write":983,"num_write_kb":76341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":67,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"48'594","reported_seq":837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.992961+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:33:29.992961+0000","last_peered":"2024-05-27T00:33:29.992961+0000","last_clean":"2024-05-27T00:33:29.992961+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:33:29.992961+0000","last_undegraded":"2024-05-27T00:33:29.992961+0000","last_fullsized":"2024-05-27T00:33:29.992961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:46:08.645927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":317,"num_read_kb":129314,"num_write":740,"num_write_kb":59170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"48'1791","reported_seq":3623,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.885090+0000","last_change":"2024-05-27T00:17:54.309270+0000","last_active":"2024-05-27T00:33:33.885090+0000","last_peered":"2024-05-27T00:33:33.885090+0000","last_clean":"2024-05-27T00:33:33.885090+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:33:33.885090+0000","last_undegraded":"2024-05-27T00:33:33.885090+0000","last_fullsized":"2024-05-27T00:33:33.885090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1791,"log_dups_size":0,"ondisk_log_size":1791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:49:00.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087892630000000003,"stat_sum":{"num_bytes":15177983,"num_objects":665,"num_object_clones":314,"num_object_copies":1330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":214,"num_read":52,"num_read_kb":17293,"num_write":1633,"num_write_kb":25745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"48'1911","reported_seq":2158,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.499719+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:33:33.499719+0000","last_peered":"2024-05-27T00:33:33.499719+0000","last_clean":"2024-05-27T00:33:33.499719+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:33:33.499719+0000","last_undegraded":"2024-05-27T00:33:33.499719+0000","last_fullsized":"2024-05-27T00:33:33.499719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1911,"log_dups_size":0,"ondisk_log_size":1911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":310,"num_read_kb":210919,"num_write":2080,"num_write_kb":80661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"48'1555","reported_seq":3431,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.839742+0000","last_change":"2024-05-27T00:17:54.489271+0000","last_active":"2024-05-27T00:33:29.839742+0000","last_peered":"2024-05-27T00:33:29.839742+0000","last_clean":"2024-05-27T00:33:29.839742+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:33:29.839742+0000","last_undegraded":"2024-05-27T00:33:29.839742+0000","last_fullsized":"2024-05-27T00:33:29.839742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:49:06.355571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014633893,"stat_sum":{"num_bytes":23977264,"num_objects":602,"num_object_clones":282,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":198,"num_read":64,"num_read_kb":52608,"num_write":1398,"num_write_kb":30195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"48'592","reported_seq":854,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.900929+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:33:33.900929+0000","last_peered":"2024-05-27T00:33:33.900929+0000","last_clean":"2024-05-27T00:33:33.900929+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:33:33.900929+0000","last_undegraded":"2024-05-27T00:33:33.900929+0000","last_fullsized":"2024-05-27T00:33:33.900929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21338252,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":289,"num_read_kb":124309,"num_write":674,"num_write_kb":63278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"48'1500","reported_seq":3389,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.928969+0000","last_change":"2024-05-27T00:17:54.512612+0000","last_active":"2024-05-27T00:33:30.928969+0000","last_peered":"2024-05-27T00:33:30.928969+0000","last_clean":"2024-05-27T00:33:30.928969+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:33:30.928969+0000","last_undegraded":"2024-05-27T00:33:30.928969+0000","last_fullsized":"2024-05-27T00:33:30.928969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:07:00.308291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034220156000000002,"stat_sum":{"num_bytes":22845870,"num_objects":565,"num_object_clones":264,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":188,"num_read":59,"num_read_kb":32908,"num_write":1344,"num_write_kb":23966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"48'1486","reported_seq":1692,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.894653+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:33:33.894653+0000","last_peered":"2024-05-27T00:33:33.894653+0000","last_clean":"2024-05-27T00:33:33.894653+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:33:33.894653+0000","last_undegraded":"2024-05-27T00:33:33.894653+0000","last_fullsized":"2024-05-27T00:33:33.894653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":244,"num_read_kb":161522,"num_write":1589,"num_write_kb":88109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"48'1752","reported_seq":3524,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.367187+0000","last_change":"2024-05-27T00:17:54.493629+0000","last_active":"2024-05-27T00:33:33.367187+0000","last_peered":"2024-05-27T00:33:33.367187+0000","last_clean":"2024-05-27T00:33:33.367187+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:33:33.367187+0000","last_undegraded":"2024-05-27T00:33:33.367187+0000","last_fullsized":"2024-05-27T00:33:33.367187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1752,"log_dups_size":0,"ondisk_log_size":1752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:26:59.993852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017358472,"stat_sum":{"num_bytes":15481537,"num_objects":707,"num_object_clones":381,"num_object_copies":1414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":707,"num_whiteouts":205,"num_read":59,"num_read_kb":17601,"num_write":1504,"num_write_kb":17792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"48'1524","reported_seq":1820,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.936597+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:33:33.936597+0000","last_peered":"2024-05-27T00:33:33.936597+0000","last_clean":"2024-05-27T00:33:33.936597+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:33:33.936597+0000","last_undegraded":"2024-05-27T00:33:33.936597+0000","last_fullsized":"2024-05-27T00:33:33.936597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":428,"num_read_kb":130260,"num_write":1739,"num_write_kb":60710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"48'1583","reported_seq":3446,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.901089+0000","last_change":"2024-05-27T00:17:54.506220+0000","last_active":"2024-05-27T00:33:30.901089+0000","last_peered":"2024-05-27T00:33:30.901089+0000","last_clean":"2024-05-27T00:33:30.901089+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:33:30.901089+0000","last_undegraded":"2024-05-27T00:33:30.901089+0000","last_fullsized":"2024-05-27T00:33:30.901089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1583,"log_dups_size":0,"ondisk_log_size":1583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:32:01.052698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022583427,"stat_sum":{"num_bytes":13475539,"num_objects":608,"num_object_clones":285,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":200,"num_read":54,"num_read_kb":12130,"num_write":1429,"num_write_kb":19813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"48'1064","reported_seq":1346,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.872488+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:33:33.872488+0000","last_peered":"2024-05-27T00:33:33.872488+0000","last_clean":"2024-05-27T00:33:33.872488+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:33:33.872488+0000","last_undegraded":"2024-05-27T00:33:33.872488+0000","last_fullsized":"2024-05-27T00:33:33.872488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555104,"num_objects":77,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":365,"num_read_kb":194738,"num_write":1216,"num_write_kb":100979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"48'1567","reported_seq":3423,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.931447+0000","last_change":"2024-05-27T00:17:54.336019+0000","last_active":"2024-05-27T00:33:33.931447+0000","last_peered":"2024-05-27T00:33:33.931447+0000","last_clean":"2024-05-27T00:33:33.931447+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:33:33.931447+0000","last_undegraded":"2024-05-27T00:33:33.931447+0000","last_fullsized":"2024-05-27T00:33:33.931447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:48.582600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.030558409000000002,"stat_sum":{"num_bytes":10200444,"num_objects":601,"num_object_clones":280,"num_object_copies":1202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":188,"num_read":42,"num_read_kb":12368,"num_write":1435,"num_write_kb":13032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"48'830","reported_seq":1084,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.892943+0000","last_change":"2024-05-27T00:15:24.605986+0000","last_active":"2024-05-27T00:33:33.892943+0000","last_peered":"2024-05-27T00:33:33.892943+0000","last_clean":"2024-05-27T00:33:33.892943+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:33:33.892943+0000","last_undegraded":"2024-05-27T00:33:33.892943+0000","last_fullsized":"2024-05-27T00:33:33.892943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971554,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":336,"num_read_kb":131285,"num_write":974,"num_write_kb":63316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":56,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"48'1706","reported_seq":3516,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.390563+0000","last_change":"2024-05-27T00:17:54.493027+0000","last_active":"2024-05-27T00:33:33.390563+0000","last_peered":"2024-05-27T00:33:33.390563+0000","last_clean":"2024-05-27T00:33:33.390563+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:33:33.390563+0000","last_undegraded":"2024-05-27T00:33:33.390563+0000","last_fullsized":"2024-05-27T00:33:33.390563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1706,"log_dups_size":0,"ondisk_log_size":1706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:38:11.155792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018271039999999999,"stat_sum":{"num_bytes":20173932,"num_objects":652,"num_object_clones":356,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":187,"num_read":72,"num_read_kb":36313,"num_write":1471,"num_write_kb":31527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"48'502","reported_seq":699,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.529271+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:33:34.529271+0000","last_peered":"2024-05-27T00:33:34.529271+0000","last_clean":"2024-05-27T00:33:34.529271+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:33:34.529271+0000","last_undegraded":"2024-05-27T00:33:34.529271+0000","last_fullsized":"2024-05-27T00:33:34.529271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":232,"num_read_kb":71610,"num_write":625,"num_write_kb":47247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"48'1677","reported_seq":3546,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.955606+0000","last_change":"2024-05-27T00:17:54.334890+0000","last_active":"2024-05-27T00:33:30.955606+0000","last_peered":"2024-05-27T00:33:30.955606+0000","last_clean":"2024-05-27T00:33:30.955606+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:33:30.955606+0000","last_undegraded":"2024-05-27T00:33:30.955606+0000","last_fullsized":"2024-05-27T00:33:30.955606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1677,"log_dups_size":0,"ondisk_log_size":1677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:55.040370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028694234999999998,"stat_sum":{"num_bytes":27706000,"num_objects":637,"num_object_clones":296,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":195,"num_read":71,"num_read_kb":46547,"num_write":1532,"num_write_kb":29714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"48'364","reported_seq":562,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.779855+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:33:29.779855+0000","last_peered":"2024-05-27T00:33:29.779855+0000","last_clean":"2024-05-27T00:33:29.779855+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:33:29.779855+0000","last_undegraded":"2024-05-27T00:33:29.779855+0000","last_fullsized":"2024-05-27T00:33:29.779855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":244,"num_read_kb":99922,"num_write":491,"num_write_kb":51118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"48'1573","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.391959+0000","last_change":"2024-05-27T00:17:54.332802+0000","last_active":"2024-05-27T00:33:33.391959+0000","last_peered":"2024-05-27T00:33:33.391959+0000","last_clean":"2024-05-27T00:33:33.391959+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:33:33.391959+0000","last_undegraded":"2024-05-27T00:33:33.391959+0000","last_fullsized":"2024-05-27T00:33:33.391959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:41:31.328434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025579069999999999,"stat_sum":{"num_bytes":19680366,"num_objects":592,"num_object_clones":272,"num_object_copies":1184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":592,"num_whiteouts":185,"num_read":43,"num_read_kb":14554,"num_write":1424,"num_write_kb":24215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"48'749","reported_seq":1035,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.864326+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:33:33.864326+0000","last_peered":"2024-05-27T00:33:33.864326+0000","last_clean":"2024-05-27T00:33:33.864326+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:33:33.864326+0000","last_undegraded":"2024-05-27T00:33:33.864326+0000","last_fullsized":"2024-05-27T00:33:33.864326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":749,"log_dups_size":0,"ondisk_log_size":749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":479,"num_read_kb":120710,"num_write":984,"num_write_kb":72758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"48'1617","reported_seq":3492,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.972082+0000","last_change":"2024-05-27T00:17:54.334829+0000","last_active":"2024-05-27T00:33:30.972082+0000","last_peered":"2024-05-27T00:33:30.972082+0000","last_clean":"2024-05-27T00:33:30.972082+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:33:30.972082+0000","last_undegraded":"2024-05-27T00:33:30.972082+0000","last_fullsized":"2024-05-27T00:33:30.972082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1617,"log_dups_size":0,"ondisk_log_size":1617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:00:47.756042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028937418999999999,"stat_sum":{"num_bytes":19454539,"num_objects":599,"num_object_clones":279,"num_object_copies":1198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":599,"num_whiteouts":201,"num_read":60,"num_read_kb":39320,"num_write":1444,"num_write_kb":29954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"48'926","reported_seq":1176,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.639917+0000","last_change":"2024-05-27T00:15:24.601924+0000","last_active":"2024-05-27T00:33:34.639917+0000","last_peered":"2024-05-27T00:33:34.639917+0000","last_clean":"2024-05-27T00:33:34.639917+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:33:34.639917+0000","last_undegraded":"2024-05-27T00:33:34.639917+0000","last_fullsized":"2024-05-27T00:33:34.639917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":926,"log_dups_size":0,"ondisk_log_size":926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:32.185839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971542,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":329,"num_read_kb":104768,"num_write":1076,"num_write_kb":59606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"48'1603","reported_seq":3463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.427204+0000","last_change":"2024-05-27T00:17:54.335438+0000","last_active":"2024-05-27T00:33:30.427204+0000","last_peered":"2024-05-27T00:33:30.427204+0000","last_clean":"2024-05-27T00:33:30.427204+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:33:30.427204+0000","last_undegraded":"2024-05-27T00:33:30.427204+0000","last_fullsized":"2024-05-27T00:33:30.427204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:44.778493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029056843999999998,"stat_sum":{"num_bytes":27639049,"num_objects":630,"num_object_clones":290,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":189,"num_read":55,"num_read_kb":39571,"num_write":1460,"num_write_kb":33520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"48'850","reported_seq":1063,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:28.880655+0000","last_change":"2024-05-27T00:15:24.600878+0000","last_active":"2024-05-27T00:33:28.880655+0000","last_peered":"2024-05-27T00:33:28.880655+0000","last_clean":"2024-05-27T00:33:28.880655+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:33:28.880655+0000","last_undegraded":"2024-05-27T00:33:28.880655+0000","last_fullsized":"2024-05-27T00:33:28.880655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":241,"num_read_kb":194399,"num_write":970,"num_write_kb":84408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"48'1539","reported_seq":3419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.936151+0000","last_change":"2024-05-27T00:17:54.337688+0000","last_active":"2024-05-27T00:33:30.936151+0000","last_peered":"2024-05-27T00:33:30.936151+0000","last_clean":"2024-05-27T00:33:30.936151+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:33:30.936151+0000","last_undegraded":"2024-05-27T00:33:30.936151+0000","last_fullsized":"2024-05-27T00:33:30.936151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1539,"log_dups_size":0,"ondisk_log_size":1539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:09.263437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028414170999999998,"stat_sum":{"num_bytes":22590275,"num_objects":587,"num_object_clones":272,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":186,"num_read":57,"num_read_kb":45093,"num_write":1409,"num_write_kb":28119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"48'1073","reported_seq":1310,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.705095+0000","last_change":"2024-05-27T00:15:24.603452+0000","last_active":"2024-05-27T00:33:34.705095+0000","last_peered":"2024-05-27T00:33:34.705095+0000","last_clean":"2024-05-27T00:33:34.705095+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:33:34.705095+0000","last_undegraded":"2024-05-27T00:33:34.705095+0000","last_fullsized":"2024-05-27T00:33:34.705095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1073,"log_dups_size":0,"ondisk_log_size":1073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34922452,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":293,"num_read_kb":166112,"num_write":1227,"num_write_kb":85323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"48'1558","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.931551+0000","last_change":"2024-05-27T00:17:54.512510+0000","last_active":"2024-05-27T00:33:30.931551+0000","last_peered":"2024-05-27T00:33:30.931551+0000","last_clean":"2024-05-27T00:33:30.931551+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:33:30.931551+0000","last_undegraded":"2024-05-27T00:33:30.931551+0000","last_fullsized":"2024-05-27T00:33:30.931551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1558,"log_dups_size":0,"ondisk_log_size":1558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:45.104337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034804740000000001,"stat_sum":{"num_bytes":14041732,"num_objects":561,"num_object_clones":257,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":186,"num_read":41,"num_read_kb":16491,"num_write":1423,"num_write_kb":17133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"48'755","reported_seq":983,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.895276+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:33:33.895276+0000","last_peered":"2024-05-27T00:33:33.895276+0000","last_clean":"2024-05-27T00:33:33.895276+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:33:33.895276+0000","last_undegraded":"2024-05-27T00:33:33.895276+0000","last_fullsized":"2024-05-27T00:33:33.895276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":292,"num_read_kb":141638,"num_write":926,"num_write_kb":64367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"48'1568","reported_seq":3422,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.895827+0000","last_change":"2024-05-27T00:17:54.333811+0000","last_active":"2024-05-27T00:33:33.895827+0000","last_peered":"2024-05-27T00:33:33.895827+0000","last_clean":"2024-05-27T00:33:33.895827+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:33:33.895827+0000","last_undegraded":"2024-05-27T00:33:33.895827+0000","last_fullsized":"2024-05-27T00:33:33.895827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:14.148909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028015244000000002,"stat_sum":{"num_bytes":15735198,"num_objects":613,"num_object_clones":289,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":202,"num_read":48,"num_read_kb":16478,"num_write":1397,"num_write_kb":22754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":231,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:32:10.519872+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:32:10.519872+0000","last_peered":"2024-05-27T00:32:10.519872+0000","last_clean":"2024-05-27T00:32:10.519872+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:32:10.519872+0000","last_undegraded":"2024-05-27T00:32:10.519872+0000","last_fullsized":"2024-05-27T00:32:10.519872+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"48'875","reported_seq":1072,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.499832+0000","last_change":"2024-05-27T00:15:24.603781+0000","last_active":"2024-05-27T00:33:33.499832+0000","last_peered":"2024-05-27T00:33:33.499832+0000","last_clean":"2024-05-27T00:33:33.499832+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:33:33.499832+0000","last_undegraded":"2024-05-27T00:33:33.499832+0000","last_fullsized":"2024-05-27T00:33:33.499832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":219,"num_read_kb":131843,"num_write":965,"num_write_kb":67539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"48'1616","reported_seq":3478,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.386594+0000","last_change":"2024-05-27T00:17:54.309203+0000","last_active":"2024-05-27T00:33:33.386594+0000","last_peered":"2024-05-27T00:33:33.386594+0000","last_clean":"2024-05-27T00:33:33.386594+0000","last_became_active":"2024-05-27T00:16:06.517816+0000","last_became_peered":"2024-05-27T00:16:06.517816+0000","last_unstale":"2024-05-27T00:33:33.386594+0000","last_undegraded":"2024-05-27T00:33:33.386594+0000","last_fullsized":"2024-05-27T00:33:33.386594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:08.300384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085828740000000008,"stat_sum":{"num_bytes":27686043,"num_objects":633,"num_object_clones":292,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":198,"num_read":54,"num_read_kb":34546,"num_write":1470,"num_write_kb":32830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"48'429","reported_seq":666,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.891792+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:33:33.891792+0000","last_peered":"2024-05-27T00:33:33.891792+0000","last_clean":"2024-05-27T00:33:33.891792+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:33:33.891792+0000","last_undegraded":"2024-05-27T00:33:33.891792+0000","last_fullsized":"2024-05-27T00:33:33.891792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":299,"num_read_kb":174228,"num_write":552,"num_write_kb":63235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"48'1520","reported_seq":3388,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.382336+0000","last_change":"2024-05-27T00:17:54.490594+0000","last_active":"2024-05-27T00:33:33.382336+0000","last_peered":"2024-05-27T00:33:33.382336+0000","last_clean":"2024-05-27T00:33:33.382336+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:33:33.382336+0000","last_undegraded":"2024-05-27T00:33:33.382336+0000","last_fullsized":"2024-05-27T00:33:33.382336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:45:47.096794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016778365999999999,"stat_sum":{"num_bytes":6046316,"num_objects":564,"num_object_clones":261,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":182,"num_read":34,"num_read_kb":53,"num_write":1394,"num_write_kb":7687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'1654","reported_seq":2540,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.495949+0000","last_change":"2024-05-27T00:15:24.598182+0000","last_active":"2024-05-27T00:33:33.495949+0000","last_peered":"2024-05-27T00:33:33.495949+0000","last_clean":"2024-05-27T00:33:33.495949+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:33:33.495949+0000","last_undegraded":"2024-05-27T00:33:33.495949+0000","last_fullsized":"2024-05-27T00:33:33.495949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1654,"log_dups_size":0,"ondisk_log_size":1654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":85,"num_object_clones":0,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":1008,"num_read_kb":169683,"num_write":1827,"num_write_kb":85277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"48'1615","reported_seq":3473,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.909399+0000","last_change":"2024-05-27T00:17:54.491784+0000","last_active":"2024-05-27T00:33:33.909399+0000","last_peered":"2024-05-27T00:33:33.909399+0000","last_clean":"2024-05-27T00:33:33.909399+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:33:33.909399+0000","last_undegraded":"2024-05-27T00:33:33.909399+0000","last_fullsized":"2024-05-27T00:33:33.909399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:22.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016042048999999999,"stat_sum":{"num_bytes":11249311,"num_objects":630,"num_object_clones":289,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":203,"num_read":51,"num_read_kb":12401,"num_write":1463,"num_write_kb":13659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"48'708","reported_seq":967,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.358425+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:33:30.358425+0000","last_peered":"2024-05-27T00:33:30.358425+0000","last_clean":"2024-05-27T00:33:30.358425+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:33:30.358425+0000","last_undegraded":"2024-05-27T00:33:30.358425+0000","last_fullsized":"2024-05-27T00:33:30.358425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":344,"num_read_kb":168323,"num_write":926,"num_write_kb":68366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"48'1642","reported_seq":3512,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.350257+0000","last_change":"2024-05-27T00:17:54.505548+0000","last_active":"2024-05-27T00:33:30.350257+0000","last_peered":"2024-05-27T00:33:30.350257+0000","last_clean":"2024-05-27T00:33:30.350257+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:33:30.350257+0000","last_undegraded":"2024-05-27T00:33:30.350257+0000","last_fullsized":"2024-05-27T00:33:30.350257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:08.630844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021175818999999999,"stat_sum":{"num_bytes":27156672,"num_objects":598,"num_object_clones":274,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":182,"num_read":49,"num_read_kb":37771,"num_write":1519,"num_write_kb":29445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"48'814","reported_seq":1035,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.885329+0000","last_change":"2024-05-27T00:15:24.603518+0000","last_active":"2024-05-27T00:33:33.885329+0000","last_peered":"2024-05-27T00:33:33.885329+0000","last_clean":"2024-05-27T00:33:33.885329+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:33:33.885329+0000","last_undegraded":"2024-05-27T00:33:33.885329+0000","last_fullsized":"2024-05-27T00:33:33.885329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":262,"num_read_kb":165581,"num_write":935,"num_write_kb":83695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"48'1593","reported_seq":3458,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.920096+0000","last_change":"2024-05-27T00:17:54.511904+0000","last_active":"2024-05-27T00:33:30.920096+0000","last_peered":"2024-05-27T00:33:30.920096+0000","last_clean":"2024-05-27T00:33:30.920096+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:33:30.920096+0000","last_undegraded":"2024-05-27T00:33:30.920096+0000","last_fullsized":"2024-05-27T00:33:30.920096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1593,"log_dups_size":0,"ondisk_log_size":1593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:58:08.710145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034092800999999999,"stat_sum":{"num_bytes":22484461,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":211,"num_read":50,"num_read_kb":38442,"num_write":1413,"num_write_kb":28463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"48'1412","reported_seq":1652,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.537498+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:33:34.537498+0000","last_peered":"2024-05-27T00:33:34.537498+0000","last_clean":"2024-05-27T00:33:34.537498+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:33:34.537498+0000","last_undegraded":"2024-05-27T00:33:34.537498+0000","last_fullsized":"2024-05-27T00:33:34.537498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1412,"log_dups_size":0,"ondisk_log_size":1412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":299,"num_read_kb":161725,"num_write":1529,"num_write_kb":67545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"48'1569","reported_seq":3448,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.363078+0000","last_change":"2024-05-27T00:17:54.488788+0000","last_active":"2024-05-27T00:33:33.363078+0000","last_peered":"2024-05-27T00:33:33.363078+0000","last_clean":"2024-05-27T00:33:33.363078+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:33:33.363078+0000","last_undegraded":"2024-05-27T00:33:33.363078+0000","last_fullsized":"2024-05-27T00:33:33.363078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:42:30.595550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014512239,"stat_sum":{"num_bytes":23092289,"num_objects":584,"num_object_clones":267,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":187,"num_read":51,"num_read_kb":31023,"num_write":1448,"num_write_kb":28334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'833","reported_seq":1070,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.637756+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:33:34.637756+0000","last_peered":"2024-05-27T00:33:34.637756+0000","last_clean":"2024-05-27T00:33:34.637756+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:33:34.637756+0000","last_undegraded":"2024-05-27T00:33:34.637756+0000","last_fullsized":"2024-05-27T00:33:34.637756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":833,"log_dups_size":0,"ondisk_log_size":833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":312,"num_read_kb":117165,"num_write":1596,"num_write_kb":59298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"48'1511","reported_seq":3386,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.500418+0000","last_change":"2024-05-27T00:17:54.310241+0000","last_active":"2024-05-27T00:33:30.500418+0000","last_peered":"2024-05-27T00:33:30.500418+0000","last_clean":"2024-05-27T00:33:30.500418+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:33:30.500418+0000","last_undegraded":"2024-05-27T00:33:30.500418+0000","last_fullsized":"2024-05-27T00:33:30.500418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:51.525130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089509419999999999,"stat_sum":{"num_bytes":13250576,"num_objects":583,"num_object_clones":269,"num_object_copies":1166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":181,"num_read":49,"num_read_kb":2453,"num_write":1381,"num_write_kb":19589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"48'643","reported_seq":895,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.706802+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:33:34.706802+0000","last_peered":"2024-05-27T00:33:34.706802+0000","last_clean":"2024-05-27T00:33:34.706802+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:33:34.706802+0000","last_undegraded":"2024-05-27T00:33:34.706802+0000","last_fullsized":"2024-05-27T00:33:34.706802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":352,"num_read_kb":95868,"num_write":810,"num_write_kb":47576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"48'1573","reported_seq":3446,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.501372+0000","last_change":"2024-05-27T00:17:54.492705+0000","last_active":"2024-05-27T00:33:30.501372+0000","last_peered":"2024-05-27T00:33:30.501372+0000","last_clean":"2024-05-27T00:33:30.501372+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:33:30.501372+0000","last_undegraded":"2024-05-27T00:33:30.501372+0000","last_fullsized":"2024-05-27T00:33:30.501372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:40:53.350258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018225775999999999,"stat_sum":{"num_bytes":9758467,"num_objects":579,"num_object_clones":270,"num_object_copies":1158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":186,"num_read":49,"num_read_kb":4155,"num_write":1423,"num_write_kb":16751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'471","reported_seq":692,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.886300+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:33:30.886300+0000","last_peered":"2024-05-27T00:33:30.886300+0000","last_clean":"2024-05-27T00:33:30.886300+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:33:30.886300+0000","last_undegraded":"2024-05-27T00:33:30.886300+0000","last_fullsized":"2024-05-27T00:33:30.886300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":297,"num_read_kb":59216,"num_write":633,"num_write_kb":42916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":70,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"48'1624","reported_seq":3487,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.893727+0000","last_change":"2024-05-27T00:17:54.504734+0000","last_active":"2024-05-27T00:33:33.893727+0000","last_peered":"2024-05-27T00:33:33.893727+0000","last_clean":"2024-05-27T00:33:33.893727+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:33:33.893727+0000","last_undegraded":"2024-05-27T00:33:33.893727+0000","last_fullsized":"2024-05-27T00:33:33.893727+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:34:00.119528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022802596000000001,"stat_sum":{"num_bytes":19569863,"num_objects":620,"num_object_clones":284,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":196,"num_read":52,"num_read_kb":30708,"num_write":1470,"num_write_kb":21076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"48'783","reported_seq":953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.177344+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:33:30.177344+0000","last_peered":"2024-05-27T00:33:30.177344+0000","last_clean":"2024-05-27T00:33:30.177344+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:33:30.177344+0000","last_undegraded":"2024-05-27T00:33:30.177344+0000","last_fullsized":"2024-05-27T00:33:30.177344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":783,"log_dups_size":0,"ondisk_log_size":783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":176,"num_read_kb":80391,"num_write":878,"num_write_kb":51066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"48'1663","reported_seq":3525,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.385750+0000","last_change":"2024-05-27T00:17:54.308939+0000","last_active":"2024-05-27T00:33:33.385750+0000","last_peered":"2024-05-27T00:33:33.385750+0000","last_clean":"2024-05-27T00:33:33.385750+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:33:33.385750+0000","last_undegraded":"2024-05-27T00:33:33.385750+0000","last_fullsized":"2024-05-27T00:33:33.385750+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:17:28.826160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084477519999999993,"stat_sum":{"num_bytes":19800736,"num_objects":636,"num_object_clones":295,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":205,"num_read":62,"num_read_kb":32956,"num_write":1508,"num_write_kb":26318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"48'1047","reported_seq":1271,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.532477+0000","last_change":"2024-05-27T00:15:24.598238+0000","last_active":"2024-05-27T00:33:34.532477+0000","last_peered":"2024-05-27T00:33:34.532477+0000","last_clean":"2024-05-27T00:33:34.532477+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:33:34.532477+0000","last_undegraded":"2024-05-27T00:33:34.532477+0000","last_fullsized":"2024-05-27T00:33:34.532477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1047,"log_dups_size":0,"ondisk_log_size":1047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":269,"num_read_kb":153672,"num_write":1132,"num_write_kb":75680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"48'1615","reported_seq":3456,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.386744+0000","last_change":"2024-05-27T00:17:54.310021+0000","last_active":"2024-05-27T00:33:33.386744+0000","last_peered":"2024-05-27T00:33:33.386744+0000","last_clean":"2024-05-27T00:33:33.386744+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:33:33.386744+0000","last_undegraded":"2024-05-27T00:33:33.386744+0000","last_fullsized":"2024-05-27T00:33:33.386744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:07:28.174929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091147599999999995,"stat_sum":{"num_bytes":15827867,"num_objects":624,"num_object_clones":289,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":199,"num_read":35,"num_read_kb":8228,"num_write":1456,"num_write_kb":20842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"48'689","reported_seq":942,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.707187+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:33:34.707187+0000","last_peered":"2024-05-27T00:33:34.707187+0000","last_clean":"2024-05-27T00:33:34.707187+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:33:34.707187+0000","last_undegraded":"2024-05-27T00:33:34.707187+0000","last_fullsized":"2024-05-27T00:33:34.707187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":689,"log_dups_size":0,"ondisk_log_size":689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":331,"num_read_kb":161875,"num_write":831,"num_write_kb":80242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"48'1587","reported_seq":3445,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.396200+0000","last_change":"2024-05-27T00:17:54.333938+0000","last_active":"2024-05-27T00:33:30.396200+0000","last_peered":"2024-05-27T00:33:30.396200+0000","last_clean":"2024-05-27T00:33:30.396200+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:33:30.396200+0000","last_undegraded":"2024-05-27T00:33:30.396200+0000","last_fullsized":"2024-05-27T00:33:30.396200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1587,"log_dups_size":0,"ondisk_log_size":1587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:22.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026544363000000001,"stat_sum":{"num_bytes":11103766,"num_objects":576,"num_object_clones":273,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":173,"num_read":36,"num_read_kb":3645,"num_write":1460,"num_write_kb":12414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"48'1524","reported_seq":3388,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.922267+0000","last_change":"2024-05-27T00:17:54.485932+0000","last_active":"2024-05-27T00:33:33.922267+0000","last_peered":"2024-05-27T00:33:33.922267+0000","last_clean":"2024-05-27T00:33:33.922267+0000","last_became_active":"2024-05-27T00:16:06.811922+0000","last_became_peered":"2024-05-27T00:16:06.811922+0000","last_unstale":"2024-05-27T00:33:33.922267+0000","last_undegraded":"2024-05-27T00:33:33.922267+0000","last_fullsized":"2024-05-27T00:33:33.922267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:29:39.727324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011085174,"stat_sum":{"num_bytes":14346371,"num_objects":587,"num_object_clones":272,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":198,"num_read":48,"num_read_kb":4199,"num_write":1374,"num_write_kb":25831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"48'490","reported_seq":703,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.794312+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:33:34.794312+0000","last_peered":"2024-05-27T00:33:34.794312+0000","last_clean":"2024-05-27T00:33:34.794312+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:33:34.794312+0000","last_undegraded":"2024-05-27T00:33:34.794312+0000","last_fullsized":"2024-05-27T00:33:34.794312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":267,"num_read_kb":124158,"num_write":632,"num_write_kb":63683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"48'1663","reported_seq":3491,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.392971+0000","last_change":"2024-05-27T00:17:54.511007+0000","last_active":"2024-05-27T00:33:33.392971+0000","last_peered":"2024-05-27T00:33:33.392971+0000","last_clean":"2024-05-27T00:33:33.392971+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:33:33.392971+0000","last_undegraded":"2024-05-27T00:33:33.392971+0000","last_fullsized":"2024-05-27T00:33:33.392971+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:43:33.746539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034279472999999998,"stat_sum":{"num_bytes":10536557,"num_objects":647,"num_object_clones":309,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":224,"num_read":43,"num_read_kb":11171,"num_write":1484,"num_write_kb":16947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"48'860","reported_seq":1117,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.539304+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:33:34.539304+0000","last_peered":"2024-05-27T00:33:34.539304+0000","last_clean":"2024-05-27T00:33:34.539304+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:33:34.539304+0000","last_undegraded":"2024-05-27T00:33:34.539304+0000","last_fullsized":"2024-05-27T00:33:34.539304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":860,"log_dups_size":0,"ondisk_log_size":860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":337,"num_read_kb":206604,"num_write":984,"num_write_kb":108700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"48'1538","reported_seq":3409,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.343921+0000","last_change":"2024-05-27T00:17:54.493359+0000","last_active":"2024-05-27T00:33:30.343921+0000","last_peered":"2024-05-27T00:33:30.343921+0000","last_clean":"2024-05-27T00:33:30.343921+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:33:30.343921+0000","last_undegraded":"2024-05-27T00:33:30.343921+0000","last_fullsized":"2024-05-27T00:33:30.343921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1538,"log_dups_size":0,"ondisk_log_size":1538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:32:56.589612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017325917,"stat_sum":{"num_bytes":22746478,"num_objects":600,"num_object_clones":283,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":186,"num_read":59,"num_read_kb":40021,"num_write":1401,"num_write_kb":24139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"48'1273","reported_seq":1530,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.702528+0000","last_change":"2024-05-27T00:15:24.600571+0000","last_active":"2024-05-27T00:33:34.702528+0000","last_peered":"2024-05-27T00:33:34.702528+0000","last_clean":"2024-05-27T00:33:34.702528+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:33:34.702528+0000","last_undegraded":"2024-05-27T00:33:34.702528+0000","last_fullsized":"2024-05-27T00:33:34.702528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1273,"log_dups_size":0,"ondisk_log_size":1273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:44:21.645223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":353,"num_read_kb":145683,"num_write":1459,"num_write_kb":63890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"48'1559","reported_seq":3435,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.394563+0000","last_change":"2024-05-27T00:17:54.309879+0000","last_active":"2024-05-27T00:33:33.394563+0000","last_peered":"2024-05-27T00:33:33.394563+0000","last_clean":"2024-05-27T00:33:33.394563+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:33:33.394563+0000","last_undegraded":"2024-05-27T00:33:33.394563+0000","last_fullsized":"2024-05-27T00:33:33.394563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1559,"log_dups_size":0,"ondisk_log_size":1559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:20:48.824140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089714449999999998,"stat_sum":{"num_bytes":23504918,"num_objects":585,"num_object_clones":273,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":188,"num_read":53,"num_read_kb":39809,"num_write":1415,"num_write_kb":24852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"48'514","reported_seq":753,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.528633+0000","last_change":"2024-05-27T00:15:24.615066+0000","last_active":"2024-05-27T00:33:34.528633+0000","last_peered":"2024-05-27T00:33:34.528633+0000","last_clean":"2024-05-27T00:33:34.528633+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:33:34.528633+0000","last_undegraded":"2024-05-27T00:33:34.528633+0000","last_fullsized":"2024-05-27T00:33:34.528633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":327,"num_read_kb":63931,"num_write":692,"num_write_kb":47409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1174241513,"num_objects":39009,"num_object_clones":18298,"num_object_copies":78018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39009,"num_whiteouts":12489,"num_read":3379,"num_read_kb":1679737,"num_write":92416,"num_write_kb":1475210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2440749056,"data_stored":2350449106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102162,"ondisk_log_size":102162,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":781876696,"num_objects":2104,"num_object_clones":0,"num_object_copies":4208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":10396,"num_read_kb":4541569,"num_write":33862,"num_write_kb":2225576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1834,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1563811840,"data_stored":1563753392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28655,"ondisk_log_size":28655,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542368,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746260,"kb_used_data":606860,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93625580,"statfs":{"total":96636764160,"available":95872593920,"internally_reserved":0,"allocated":621424640,"data_stored":608366707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142735818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542369,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573040,"kb_used_data":443368,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93798800,"statfs":{"total":96636764160,"available":96049971200,"internally_reserved":0,"allocated":454008832,"data_stored":445134014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542370,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741064,"kb_used_data":583616,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93630776,"statfs":{"total":96636764160,"available":95877914624,"internally_reserved":0,"allocated":597622784,"data_stored":584115090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161216965},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542368,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662452,"kb_used_data":518892,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93709388,"statfs":{"total":96636764160,"available":95958413312,"internally_reserved":0,"allocated":531345408,"data_stored":517393413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542369,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":656408,"kb_used_data":521168,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93715432,"statfs":{"total":96636764160,"available":95964602368,"internally_reserved":0,"allocated":533676032,"data_stored":519848185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138475975},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542368,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439708,"kb_used_data":336404,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93932132,"statfs":{"total":96636764160,"available":96186503168,"internally_reserved":0,"allocated":344477696,"data_stored":336181610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542369,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573740,"kb_used_data":426020,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93798100,"statfs":{"total":96636764160,"available":96049254400,"internally_reserved":0,"allocated":436244480,"data_stored":428173666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151255495},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542368,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623352,"kb_used_data":479792,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93748488,"statfs":{"total":96636764160,"available":95998451712,"internally_reserved":0,"allocated":491307008,"data_stored":478155533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167784448,"data_stored":167772798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":223674368,"data_stored":223666222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125829120,"data_stored":125829120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192946176,"data_stored":192938656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":160763904,"data_stored":160751628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":214282240,"data_stored":214276326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":222674944,"data_stored":222665516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":255856640,"data_stored":255853126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":322383872,"data_stored":309544244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212025344,"data_stored":204259321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218103808,"data_stored":210104367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340185088,"data_stored":326661406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":370036736,"data_stored":356393662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":382795776,"data_stored":369590641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":230789120,"data_stored":222220375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":364429312,"data_stored":351675090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:33:41.222 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-27T00:33:41.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:33:41.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.394+0000 7f8ca5400640 1 Processor -- start 2024-05-27T00:33:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.394+0000 7f8ca5400640 1 -- start start 2024-05-27T00:33:41.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.394+0000 7f8ca5400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8ca0154a80 0x7f8ca0072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.394+0000 7f8ca5400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca015e6b0 0x7f8ca006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.394+0000 7f8ca5400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ca00700d0 con 0x7f8ca0062720 2024-05-27T00:33:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.394+0000 7f8ca5400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f8ca00726a0 con 0x7f8ca0154a80 2024-05-27T00:33:41.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.394+0000 7f8ca5400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca0071a90 con 0x7f8ca015e6b0 2024-05-27T00:33:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.394+0000 7f8c9f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca015e6b0 0x7f8ca006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.398+0000 7f8c9e800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f8ca0154a80 0x7f8ca0072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.398+0000 7f8c9de00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f8ca0062720 0x7f8ca00c6f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:35780/0 (socket says 172.21.15.98:35780) 2024-05-27T00:33:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.398+0000 7f8c9f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca015e6b0 0x7f8ca006ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:60076/0 (socket says 172.21.15.98:60076) 2024-05-27T00:33:41.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.398+0000 7f8c9de00640 1 -- 172.21.15.98:0/215540199 learned_addr learned my addr 172.21.15.98:0/215540199 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.398+0000 7f8c9f200640 1 -- 172.21.15.98:0/215540199 >> v2:172.21.15.204:3300/0 conn(0x7f8ca0154a80 msgr2=0x7f8ca0072b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.398+0000 7f8c9f200640 1 --2- 172.21.15.98:0/215540199 >> v2:172.21.15.204:3300/0 conn(0x7f8ca0154a80 0x7f8ca0072b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.398+0000 7f8c9f200640 1 -- 172.21.15.98:0/215540199 >> v1:172.21.15.204:6789/0 conn(0x7f8ca0062720 legacy=0x7f8ca00c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.398+0000 7f8c9f200640 1 -- 172.21.15.98:0/215540199 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca0071d70 con 0x7f8ca015e6b0 2024-05-27T00:33:41.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.398+0000 7f8c9f200640 1 --2- 172.21.15.98:0/215540199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca015e6b0 0x7f8ca006ed10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8c980097c0 tx=0x7f8c9801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f33af310f96fc18b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.398+0000 7f8c9d400640 1 -- 172.21.15.98:0/215540199 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c980b0070 con 0x7f8ca015e6b0 2024-05-27T00:33:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9d400640 1 -- 172.21.15.98:0/215540199 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c980ab600 con 0x7f8ca015e6b0 2024-05-27T00:33:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9d400640 1 -- 172.21.15.98:0/215540199 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c980ab9d0 con 0x7f8ca015e6b0 2024-05-27T00:33:41.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 -- 172.21.15.98:0/215540199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca015e6b0 msgr2=0x7f8ca006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 --2- 172.21.15.98:0/215540199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca015e6b0 0x7f8ca006ed10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8c980097c0 tx=0x7f8c9801c410 comp rx=0 tx=0).stop 2024-05-27T00:33:41.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 -- 172.21.15.98:0/215540199 shutdown_connections 2024-05-27T00:33:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 --2- 172.21.15.98:0/215540199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca015e6b0 0x7f8ca006ed10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 --2- 172.21.15.98:0/215540199 >> v2:172.21.15.204:3300/0 conn(0x7f8ca0154a80 0x7f8ca0072b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 -- 172.21.15.98:0/215540199 >> 172.21.15.98:0/215540199 conn(0x7f8ca00cc560 msgr2=0x7f8ca00cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 -- 172.21.15.98:0/215540199 shutdown_connections 2024-05-27T00:33:41.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 -- 172.21.15.98:0/215540199 wait complete. 2024-05-27T00:33:41.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 Processor -- start 2024-05-27T00:33:41.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 -- start start 2024-05-27T00:33:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ca0062720 0x7f8ca00c5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca0154a80 0x7f8ca01f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ca015e6b0 0x7f8ca0142570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca0167210 con 0x7f8ca0062720 2024-05-27T00:33:41.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca0167090 con 0x7f8ca015e6b0 2024-05-27T00:33:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8ca5400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca0167390 con 0x7f8ca0154a80 2024-05-27T00:33:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9e800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ca0062720 0x7f8ca00c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9e800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ca0062720 0x7f8ca00c5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59782/0 (socket says 172.21.15.98:59782) 2024-05-27T00:33:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9e800640 1 -- 172.21.15.98:0/3722340725 learned_addr learned my addr 172.21.15.98:0/3722340725 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9f200640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ca015e6b0 0x7f8ca0142570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9de00640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca0154a80 0x7f8ca01f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9e800640 1 -- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca0154a80 msgr2=0x7f8ca01f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9e800640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca0154a80 0x7f8ca01f3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9e800640 1 -- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ca015e6b0 msgr2=0x7f8ca0142570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9e800640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ca015e6b0 0x7f8ca0142570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.402+0000 7f8c9e800640 1 -- 172.21.15.98:0/3722340725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca014ebe0 con 0x7f8ca0062720 2024-05-27T00:33:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8c9de00640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca0154a80 0x7f8ca01f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:33:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8c9e800640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ca0062720 0x7f8ca00c5930 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009db0 tx=0x7f8c9000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8c96a00640 1 -- 172.21.15.98:0/3722340725 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90018020 con 0x7f8ca0062720 2024-05-27T00:33:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8c96a00640 1 -- 172.21.15.98:0/3722340725 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c90014480 con 0x7f8ca0062720 2024-05-27T00:33:41.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8c96a00640 1 -- 172.21.15.98:0/3722340725 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90014850 con 0x7f8ca0062720 2024-05-27T00:33:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8ca5400640 1 -- 172.21.15.98:0/3722340725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ca014e330 con 0x7f8ca0062720 2024-05-27T00:33:41.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8ca5400640 1 -- 172.21.15.98:0/3722340725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ca00cc1b0 con 0x7f8ca0062720 2024-05-27T00:33:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8ca5400640 1 -- 172.21.15.98:0/3722340725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ca01f3400 con 0x7f8ca0062720 2024-05-27T00:33:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8c96a00640 1 -- 172.21.15.98:0/3722340725 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8c9001b020 con 0x7f8ca0062720 2024-05-27T00:33:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8c96a00640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8c800678c0 0x7f8c80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8c96a00640 1 -- 172.21.15.98:0/3722340725 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f8c90086c60 con 0x7f8ca0062720 2024-05-27T00:33:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.406+0000 7f8c9de00640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8c800678c0 0x7f8c80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.410+0000 7f8c9de00640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8c800678c0 0x7f8c80069d80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c007fd0 tx=0x7f8c8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:41.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.410+0000 7f8c96a00640 1 -- 172.21.15.98:0/3722340725 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f8c90022070 con 0x7f8ca0062720 2024-05-27T00:33:41.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.562+0000 7f8ca5400640 1 -- 172.21.15.98:0/3722340725 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8ca00ca360 con 0x7f8c800678c0 2024-05-27T00:33:41.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.582+0000 7f8c96a00640 1 -- 172.21.15.98:0/3722340725 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257325 (secure 0 0 0) 0x7f8ca00ca360 con 0x7f8c800678c0 2024-05-27T00:33:41.586 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:33:41.587 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:33:41.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 -- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8c800678c0 msgr2=0x7f8c80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8c800678c0 0x7f8c80069d80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c007fd0 tx=0x7f8c8c009040 comp rx=0 tx=0).stop 2024-05-27T00:33:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 -- 172.21.15.98:0/3722340725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ca0062720 msgr2=0x7f8ca00c5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ca0062720 0x7f8ca00c5930 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009db0 tx=0x7f8c9000c490 comp rx=0 tx=0).stop 2024-05-27T00:33:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 -- 172.21.15.98:0/3722340725 shutdown_connections 2024-05-27T00:33:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f8c800678c0 0x7f8c80069d80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ca015e6b0 0x7f8ca0142570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ca0154a80 0x7f8ca01f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 --2- 172.21.15.98:0/3722340725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ca0062720 0x7f8ca00c5930 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 -- 172.21.15.98:0/3722340725 >> 172.21.15.98:0/3722340725 conn(0x7f8ca00cc560 msgr2=0x7f8ca00cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 -- 172.21.15.98:0/3722340725 shutdown_connections 2024-05-27T00:33:41.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.586+0000 7f8ca5400640 1 -- 172.21.15.98:0/3722340725 wait complete. 2024-05-27T00:33:41.605 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2024-05-27T00:33:41.543203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1956708577,"num_objects":41115,"num_object_clones":18298,"num_object_copies":82230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41115,"num_whiteouts":12489,"num_read":13891,"num_read_kb":6221406,"num_write":126437,"num_write_kb":3703202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1834,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130934,"ondisk_log_size":130934,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5016024,"kb_used_data":3916120,"kb_used_omap":12,"kb_used_meta":1099827,"kb_avail":749958696,"statfs":{"total":773094113280,"available":767957704704,"internally_reserved":0,"allocated":4010106880,"data_stored":3917368218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1126223410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":63167789,"num_objects":639,"num_object_clones":225,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":203,"num_read":456,"num_read_kb":49335,"num_write":1016,"num_write_kb":62825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004467"},"pg_stats":[{"pgid":"3.3f","version":"48'1517","reported_seq":3393,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.925854+0000","last_change":"2024-05-27T00:17:54.336701+0000","last_active":"2024-05-27T00:33:33.925854+0000","last_peered":"2024-05-27T00:33:33.925854+0000","last_clean":"2024-05-27T00:33:33.925854+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:33:33.925854+0000","last_undegraded":"2024-05-27T00:33:33.925854+0000","last_fullsized":"2024-05-27T00:33:33.925854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:39.250119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027619028,"stat_sum":{"num_bytes":17373324,"num_objects":575,"num_object_clones":268,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":196,"num_read":49,"num_read_kb":31548,"num_write":1364,"num_write_kb":18943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"48'1504","reported_seq":3403,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.895060+0000","last_change":"2024-05-27T00:17:54.490184+0000","last_active":"2024-05-27T00:33:30.895060+0000","last_peered":"2024-05-27T00:33:30.895060+0000","last_clean":"2024-05-27T00:33:30.895060+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:33:30.895060+0000","last_undegraded":"2024-05-27T00:33:30.895060+0000","last_fullsized":"2024-05-27T00:33:30.895060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1504,"log_dups_size":0,"ondisk_log_size":1504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:05:48.629948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015469564,"stat_sum":{"num_bytes":18053131,"num_objects":546,"num_object_clones":255,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":169,"num_read":59,"num_read_kb":35851,"num_write":1378,"num_write_kb":19996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"48'1616","reported_seq":3466,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.917802+0000","last_change":"2024-05-27T00:17:54.493058+0000","last_active":"2024-05-27T00:33:33.917802+0000","last_peered":"2024-05-27T00:33:33.917802+0000","last_clean":"2024-05-27T00:33:33.917802+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:33:33.917802+0000","last_undegraded":"2024-05-27T00:33:33.917802+0000","last_fullsized":"2024-05-27T00:33:33.917802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:59:53.965584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018507847000000001,"stat_sum":{"num_bytes":15200236,"num_objects":629,"num_object_clones":298,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":212,"num_read":53,"num_read_kb":23374,"num_write":1443,"num_write_kb":25758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"48'1615","reported_seq":3479,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.883919+0000","last_change":"2024-05-27T00:17:54.510484+0000","last_active":"2024-05-27T00:33:30.883919+0000","last_peered":"2024-05-27T00:33:30.883919+0000","last_clean":"2024-05-27T00:33:30.883919+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:33:30.883919+0000","last_undegraded":"2024-05-27T00:33:30.883919+0000","last_fullsized":"2024-05-27T00:33:30.883919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:39.250119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034338709000000002,"stat_sum":{"num_bytes":10173730,"num_objects":618,"num_object_clones":288,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":202,"num_read":56,"num_read_kb":12442,"num_write":1451,"num_write_kb":16900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"48'1701","reported_seq":3542,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.395661+0000","last_change":"2024-05-27T00:17:54.331579+0000","last_active":"2024-05-27T00:33:33.395661+0000","last_peered":"2024-05-27T00:33:33.395661+0000","last_clean":"2024-05-27T00:33:33.395661+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:33:33.395661+0000","last_undegraded":"2024-05-27T00:33:33.395661+0000","last_fullsized":"2024-05-27T00:33:33.395661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1701,"log_dups_size":0,"ondisk_log_size":1701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:35:37.983576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026116615999999999,"stat_sum":{"num_bytes":20047775,"num_objects":666,"num_object_clones":311,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":219,"num_read":58,"num_read_kb":29919,"num_write":1529,"num_write_kb":21884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"48'1457","reported_seq":3333,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.926396+0000","last_change":"2024-05-27T00:17:54.507830+0000","last_active":"2024-05-27T00:33:33.926396+0000","last_peered":"2024-05-27T00:33:33.926396+0000","last_clean":"2024-05-27T00:33:33.926396+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:33:33.926396+0000","last_undegraded":"2024-05-27T00:33:33.926396+0000","last_fullsized":"2024-05-27T00:33:33.926396+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1457,"log_dups_size":0,"ondisk_log_size":1457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:48.582600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032595461999999999,"stat_sum":{"num_bytes":14545166,"num_objects":545,"num_object_clones":256,"num_object_copies":1090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":181,"num_read":38,"num_read_kb":8264,"num_write":1295,"num_write_kb":15901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"48'1575","reported_seq":3443,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.871914+0000","last_change":"2024-05-27T00:17:54.335843+0000","last_active":"2024-05-27T00:33:29.871914+0000","last_peered":"2024-05-27T00:33:29.871914+0000","last_clean":"2024-05-27T00:33:29.871914+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:33:29.871914+0000","last_undegraded":"2024-05-27T00:33:29.871914+0000","last_fullsized":"2024-05-27T00:33:29.871914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1575,"log_dups_size":0,"ondisk_log_size":1575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:27:21.913063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029344160000000001,"stat_sum":{"num_bytes":19339722,"num_objects":602,"num_object_clones":277,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":190,"num_read":51,"num_read_kb":28747,"num_write":1431,"num_write_kb":22126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"48'1515","reported_seq":3386,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.468342+0000","last_change":"2024-05-27T00:17:54.485261+0000","last_active":"2024-05-27T00:33:30.468342+0000","last_peered":"2024-05-27T00:33:30.468342+0000","last_clean":"2024-05-27T00:33:30.468342+0000","last_became_active":"2024-05-27T00:16:06.280009+0000","last_became_peered":"2024-05-27T00:16:06.280009+0000","last_unstale":"2024-05-27T00:33:30.468342+0000","last_undegraded":"2024-05-27T00:33:30.468342+0000","last_fullsized":"2024-05-27T00:33:30.468342+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:14:46.376718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012011287000000001,"stat_sum":{"num_bytes":35835659,"num_objects":577,"num_object_clones":269,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":179,"num_read":52,"num_read_kb":60147,"num_write":1378,"num_write_kb":38796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.37","version":"48'1620","reported_seq":3476,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.395963+0000","last_change":"2024-05-27T00:17:54.485867+0000","last_active":"2024-05-27T00:33:33.395963+0000","last_peered":"2024-05-27T00:33:33.395963+0000","last_clean":"2024-05-27T00:33:33.395963+0000","last_became_active":"2024-05-27T00:16:06.280544+0000","last_became_peered":"2024-05-27T00:16:06.280544+0000","last_unstale":"2024-05-27T00:33:33.395963+0000","last_undegraded":"2024-05-27T00:33:33.395963+0000","last_fullsized":"2024-05-27T00:33:33.395963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:20:13.063312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096681779999999995,"stat_sum":{"num_bytes":20135711,"num_objects":629,"num_object_clones":292,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":198,"num_read":60,"num_read_kb":29133,"num_write":1469,"num_write_kb":25955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"48'1514","reported_seq":3391,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.450019+0000","last_change":"2024-05-27T00:17:54.337723+0000","last_active":"2024-05-27T00:33:30.450019+0000","last_peered":"2024-05-27T00:33:30.450019+0000","last_clean":"2024-05-27T00:33:30.450019+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:33:30.450019+0000","last_undegraded":"2024-05-27T00:33:30.450019+0000","last_fullsized":"2024-05-27T00:33:30.450019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1514,"log_dups_size":0,"ondisk_log_size":1514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:55:01.773614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027993297,"stat_sum":{"num_bytes":14778263,"num_objects":589,"num_object_clones":272,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":186,"num_read":55,"num_read_kb":22038,"num_write":1380,"num_write_kb":16492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"48'1627","reported_seq":3458,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.895588+0000","last_change":"2024-05-27T00:17:54.335171+0000","last_active":"2024-05-27T00:33:33.895588+0000","last_peered":"2024-05-27T00:33:33.895588+0000","last_clean":"2024-05-27T00:33:33.895588+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:33:33.895588+0000","last_undegraded":"2024-05-27T00:33:33.895588+0000","last_fullsized":"2024-05-27T00:33:33.895588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1627,"log_dups_size":0,"ondisk_log_size":1627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:11:26.796581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027428447000000002,"stat_sum":{"num_bytes":10658234,"num_objects":659,"num_object_clones":308,"num_object_copies":1318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":204,"num_read":45,"num_read_kb":12410,"num_write":1475,"num_write_kb":11557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"48'1609","reported_seq":3434,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.363658+0000","last_change":"2024-05-27T00:17:54.504751+0000","last_active":"2024-05-27T00:33:33.363658+0000","last_peered":"2024-05-27T00:33:33.363658+0000","last_clean":"2024-05-27T00:33:33.363658+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:33:33.363658+0000","last_undegraded":"2024-05-27T00:33:33.363658+0000","last_fullsized":"2024-05-27T00:33:33.363658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:51:55.125064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021724307000000002,"stat_sum":{"num_bytes":19374715,"num_objects":636,"num_object_clones":300,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":200,"num_read":31,"num_read_kb":14211,"num_write":1467,"num_write_kb":31877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"48'1547","reported_seq":3401,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.417979+0000","last_change":"2024-05-27T00:17:54.309597+0000","last_active":"2024-05-27T00:33:30.417979+0000","last_peered":"2024-05-27T00:33:30.417979+0000","last_clean":"2024-05-27T00:33:30.417979+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:33:30.417979+0000","last_undegraded":"2024-05-27T00:33:30.417979+0000","last_fullsized":"2024-05-27T00:33:30.417979+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1547,"log_dups_size":0,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:01:46.216541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089222539999999993,"stat_sum":{"num_bytes":26519812,"num_objects":619,"num_object_clones":291,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":212,"num_read":51,"num_read_kb":32873,"num_write":1364,"num_write_kb":36569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"48'1587","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.412049+0000","last_change":"2024-05-27T00:17:54.491142+0000","last_active":"2024-05-27T00:33:30.412049+0000","last_peered":"2024-05-27T00:33:30.412049+0000","last_clean":"2024-05-27T00:33:30.412049+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:33:30.412049+0000","last_undegraded":"2024-05-27T00:33:30.412049+0000","last_fullsized":"2024-05-27T00:33:30.412049+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1587,"log_dups_size":0,"ondisk_log_size":1587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:43:35.726386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016842227000000001,"stat_sum":{"num_bytes":10308290,"num_objects":608,"num_object_clones":288,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":204,"num_read":44,"num_read_kb":16487,"num_write":1426,"num_write_kb":16803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"48'1620","reported_seq":3487,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.955725+0000","last_change":"2024-05-27T00:17:54.491503+0000","last_active":"2024-05-27T00:33:30.955725+0000","last_peered":"2024-05-27T00:33:30.955725+0000","last_clean":"2024-05-27T00:33:30.955725+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:33:30.955725+0000","last_undegraded":"2024-05-27T00:33:30.955725+0000","last_fullsized":"2024-05-27T00:33:30.955725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:33:06.746690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016821815,"stat_sum":{"num_bytes":22996149,"num_objects":605,"num_object_clones":283,"num_object_copies":1210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":605,"num_whiteouts":212,"num_read":56,"num_read_kb":39458,"num_write":1456,"num_write_kb":27752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"48'1589","reported_seq":3458,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.899422+0000","last_change":"2024-05-27T00:17:54.506224+0000","last_active":"2024-05-27T00:33:30.899422+0000","last_peered":"2024-05-27T00:33:30.899422+0000","last_clean":"2024-05-27T00:33:30.899422+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:33:30.899422+0000","last_undegraded":"2024-05-27T00:33:30.899422+0000","last_fullsized":"2024-05-27T00:33:30.899422+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1589,"log_dups_size":0,"ondisk_log_size":1589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:44.778493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.031092022,"stat_sum":{"num_bytes":14532615,"num_objects":603,"num_object_clones":288,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":201,"num_read":63,"num_read_kb":16619,"num_write":1426,"num_write_kb":20413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"48'1709","reported_seq":3577,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.368113+0000","last_change":"2024-05-27T00:17:54.511207+0000","last_active":"2024-05-27T00:33:33.368113+0000","last_peered":"2024-05-27T00:33:33.368113+0000","last_clean":"2024-05-27T00:33:33.368113+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:33:33.368113+0000","last_undegraded":"2024-05-27T00:33:33.368113+0000","last_fullsized":"2024-05-27T00:33:33.368113+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1709,"log_dups_size":0,"ondisk_log_size":1709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:03:55.531180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034311531999999999,"stat_sum":{"num_bytes":28217639,"num_objects":653,"num_object_clones":304,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":203,"num_read":78,"num_read_kb":63526,"num_write":1561,"num_write_kb":34080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"48'1512","reported_seq":3382,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.394396+0000","last_change":"2024-05-27T00:17:54.487572+0000","last_active":"2024-05-27T00:33:33.394396+0000","last_peered":"2024-05-27T00:33:33.394396+0000","last_clean":"2024-05-27T00:33:33.394396+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:33:33.394396+0000","last_undegraded":"2024-05-27T00:33:33.394396+0000","last_fullsized":"2024-05-27T00:33:33.394396+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:07:28.224179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010480461999999999,"stat_sum":{"num_bytes":6654877,"num_objects":588,"num_object_clones":271,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":191,"num_read":47,"num_read_kb":139,"num_write":1368,"num_write_kb":8104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"48'1671","reported_seq":3510,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.397671+0000","last_change":"2024-05-27T00:17:54.337799+0000","last_active":"2024-05-27T00:33:33.397671+0000","last_peered":"2024-05-27T00:33:33.397671+0000","last_clean":"2024-05-27T00:33:33.397671+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:33:33.397671+0000","last_undegraded":"2024-05-27T00:33:33.397671+0000","last_fullsized":"2024-05-27T00:33:33.397671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:13.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029916169999999999,"stat_sum":{"num_bytes":16039418,"num_objects":654,"num_object_clones":310,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":208,"num_read":54,"num_read_kb":28815,"num_write":1504,"num_write_kb":20996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"48'1603","reported_seq":3487,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.440010+0000","last_change":"2024-05-27T00:17:54.335854+0000","last_active":"2024-05-27T00:33:30.440010+0000","last_peered":"2024-05-27T00:33:30.440010+0000","last_clean":"2024-05-27T00:33:30.440010+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:33:30.440010+0000","last_undegraded":"2024-05-27T00:33:30.440010+0000","last_fullsized":"2024-05-27T00:33:30.440010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:13:02.845565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027334256000000001,"stat_sum":{"num_bytes":31314512,"num_objects":608,"num_object_clones":281,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":187,"num_read":70,"num_read_kb":56922,"num_write":1458,"num_write_kb":40262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"48'1614","reported_seq":3479,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.930688+0000","last_change":"2024-05-27T00:17:54.493382+0000","last_active":"2024-05-27T00:33:33.930688+0000","last_peered":"2024-05-27T00:33:33.930688+0000","last_clean":"2024-05-27T00:33:33.930688+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:33:33.930688+0000","last_undegraded":"2024-05-27T00:33:33.930688+0000","last_fullsized":"2024-05-27T00:33:33.930688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1614,"log_dups_size":0,"ondisk_log_size":1614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:11:52.944101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017108005999999999,"stat_sum":{"num_bytes":14791903,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":193,"num_read":59,"num_read_kb":12448,"num_write":1482,"num_write_kb":20672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"48'1678","reported_seq":3512,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.367645+0000","last_change":"2024-05-27T00:17:54.509854+0000","last_active":"2024-05-27T00:33:33.367645+0000","last_peered":"2024-05-27T00:33:33.367645+0000","last_clean":"2024-05-27T00:33:33.367645+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:33:33.367645+0000","last_undegraded":"2024-05-27T00:33:33.367645+0000","last_fullsized":"2024-05-27T00:33:33.367645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1678,"log_dups_size":0,"ondisk_log_size":1678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:13.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033862588999999998,"stat_sum":{"num_bytes":18976923,"num_objects":661,"num_object_clones":315,"num_object_copies":1322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":222,"num_read":54,"num_read_kb":29811,"num_write":1493,"num_write_kb":24680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"48'1529","reported_seq":3405,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.389080+0000","last_change":"2024-05-27T00:17:54.309499+0000","last_active":"2024-05-27T00:33:33.389080+0000","last_peered":"2024-05-27T00:33:33.389080+0000","last_clean":"2024-05-27T00:33:33.389080+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:33:33.389080+0000","last_undegraded":"2024-05-27T00:33:33.389080+0000","last_fullsized":"2024-05-27T00:33:33.389080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1529,"log_dups_size":0,"ondisk_log_size":1529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:32:55.895752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088470660000000007,"stat_sum":{"num_bytes":9565297,"num_objects":574,"num_object_clones":267,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":178,"num_read":48,"num_read_kb":13982,"num_write":1418,"num_write_kb":15615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"48'1569","reported_seq":3463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.893810+0000","last_change":"2024-05-27T00:17:54.509276+0000","last_active":"2024-05-27T00:33:29.893810+0000","last_peered":"2024-05-27T00:33:29.893810+0000","last_clean":"2024-05-27T00:33:29.893810+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:33:29.893810+0000","last_undegraded":"2024-05-27T00:33:29.893810+0000","last_fullsized":"2024-05-27T00:33:29.893810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:55:24.602133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033457270999999997,"stat_sum":{"num_bytes":30464560,"num_objects":585,"num_object_clones":269,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":175,"num_read":68,"num_read_kb":58155,"num_write":1439,"num_write_kb":31950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"48'1566","reported_seq":3425,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.854148+0000","last_change":"2024-05-27T00:17:54.310134+0000","last_active":"2024-05-27T00:33:29.854148+0000","last_peered":"2024-05-27T00:33:29.854148+0000","last_clean":"2024-05-27T00:33:29.854148+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:33:29.854148+0000","last_undegraded":"2024-05-27T00:33:29.854148+0000","last_fullsized":"2024-05-27T00:33:29.854148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1566,"log_dups_size":0,"ondisk_log_size":1566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:37:27.481263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089651499999999999,"stat_sum":{"num_bytes":14362147,"num_objects":603,"num_object_clones":285,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":194,"num_read":49,"num_read_kb":20561,"num_write":1422,"num_write_kb":17027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"48'1667","reported_seq":3529,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.893411+0000","last_change":"2024-05-27T00:17:54.337554+0000","last_active":"2024-05-27T00:33:33.893411+0000","last_peered":"2024-05-27T00:33:33.893411+0000","last_clean":"2024-05-27T00:33:33.893411+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:33:33.893411+0000","last_undegraded":"2024-05-27T00:33:33.893411+0000","last_fullsized":"2024-05-27T00:33:33.893411+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1667,"log_dups_size":0,"ondisk_log_size":1667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:02:46.037448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024583906999999999,"stat_sum":{"num_bytes":23325444,"num_objects":624,"num_object_clones":288,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":196,"num_read":55,"num_read_kb":32562,"num_write":1514,"num_write_kb":26070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"48'1526","reported_seq":3399,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.912844+0000","last_change":"2024-05-27T00:17:54.491723+0000","last_active":"2024-05-27T00:33:33.912844+0000","last_peered":"2024-05-27T00:33:33.912844+0000","last_clean":"2024-05-27T00:33:33.912844+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:33:33.912844+0000","last_undegraded":"2024-05-27T00:33:33.912844+0000","last_fullsized":"2024-05-27T00:33:33.912844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:41:31.328434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016722006000000001,"stat_sum":{"num_bytes":10472871,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":180,"num_read":47,"num_read_kb":12352,"num_write":1385,"num_write_kb":12462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"48'1571","reported_seq":3430,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.392636+0000","last_change":"2024-05-27T00:17:54.309792+0000","last_active":"2024-05-27T00:33:33.392636+0000","last_peered":"2024-05-27T00:33:33.392636+0000","last_clean":"2024-05-27T00:33:33.392636+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:33:33.392636+0000","last_undegraded":"2024-05-27T00:33:33.392636+0000","last_fullsized":"2024-05-27T00:33:33.392636+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:03:52.656303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089478649999999993,"stat_sum":{"num_bytes":10328890,"num_objects":604,"num_object_clones":285,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":194,"num_read":49,"num_read_kb":15706,"num_write":1422,"num_write_kb":15432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"48'1703","reported_seq":3521,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.956715+0000","last_change":"2024-05-27T00:17:54.494236+0000","last_active":"2024-05-27T00:33:30.956715+0000","last_peered":"2024-05-27T00:33:30.956715+0000","last_clean":"2024-05-27T00:33:30.956715+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:33:30.956715+0000","last_undegraded":"2024-05-27T00:33:30.956715+0000","last_fullsized":"2024-05-27T00:33:30.956715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1703,"log_dups_size":0,"ondisk_log_size":1703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:17:24.056537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017911486000000001,"stat_sum":{"num_bytes":9846144,"num_objects":668,"num_object_clones":317,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":212,"num_read":40,"num_read_kb":1060,"num_write":1530,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"48'1604","reported_seq":3472,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.396278+0000","last_change":"2024-05-27T00:17:54.510031+0000","last_active":"2024-05-27T00:33:33.396278+0000","last_peered":"2024-05-27T00:33:33.396278+0000","last_clean":"2024-05-27T00:33:33.396278+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:33:33.396278+0000","last_undegraded":"2024-05-27T00:33:33.396278+0000","last_fullsized":"2024-05-27T00:33:33.396278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1604,"log_dups_size":0,"ondisk_log_size":1604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:09.263437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033387499000000001,"stat_sum":{"num_bytes":18975931,"num_objects":601,"num_object_clones":281,"num_object_copies":1202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":187,"num_read":54,"num_read_kb":24794,"num_write":1457,"num_write_kb":20478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"48'1651","reported_seq":3529,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.347644+0000","last_change":"2024-05-27T00:17:54.484908+0000","last_active":"2024-05-27T00:33:30.347644+0000","last_peered":"2024-05-27T00:33:30.347644+0000","last_clean":"2024-05-27T00:33:30.347644+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:33:30.347644+0000","last_undegraded":"2024-05-27T00:33:30.347644+0000","last_fullsized":"2024-05-27T00:33:30.347644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1651,"log_dups_size":0,"ondisk_log_size":1651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:10:39.525108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009527878,"stat_sum":{"num_bytes":27932857,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":194,"num_read":62,"num_read_kb":41489,"num_write":1506,"num_write_kb":37195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"48'1571","reported_seq":3467,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.389518+0000","last_change":"2024-05-27T00:17:54.331060+0000","last_active":"2024-05-27T00:33:33.389518+0000","last_peered":"2024-05-27T00:33:33.389518+0000","last_clean":"2024-05-27T00:33:33.389518+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:33:33.389518+0000","last_undegraded":"2024-05-27T00:33:33.389518+0000","last_fullsized":"2024-05-27T00:33:33.389518+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:05:48.629948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025435675000000001,"stat_sum":{"num_bytes":17932144,"num_objects":570,"num_object_clones":264,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":185,"num_read":65,"num_read_kb":34529,"num_write":1438,"num_write_kb":20974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"48'1632","reported_seq":3479,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.383849+0000","last_change":"2024-05-27T00:17:54.501574+0000","last_active":"2024-05-27T00:33:33.383849+0000","last_peered":"2024-05-27T00:33:33.383849+0000","last_clean":"2024-05-27T00:33:33.383849+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:33:33.383849+0000","last_undegraded":"2024-05-27T00:33:33.383849+0000","last_fullsized":"2024-05-27T00:33:33.383849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1632,"log_dups_size":0,"ondisk_log_size":1632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:16:03.345753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019320113,"stat_sum":{"num_bytes":26864770,"num_objects":637,"num_object_clones":297,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":204,"num_read":50,"num_read_kb":43416,"num_write":1495,"num_write_kb":28253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"48'1097","reported_seq":1312,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.639463+0000","last_change":"2024-05-27T00:15:24.611108+0000","last_active":"2024-05-27T00:33:34.639463+0000","last_peered":"2024-05-27T00:33:34.639463+0000","last_clean":"2024-05-27T00:33:34.639463+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:33:34.639463+0000","last_undegraded":"2024-05-27T00:33:34.639463+0000","last_fullsized":"2024-05-27T00:33:34.639463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:11.913978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":240,"num_read_kb":174989,"num_write":1187,"num_write_kb":92068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"48'1572","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.366164+0000","last_change":"2024-05-27T00:17:54.493715+0000","last_active":"2024-05-27T00:33:33.366164+0000","last_peered":"2024-05-27T00:33:33.366164+0000","last_clean":"2024-05-27T00:33:33.366164+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:33:33.366164+0000","last_undegraded":"2024-05-27T00:33:33.366164+0000","last_fullsized":"2024-05-27T00:33:33.366164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1572,"log_dups_size":0,"ondisk_log_size":1572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:50:38.769021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017111721,"stat_sum":{"num_bytes":21962235,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":198,"num_read":57,"num_read_kb":45093,"num_write":1410,"num_write_kb":24294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"48'843","reported_seq":1087,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.869212+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:33:33.869212+0000","last_peered":"2024-05-27T00:33:33.869212+0000","last_clean":"2024-05-27T00:33:33.869212+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:33:33.869212+0000","last_undegraded":"2024-05-27T00:33:33.869212+0000","last_fullsized":"2024-05-27T00:33:33.869212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":305,"num_read_kb":205128,"num_write":983,"num_write_kb":76341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":67,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"48'594","reported_seq":837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.992961+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:33:29.992961+0000","last_peered":"2024-05-27T00:33:29.992961+0000","last_clean":"2024-05-27T00:33:29.992961+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:33:29.992961+0000","last_undegraded":"2024-05-27T00:33:29.992961+0000","last_fullsized":"2024-05-27T00:33:29.992961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:46:08.645927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":317,"num_read_kb":129314,"num_write":740,"num_write_kb":59170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"48'1791","reported_seq":3623,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.885090+0000","last_change":"2024-05-27T00:17:54.309270+0000","last_active":"2024-05-27T00:33:33.885090+0000","last_peered":"2024-05-27T00:33:33.885090+0000","last_clean":"2024-05-27T00:33:33.885090+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:33:33.885090+0000","last_undegraded":"2024-05-27T00:33:33.885090+0000","last_fullsized":"2024-05-27T00:33:33.885090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1791,"log_dups_size":0,"ondisk_log_size":1791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:49:00.352259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087892630000000003,"stat_sum":{"num_bytes":15177983,"num_objects":665,"num_object_clones":314,"num_object_copies":1330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":214,"num_read":52,"num_read_kb":17293,"num_write":1633,"num_write_kb":25745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"48'1911","reported_seq":2158,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.499719+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:33:33.499719+0000","last_peered":"2024-05-27T00:33:33.499719+0000","last_clean":"2024-05-27T00:33:33.499719+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:33:33.499719+0000","last_undegraded":"2024-05-27T00:33:33.499719+0000","last_fullsized":"2024-05-27T00:33:33.499719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1911,"log_dups_size":0,"ondisk_log_size":1911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":310,"num_read_kb":210919,"num_write":2080,"num_write_kb":80661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"48'1555","reported_seq":3431,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.839742+0000","last_change":"2024-05-27T00:17:54.489271+0000","last_active":"2024-05-27T00:33:29.839742+0000","last_peered":"2024-05-27T00:33:29.839742+0000","last_clean":"2024-05-27T00:33:29.839742+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:33:29.839742+0000","last_undegraded":"2024-05-27T00:33:29.839742+0000","last_fullsized":"2024-05-27T00:33:29.839742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:49:06.355571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014633893,"stat_sum":{"num_bytes":23977264,"num_objects":602,"num_object_clones":282,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":198,"num_read":64,"num_read_kb":52608,"num_write":1398,"num_write_kb":30195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"48'592","reported_seq":854,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.900929+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:33:33.900929+0000","last_peered":"2024-05-27T00:33:33.900929+0000","last_clean":"2024-05-27T00:33:33.900929+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:33:33.900929+0000","last_undegraded":"2024-05-27T00:33:33.900929+0000","last_fullsized":"2024-05-27T00:33:33.900929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21338252,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":289,"num_read_kb":124309,"num_write":674,"num_write_kb":63278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"48'1500","reported_seq":3389,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.928969+0000","last_change":"2024-05-27T00:17:54.512612+0000","last_active":"2024-05-27T00:33:30.928969+0000","last_peered":"2024-05-27T00:33:30.928969+0000","last_clean":"2024-05-27T00:33:30.928969+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:33:30.928969+0000","last_undegraded":"2024-05-27T00:33:30.928969+0000","last_fullsized":"2024-05-27T00:33:30.928969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:07:00.308291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034220156000000002,"stat_sum":{"num_bytes":22845870,"num_objects":565,"num_object_clones":264,"num_object_copies":1130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":188,"num_read":59,"num_read_kb":32908,"num_write":1344,"num_write_kb":23966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"48'1486","reported_seq":1692,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.894653+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:33:33.894653+0000","last_peered":"2024-05-27T00:33:33.894653+0000","last_clean":"2024-05-27T00:33:33.894653+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:33:33.894653+0000","last_undegraded":"2024-05-27T00:33:33.894653+0000","last_fullsized":"2024-05-27T00:33:33.894653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":244,"num_read_kb":161522,"num_write":1589,"num_write_kb":88109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"48'1752","reported_seq":3524,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.367187+0000","last_change":"2024-05-27T00:17:54.493629+0000","last_active":"2024-05-27T00:33:33.367187+0000","last_peered":"2024-05-27T00:33:33.367187+0000","last_clean":"2024-05-27T00:33:33.367187+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:33:33.367187+0000","last_undegraded":"2024-05-27T00:33:33.367187+0000","last_fullsized":"2024-05-27T00:33:33.367187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1752,"log_dups_size":0,"ondisk_log_size":1752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:26:59.993852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017358472,"stat_sum":{"num_bytes":15481537,"num_objects":707,"num_object_clones":381,"num_object_copies":1414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":707,"num_whiteouts":205,"num_read":59,"num_read_kb":17601,"num_write":1504,"num_write_kb":17792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"48'1524","reported_seq":1820,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.936597+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:33:33.936597+0000","last_peered":"2024-05-27T00:33:33.936597+0000","last_clean":"2024-05-27T00:33:33.936597+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:33:33.936597+0000","last_undegraded":"2024-05-27T00:33:33.936597+0000","last_fullsized":"2024-05-27T00:33:33.936597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":428,"num_read_kb":130260,"num_write":1739,"num_write_kb":60710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"48'1583","reported_seq":3446,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.901089+0000","last_change":"2024-05-27T00:17:54.506220+0000","last_active":"2024-05-27T00:33:30.901089+0000","last_peered":"2024-05-27T00:33:30.901089+0000","last_clean":"2024-05-27T00:33:30.901089+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:33:30.901089+0000","last_undegraded":"2024-05-27T00:33:30.901089+0000","last_fullsized":"2024-05-27T00:33:30.901089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1583,"log_dups_size":0,"ondisk_log_size":1583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:32:01.052698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022583427,"stat_sum":{"num_bytes":13475539,"num_objects":608,"num_object_clones":285,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":200,"num_read":54,"num_read_kb":12130,"num_write":1429,"num_write_kb":19813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"48'1064","reported_seq":1346,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.872488+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:33:33.872488+0000","last_peered":"2024-05-27T00:33:33.872488+0000","last_clean":"2024-05-27T00:33:33.872488+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:33:33.872488+0000","last_undegraded":"2024-05-27T00:33:33.872488+0000","last_fullsized":"2024-05-27T00:33:33.872488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555104,"num_objects":77,"num_object_clones":0,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":365,"num_read_kb":194738,"num_write":1216,"num_write_kb":100979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"48'1567","reported_seq":3423,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.931447+0000","last_change":"2024-05-27T00:17:54.336019+0000","last_active":"2024-05-27T00:33:33.931447+0000","last_peered":"2024-05-27T00:33:33.931447+0000","last_clean":"2024-05-27T00:33:33.931447+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:33:33.931447+0000","last_undegraded":"2024-05-27T00:33:33.931447+0000","last_fullsized":"2024-05-27T00:33:33.931447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:48.582600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.030558409000000002,"stat_sum":{"num_bytes":10200444,"num_objects":601,"num_object_clones":280,"num_object_copies":1202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":188,"num_read":42,"num_read_kb":12368,"num_write":1435,"num_write_kb":13032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"48'830","reported_seq":1084,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.892943+0000","last_change":"2024-05-27T00:15:24.605986+0000","last_active":"2024-05-27T00:33:33.892943+0000","last_peered":"2024-05-27T00:33:33.892943+0000","last_clean":"2024-05-27T00:33:33.892943+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:33:33.892943+0000","last_undegraded":"2024-05-27T00:33:33.892943+0000","last_fullsized":"2024-05-27T00:33:33.892943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971554,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":336,"num_read_kb":131285,"num_write":974,"num_write_kb":63316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":56,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"48'1706","reported_seq":3516,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.390563+0000","last_change":"2024-05-27T00:17:54.493027+0000","last_active":"2024-05-27T00:33:33.390563+0000","last_peered":"2024-05-27T00:33:33.390563+0000","last_clean":"2024-05-27T00:33:33.390563+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:33:33.390563+0000","last_undegraded":"2024-05-27T00:33:33.390563+0000","last_fullsized":"2024-05-27T00:33:33.390563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1706,"log_dups_size":0,"ondisk_log_size":1706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:38:11.155792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018271039999999999,"stat_sum":{"num_bytes":20173932,"num_objects":652,"num_object_clones":356,"num_object_copies":1304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":187,"num_read":72,"num_read_kb":36313,"num_write":1471,"num_write_kb":31527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"48'502","reported_seq":699,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.529271+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:33:34.529271+0000","last_peered":"2024-05-27T00:33:34.529271+0000","last_clean":"2024-05-27T00:33:34.529271+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:33:34.529271+0000","last_undegraded":"2024-05-27T00:33:34.529271+0000","last_fullsized":"2024-05-27T00:33:34.529271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":232,"num_read_kb":71610,"num_write":625,"num_write_kb":47247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"48'1677","reported_seq":3546,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.955606+0000","last_change":"2024-05-27T00:17:54.334890+0000","last_active":"2024-05-27T00:33:30.955606+0000","last_peered":"2024-05-27T00:33:30.955606+0000","last_clean":"2024-05-27T00:33:30.955606+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:33:30.955606+0000","last_undegraded":"2024-05-27T00:33:30.955606+0000","last_fullsized":"2024-05-27T00:33:30.955606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1677,"log_dups_size":0,"ondisk_log_size":1677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:55.040370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028694234999999998,"stat_sum":{"num_bytes":27706000,"num_objects":637,"num_object_clones":296,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":195,"num_read":71,"num_read_kb":46547,"num_write":1532,"num_write_kb":29714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"48'364","reported_seq":562,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:29.779855+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:33:29.779855+0000","last_peered":"2024-05-27T00:33:29.779855+0000","last_clean":"2024-05-27T00:33:29.779855+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:33:29.779855+0000","last_undegraded":"2024-05-27T00:33:29.779855+0000","last_fullsized":"2024-05-27T00:33:29.779855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":244,"num_read_kb":99922,"num_write":491,"num_write_kb":51118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"48'1573","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.391959+0000","last_change":"2024-05-27T00:17:54.332802+0000","last_active":"2024-05-27T00:33:33.391959+0000","last_peered":"2024-05-27T00:33:33.391959+0000","last_clean":"2024-05-27T00:33:33.391959+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:33:33.391959+0000","last_undegraded":"2024-05-27T00:33:33.391959+0000","last_fullsized":"2024-05-27T00:33:33.391959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:41:31.328434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025579069999999999,"stat_sum":{"num_bytes":19680366,"num_objects":592,"num_object_clones":272,"num_object_copies":1184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":592,"num_whiteouts":185,"num_read":43,"num_read_kb":14554,"num_write":1424,"num_write_kb":24215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"48'749","reported_seq":1035,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.864326+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:33:33.864326+0000","last_peered":"2024-05-27T00:33:33.864326+0000","last_clean":"2024-05-27T00:33:33.864326+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:33:33.864326+0000","last_undegraded":"2024-05-27T00:33:33.864326+0000","last_fullsized":"2024-05-27T00:33:33.864326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":749,"log_dups_size":0,"ondisk_log_size":749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":479,"num_read_kb":120710,"num_write":984,"num_write_kb":72758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":71,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"48'1617","reported_seq":3492,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.972082+0000","last_change":"2024-05-27T00:17:54.334829+0000","last_active":"2024-05-27T00:33:30.972082+0000","last_peered":"2024-05-27T00:33:30.972082+0000","last_clean":"2024-05-27T00:33:30.972082+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:33:30.972082+0000","last_undegraded":"2024-05-27T00:33:30.972082+0000","last_fullsized":"2024-05-27T00:33:30.972082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1617,"log_dups_size":0,"ondisk_log_size":1617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:00:47.756042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028937418999999999,"stat_sum":{"num_bytes":19454539,"num_objects":599,"num_object_clones":279,"num_object_copies":1198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":599,"num_whiteouts":201,"num_read":60,"num_read_kb":39320,"num_write":1444,"num_write_kb":29954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"48'926","reported_seq":1176,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.639917+0000","last_change":"2024-05-27T00:15:24.601924+0000","last_active":"2024-05-27T00:33:34.639917+0000","last_peered":"2024-05-27T00:33:34.639917+0000","last_clean":"2024-05-27T00:33:34.639917+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:33:34.639917+0000","last_undegraded":"2024-05-27T00:33:34.639917+0000","last_fullsized":"2024-05-27T00:33:34.639917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":926,"log_dups_size":0,"ondisk_log_size":926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:26:32.185839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971542,"num_objects":69,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":329,"num_read_kb":104768,"num_write":1076,"num_write_kb":59606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"48'1603","reported_seq":3463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.427204+0000","last_change":"2024-05-27T00:17:54.335438+0000","last_active":"2024-05-27T00:33:30.427204+0000","last_peered":"2024-05-27T00:33:30.427204+0000","last_clean":"2024-05-27T00:33:30.427204+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:33:30.427204+0000","last_undegraded":"2024-05-27T00:33:30.427204+0000","last_fullsized":"2024-05-27T00:33:30.427204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:44.778493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029056843999999998,"stat_sum":{"num_bytes":27639049,"num_objects":630,"num_object_clones":290,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":189,"num_read":55,"num_read_kb":39571,"num_write":1460,"num_write_kb":33520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"48'850","reported_seq":1063,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:28.880655+0000","last_change":"2024-05-27T00:15:24.600878+0000","last_active":"2024-05-27T00:33:28.880655+0000","last_peered":"2024-05-27T00:33:28.880655+0000","last_clean":"2024-05-27T00:33:28.880655+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:33:28.880655+0000","last_undegraded":"2024-05-27T00:33:28.880655+0000","last_fullsized":"2024-05-27T00:33:28.880655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":241,"num_read_kb":194399,"num_write":970,"num_write_kb":84408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"48'1539","reported_seq":3419,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.936151+0000","last_change":"2024-05-27T00:17:54.337688+0000","last_active":"2024-05-27T00:33:30.936151+0000","last_peered":"2024-05-27T00:33:30.936151+0000","last_clean":"2024-05-27T00:33:30.936151+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:33:30.936151+0000","last_undegraded":"2024-05-27T00:33:30.936151+0000","last_fullsized":"2024-05-27T00:33:30.936151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1539,"log_dups_size":0,"ondisk_log_size":1539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:09.263437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028414170999999998,"stat_sum":{"num_bytes":22590275,"num_objects":587,"num_object_clones":272,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":186,"num_read":57,"num_read_kb":45093,"num_write":1409,"num_write_kb":28119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"48'1073","reported_seq":1310,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.705095+0000","last_change":"2024-05-27T00:15:24.603452+0000","last_active":"2024-05-27T00:33:34.705095+0000","last_peered":"2024-05-27T00:33:34.705095+0000","last_clean":"2024-05-27T00:33:34.705095+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:33:34.705095+0000","last_undegraded":"2024-05-27T00:33:34.705095+0000","last_fullsized":"2024-05-27T00:33:34.705095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1073,"log_dups_size":0,"ondisk_log_size":1073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34922452,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":293,"num_read_kb":166112,"num_write":1227,"num_write_kb":85323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"48'1558","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.931551+0000","last_change":"2024-05-27T00:17:54.512510+0000","last_active":"2024-05-27T00:33:30.931551+0000","last_peered":"2024-05-27T00:33:30.931551+0000","last_clean":"2024-05-27T00:33:30.931551+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:33:30.931551+0000","last_undegraded":"2024-05-27T00:33:30.931551+0000","last_fullsized":"2024-05-27T00:33:30.931551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1558,"log_dups_size":0,"ondisk_log_size":1558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:13:45.104337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034804740000000001,"stat_sum":{"num_bytes":14041732,"num_objects":561,"num_object_clones":257,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":186,"num_read":41,"num_read_kb":16491,"num_write":1423,"num_write_kb":17133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"48'755","reported_seq":983,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.895276+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:33:33.895276+0000","last_peered":"2024-05-27T00:33:33.895276+0000","last_clean":"2024-05-27T00:33:33.895276+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:33:33.895276+0000","last_undegraded":"2024-05-27T00:33:33.895276+0000","last_fullsized":"2024-05-27T00:33:33.895276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":292,"num_read_kb":141638,"num_write":926,"num_write_kb":64367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"48'1568","reported_seq":3422,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.895827+0000","last_change":"2024-05-27T00:17:54.333811+0000","last_active":"2024-05-27T00:33:33.895827+0000","last_peered":"2024-05-27T00:33:33.895827+0000","last_clean":"2024-05-27T00:33:33.895827+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:33:33.895827+0000","last_undegraded":"2024-05-27T00:33:33.895827+0000","last_fullsized":"2024-05-27T00:33:33.895827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:14.148909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028015244000000002,"stat_sum":{"num_bytes":15735198,"num_objects":613,"num_object_clones":289,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":202,"num_read":48,"num_read_kb":16478,"num_write":1397,"num_write_kb":22754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":231,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:32:10.519872+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:32:10.519872+0000","last_peered":"2024-05-27T00:32:10.519872+0000","last_clean":"2024-05-27T00:32:10.519872+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:32:10.519872+0000","last_undegraded":"2024-05-27T00:32:10.519872+0000","last_fullsized":"2024-05-27T00:32:10.519872+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"48'875","reported_seq":1072,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.499832+0000","last_change":"2024-05-27T00:15:24.603781+0000","last_active":"2024-05-27T00:33:33.499832+0000","last_peered":"2024-05-27T00:33:33.499832+0000","last_clean":"2024-05-27T00:33:33.499832+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:33:33.499832+0000","last_undegraded":"2024-05-27T00:33:33.499832+0000","last_fullsized":"2024-05-27T00:33:33.499832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":219,"num_read_kb":131843,"num_write":965,"num_write_kb":67539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"48'1616","reported_seq":3478,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.386594+0000","last_change":"2024-05-27T00:17:54.309203+0000","last_active":"2024-05-27T00:33:33.386594+0000","last_peered":"2024-05-27T00:33:33.386594+0000","last_clean":"2024-05-27T00:33:33.386594+0000","last_became_active":"2024-05-27T00:16:06.517816+0000","last_became_peered":"2024-05-27T00:16:06.517816+0000","last_unstale":"2024-05-27T00:33:33.386594+0000","last_undegraded":"2024-05-27T00:33:33.386594+0000","last_fullsized":"2024-05-27T00:33:33.386594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:03:08.300384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085828740000000008,"stat_sum":{"num_bytes":27686043,"num_objects":633,"num_object_clones":292,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":198,"num_read":54,"num_read_kb":34546,"num_write":1470,"num_write_kb":32830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"48'429","reported_seq":666,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.891792+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:33:33.891792+0000","last_peered":"2024-05-27T00:33:33.891792+0000","last_clean":"2024-05-27T00:33:33.891792+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:33:33.891792+0000","last_undegraded":"2024-05-27T00:33:33.891792+0000","last_fullsized":"2024-05-27T00:33:33.891792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":299,"num_read_kb":174228,"num_write":552,"num_write_kb":63235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"48'1520","reported_seq":3388,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.382336+0000","last_change":"2024-05-27T00:17:54.490594+0000","last_active":"2024-05-27T00:33:33.382336+0000","last_peered":"2024-05-27T00:33:33.382336+0000","last_clean":"2024-05-27T00:33:33.382336+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:33:33.382336+0000","last_undegraded":"2024-05-27T00:33:33.382336+0000","last_fullsized":"2024-05-27T00:33:33.382336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:45:47.096794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016778365999999999,"stat_sum":{"num_bytes":6046316,"num_objects":564,"num_object_clones":261,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":182,"num_read":34,"num_read_kb":53,"num_write":1394,"num_write_kb":7687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'1654","reported_seq":2540,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.495949+0000","last_change":"2024-05-27T00:15:24.598182+0000","last_active":"2024-05-27T00:33:33.495949+0000","last_peered":"2024-05-27T00:33:33.495949+0000","last_clean":"2024-05-27T00:33:33.495949+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:33:33.495949+0000","last_undegraded":"2024-05-27T00:33:33.495949+0000","last_fullsized":"2024-05-27T00:33:33.495949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1654,"log_dups_size":0,"ondisk_log_size":1654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":85,"num_object_clones":0,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":1008,"num_read_kb":169683,"num_write":1827,"num_write_kb":85277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"48'1615","reported_seq":3473,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.909399+0000","last_change":"2024-05-27T00:17:54.491784+0000","last_active":"2024-05-27T00:33:33.909399+0000","last_peered":"2024-05-27T00:33:33.909399+0000","last_clean":"2024-05-27T00:33:33.909399+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:33:33.909399+0000","last_undegraded":"2024-05-27T00:33:33.909399+0000","last_fullsized":"2024-05-27T00:33:33.909399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:22.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016042048999999999,"stat_sum":{"num_bytes":11249311,"num_objects":630,"num_object_clones":289,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":203,"num_read":51,"num_read_kb":12401,"num_write":1463,"num_write_kb":13659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"48'708","reported_seq":967,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.358425+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:33:30.358425+0000","last_peered":"2024-05-27T00:33:30.358425+0000","last_clean":"2024-05-27T00:33:30.358425+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:33:30.358425+0000","last_undegraded":"2024-05-27T00:33:30.358425+0000","last_fullsized":"2024-05-27T00:33:30.358425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":344,"num_read_kb":168323,"num_write":926,"num_write_kb":68366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"48'1642","reported_seq":3512,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.350257+0000","last_change":"2024-05-27T00:17:54.505548+0000","last_active":"2024-05-27T00:33:30.350257+0000","last_peered":"2024-05-27T00:33:30.350257+0000","last_clean":"2024-05-27T00:33:30.350257+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:33:30.350257+0000","last_undegraded":"2024-05-27T00:33:30.350257+0000","last_fullsized":"2024-05-27T00:33:30.350257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:08.630844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021175818999999999,"stat_sum":{"num_bytes":27156672,"num_objects":598,"num_object_clones":274,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":182,"num_read":49,"num_read_kb":37771,"num_write":1519,"num_write_kb":29445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"48'814","reported_seq":1035,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.885329+0000","last_change":"2024-05-27T00:15:24.603518+0000","last_active":"2024-05-27T00:33:33.885329+0000","last_peered":"2024-05-27T00:33:33.885329+0000","last_clean":"2024-05-27T00:33:33.885329+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:33:33.885329+0000","last_undegraded":"2024-05-27T00:33:33.885329+0000","last_fullsized":"2024-05-27T00:33:33.885329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":262,"num_read_kb":165581,"num_write":935,"num_write_kb":83695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"48'1593","reported_seq":3458,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.920096+0000","last_change":"2024-05-27T00:17:54.511904+0000","last_active":"2024-05-27T00:33:30.920096+0000","last_peered":"2024-05-27T00:33:30.920096+0000","last_clean":"2024-05-27T00:33:30.920096+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:33:30.920096+0000","last_undegraded":"2024-05-27T00:33:30.920096+0000","last_fullsized":"2024-05-27T00:33:30.920096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1593,"log_dups_size":0,"ondisk_log_size":1593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:58:08.710145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034092800999999999,"stat_sum":{"num_bytes":22484461,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":211,"num_read":50,"num_read_kb":38442,"num_write":1413,"num_write_kb":28463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"48'1412","reported_seq":1652,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.537498+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:33:34.537498+0000","last_peered":"2024-05-27T00:33:34.537498+0000","last_clean":"2024-05-27T00:33:34.537498+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:33:34.537498+0000","last_undegraded":"2024-05-27T00:33:34.537498+0000","last_fullsized":"2024-05-27T00:33:34.537498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1412,"log_dups_size":0,"ondisk_log_size":1412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":299,"num_read_kb":161725,"num_write":1529,"num_write_kb":67545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"48'1569","reported_seq":3448,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.363078+0000","last_change":"2024-05-27T00:17:54.488788+0000","last_active":"2024-05-27T00:33:33.363078+0000","last_peered":"2024-05-27T00:33:33.363078+0000","last_clean":"2024-05-27T00:33:33.363078+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:33:33.363078+0000","last_undegraded":"2024-05-27T00:33:33.363078+0000","last_fullsized":"2024-05-27T00:33:33.363078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:42:30.595550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014512239,"stat_sum":{"num_bytes":23092289,"num_objects":584,"num_object_clones":267,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":187,"num_read":51,"num_read_kb":31023,"num_write":1448,"num_write_kb":28334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'833","reported_seq":1070,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.637756+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:33:34.637756+0000","last_peered":"2024-05-27T00:33:34.637756+0000","last_clean":"2024-05-27T00:33:34.637756+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:33:34.637756+0000","last_undegraded":"2024-05-27T00:33:34.637756+0000","last_fullsized":"2024-05-27T00:33:34.637756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":833,"log_dups_size":0,"ondisk_log_size":833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":312,"num_read_kb":117165,"num_write":1596,"num_write_kb":59298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"48'1511","reported_seq":3386,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.500418+0000","last_change":"2024-05-27T00:17:54.310241+0000","last_active":"2024-05-27T00:33:30.500418+0000","last_peered":"2024-05-27T00:33:30.500418+0000","last_clean":"2024-05-27T00:33:30.500418+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:33:30.500418+0000","last_undegraded":"2024-05-27T00:33:30.500418+0000","last_fullsized":"2024-05-27T00:33:30.500418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:45:51.525130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089509419999999999,"stat_sum":{"num_bytes":13250576,"num_objects":583,"num_object_clones":269,"num_object_copies":1166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":181,"num_read":49,"num_read_kb":2453,"num_write":1381,"num_write_kb":19589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"48'643","reported_seq":895,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.706802+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:33:34.706802+0000","last_peered":"2024-05-27T00:33:34.706802+0000","last_clean":"2024-05-27T00:33:34.706802+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:33:34.706802+0000","last_undegraded":"2024-05-27T00:33:34.706802+0000","last_fullsized":"2024-05-27T00:33:34.706802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":352,"num_read_kb":95868,"num_write":810,"num_write_kb":47576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"48'1573","reported_seq":3446,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.501372+0000","last_change":"2024-05-27T00:17:54.492705+0000","last_active":"2024-05-27T00:33:30.501372+0000","last_peered":"2024-05-27T00:33:30.501372+0000","last_clean":"2024-05-27T00:33:30.501372+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:33:30.501372+0000","last_undegraded":"2024-05-27T00:33:30.501372+0000","last_fullsized":"2024-05-27T00:33:30.501372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:40:53.350258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018225775999999999,"stat_sum":{"num_bytes":9758467,"num_objects":579,"num_object_clones":270,"num_object_copies":1158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":186,"num_read":49,"num_read_kb":4155,"num_write":1423,"num_write_kb":16751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'471","reported_seq":692,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.886300+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:33:30.886300+0000","last_peered":"2024-05-27T00:33:30.886300+0000","last_clean":"2024-05-27T00:33:30.886300+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:33:30.886300+0000","last_undegraded":"2024-05-27T00:33:30.886300+0000","last_fullsized":"2024-05-27T00:33:30.886300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":297,"num_read_kb":59216,"num_write":633,"num_write_kb":42916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":70,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"48'1624","reported_seq":3487,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.893727+0000","last_change":"2024-05-27T00:17:54.504734+0000","last_active":"2024-05-27T00:33:33.893727+0000","last_peered":"2024-05-27T00:33:33.893727+0000","last_clean":"2024-05-27T00:33:33.893727+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:33:33.893727+0000","last_undegraded":"2024-05-27T00:33:33.893727+0000","last_fullsized":"2024-05-27T00:33:33.893727+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:34:00.119528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022802596000000001,"stat_sum":{"num_bytes":19569863,"num_objects":620,"num_object_clones":284,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":196,"num_read":52,"num_read_kb":30708,"num_write":1470,"num_write_kb":21076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"48'783","reported_seq":953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.177344+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:33:30.177344+0000","last_peered":"2024-05-27T00:33:30.177344+0000","last_clean":"2024-05-27T00:33:30.177344+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:33:30.177344+0000","last_undegraded":"2024-05-27T00:33:30.177344+0000","last_fullsized":"2024-05-27T00:33:30.177344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":783,"log_dups_size":0,"ondisk_log_size":783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":176,"num_read_kb":80391,"num_write":878,"num_write_kb":51066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"48'1663","reported_seq":3525,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.385750+0000","last_change":"2024-05-27T00:17:54.308939+0000","last_active":"2024-05-27T00:33:33.385750+0000","last_peered":"2024-05-27T00:33:33.385750+0000","last_clean":"2024-05-27T00:33:33.385750+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:33:33.385750+0000","last_undegraded":"2024-05-27T00:33:33.385750+0000","last_fullsized":"2024-05-27T00:33:33.385750+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:17:28.826160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084477519999999993,"stat_sum":{"num_bytes":19800736,"num_objects":636,"num_object_clones":295,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":205,"num_read":62,"num_read_kb":32956,"num_write":1508,"num_write_kb":26318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"48'1047","reported_seq":1271,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.532477+0000","last_change":"2024-05-27T00:15:24.598238+0000","last_active":"2024-05-27T00:33:34.532477+0000","last_peered":"2024-05-27T00:33:34.532477+0000","last_clean":"2024-05-27T00:33:34.532477+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:33:34.532477+0000","last_undegraded":"2024-05-27T00:33:34.532477+0000","last_fullsized":"2024-05-27T00:33:34.532477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1047,"log_dups_size":0,"ondisk_log_size":1047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":269,"num_read_kb":153672,"num_write":1132,"num_write_kb":75680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"48'1615","reported_seq":3456,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.386744+0000","last_change":"2024-05-27T00:17:54.310021+0000","last_active":"2024-05-27T00:33:33.386744+0000","last_peered":"2024-05-27T00:33:33.386744+0000","last_clean":"2024-05-27T00:33:33.386744+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:33:33.386744+0000","last_undegraded":"2024-05-27T00:33:33.386744+0000","last_fullsized":"2024-05-27T00:33:33.386744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:07:28.174929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091147599999999995,"stat_sum":{"num_bytes":15827867,"num_objects":624,"num_object_clones":289,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":199,"num_read":35,"num_read_kb":8228,"num_write":1456,"num_write_kb":20842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"48'689","reported_seq":942,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.707187+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:33:34.707187+0000","last_peered":"2024-05-27T00:33:34.707187+0000","last_clean":"2024-05-27T00:33:34.707187+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:33:34.707187+0000","last_undegraded":"2024-05-27T00:33:34.707187+0000","last_fullsized":"2024-05-27T00:33:34.707187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":689,"log_dups_size":0,"ondisk_log_size":689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":331,"num_read_kb":161875,"num_write":831,"num_write_kb":80242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"48'1587","reported_seq":3445,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.396200+0000","last_change":"2024-05-27T00:17:54.333938+0000","last_active":"2024-05-27T00:33:30.396200+0000","last_peered":"2024-05-27T00:33:30.396200+0000","last_clean":"2024-05-27T00:33:30.396200+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:33:30.396200+0000","last_undegraded":"2024-05-27T00:33:30.396200+0000","last_fullsized":"2024-05-27T00:33:30.396200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1587,"log_dups_size":0,"ondisk_log_size":1587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:22.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026544363000000001,"stat_sum":{"num_bytes":11103766,"num_objects":576,"num_object_clones":273,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":173,"num_read":36,"num_read_kb":3645,"num_write":1460,"num_write_kb":12414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"48'1524","reported_seq":3388,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.922267+0000","last_change":"2024-05-27T00:17:54.485932+0000","last_active":"2024-05-27T00:33:33.922267+0000","last_peered":"2024-05-27T00:33:33.922267+0000","last_clean":"2024-05-27T00:33:33.922267+0000","last_became_active":"2024-05-27T00:16:06.811922+0000","last_became_peered":"2024-05-27T00:16:06.811922+0000","last_unstale":"2024-05-27T00:33:33.922267+0000","last_undegraded":"2024-05-27T00:33:33.922267+0000","last_fullsized":"2024-05-27T00:33:33.922267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:29:39.727324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011085174,"stat_sum":{"num_bytes":14346371,"num_objects":587,"num_object_clones":272,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":198,"num_read":48,"num_read_kb":4199,"num_write":1374,"num_write_kb":25831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"48'490","reported_seq":703,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.794312+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:33:34.794312+0000","last_peered":"2024-05-27T00:33:34.794312+0000","last_clean":"2024-05-27T00:33:34.794312+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:33:34.794312+0000","last_undegraded":"2024-05-27T00:33:34.794312+0000","last_fullsized":"2024-05-27T00:33:34.794312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":267,"num_read_kb":124158,"num_write":632,"num_write_kb":63683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"48'1663","reported_seq":3491,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.392971+0000","last_change":"2024-05-27T00:17:54.511007+0000","last_active":"2024-05-27T00:33:33.392971+0000","last_peered":"2024-05-27T00:33:33.392971+0000","last_clean":"2024-05-27T00:33:33.392971+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:33:33.392971+0000","last_undegraded":"2024-05-27T00:33:33.392971+0000","last_fullsized":"2024-05-27T00:33:33.392971+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:43:33.746539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034279472999999998,"stat_sum":{"num_bytes":10536557,"num_objects":647,"num_object_clones":309,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":224,"num_read":43,"num_read_kb":11171,"num_write":1484,"num_write_kb":16947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"48'860","reported_seq":1117,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.539304+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:33:34.539304+0000","last_peered":"2024-05-27T00:33:34.539304+0000","last_clean":"2024-05-27T00:33:34.539304+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:33:34.539304+0000","last_undegraded":"2024-05-27T00:33:34.539304+0000","last_fullsized":"2024-05-27T00:33:34.539304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":860,"log_dups_size":0,"ondisk_log_size":860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":337,"num_read_kb":206604,"num_write":984,"num_write_kb":108700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"48'1538","reported_seq":3409,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:30.343921+0000","last_change":"2024-05-27T00:17:54.493359+0000","last_active":"2024-05-27T00:33:30.343921+0000","last_peered":"2024-05-27T00:33:30.343921+0000","last_clean":"2024-05-27T00:33:30.343921+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:33:30.343921+0000","last_undegraded":"2024-05-27T00:33:30.343921+0000","last_fullsized":"2024-05-27T00:33:30.343921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1538,"log_dups_size":0,"ondisk_log_size":1538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:32:56.589612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017325917,"stat_sum":{"num_bytes":22746478,"num_objects":600,"num_object_clones":283,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":186,"num_read":59,"num_read_kb":40021,"num_write":1401,"num_write_kb":24139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"48'1273","reported_seq":1530,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.702528+0000","last_change":"2024-05-27T00:15:24.600571+0000","last_active":"2024-05-27T00:33:34.702528+0000","last_peered":"2024-05-27T00:33:34.702528+0000","last_clean":"2024-05-27T00:33:34.702528+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:33:34.702528+0000","last_undegraded":"2024-05-27T00:33:34.702528+0000","last_fullsized":"2024-05-27T00:33:34.702528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1273,"log_dups_size":0,"ondisk_log_size":1273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:44:21.645223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":353,"num_read_kb":145683,"num_write":1459,"num_write_kb":63890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"48'1559","reported_seq":3435,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:33.394563+0000","last_change":"2024-05-27T00:17:54.309879+0000","last_active":"2024-05-27T00:33:33.394563+0000","last_peered":"2024-05-27T00:33:33.394563+0000","last_clean":"2024-05-27T00:33:33.394563+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:33:33.394563+0000","last_undegraded":"2024-05-27T00:33:33.394563+0000","last_fullsized":"2024-05-27T00:33:33.394563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1559,"log_dups_size":0,"ondisk_log_size":1559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:20:48.824140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089714449999999998,"stat_sum":{"num_bytes":23504918,"num_objects":585,"num_object_clones":273,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":188,"num_read":53,"num_read_kb":39809,"num_write":1415,"num_write_kb":24852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"48'514","reported_seq":753,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:34.528633+0000","last_change":"2024-05-27T00:15:24.615066+0000","last_active":"2024-05-27T00:33:34.528633+0000","last_peered":"2024-05-27T00:33:34.528633+0000","last_clean":"2024-05-27T00:33:34.528633+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:33:34.528633+0000","last_undegraded":"2024-05-27T00:33:34.528633+0000","last_fullsized":"2024-05-27T00:33:34.528633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:28:55.614624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":327,"num_read_kb":63931,"num_write":692,"num_write_kb":47409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1174241513,"num_objects":39009,"num_object_clones":18298,"num_object_copies":78018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39009,"num_whiteouts":12489,"num_read":3379,"num_read_kb":1679737,"num_write":92416,"num_write_kb":1475210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2440749056,"data_stored":2350449106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102162,"ondisk_log_size":102162,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":781876696,"num_objects":2104,"num_object_clones":0,"num_object_copies":4208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":10396,"num_read_kb":4541569,"num_write":33862,"num_write_kb":2225576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1834,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1563811840,"data_stored":1563753392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28655,"ondisk_log_size":28655,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542368,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746260,"kb_used_data":606860,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93625580,"statfs":{"total":96636764160,"available":95872593920,"internally_reserved":0,"allocated":621424640,"data_stored":608366707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142735818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542369,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573040,"kb_used_data":443368,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93798800,"statfs":{"total":96636764160,"available":96049971200,"internally_reserved":0,"allocated":454008832,"data_stored":445134014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542370,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741064,"kb_used_data":583616,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93630776,"statfs":{"total":96636764160,"available":95877914624,"internally_reserved":0,"allocated":597622784,"data_stored":584115090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161216965},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542368,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662452,"kb_used_data":518892,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93709388,"statfs":{"total":96636764160,"available":95958413312,"internally_reserved":0,"allocated":531345408,"data_stored":517393413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542369,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":656408,"kb_used_data":521168,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93715432,"statfs":{"total":96636764160,"available":95964602368,"internally_reserved":0,"allocated":533676032,"data_stored":519848185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138475975},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542368,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439708,"kb_used_data":336404,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93932132,"statfs":{"total":96636764160,"available":96186503168,"internally_reserved":0,"allocated":344477696,"data_stored":336181610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105773509},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542369,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573740,"kb_used_data":426020,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93798100,"statfs":{"total":96636764160,"available":96049254400,"internally_reserved":0,"allocated":436244480,"data_stored":428173666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151255495},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542368,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623352,"kb_used_data":479792,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93748488,"statfs":{"total":96636764160,"available":95998451712,"internally_reserved":0,"allocated":491307008,"data_stored":478155533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":146995653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167784448,"data_stored":167772798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":223674368,"data_stored":223666222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125829120,"data_stored":125829120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192946176,"data_stored":192938656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":160763904,"data_stored":160751628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":214282240,"data_stored":214276326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":222674944,"data_stored":222665516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":255856640,"data_stored":255853126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":322383872,"data_stored":309544244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212025344,"data_stored":204259321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218103808,"data_stored":210104367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340185088,"data_stored":326661406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":370036736,"data_stored":356393662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":382795776,"data_stored":369590641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":230789120,"data_stored":222220375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":364429312,"data_stored":351675090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:33:41.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:33:41.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa89d600640 1 Processor -- start 2024-05-27T00:33:41.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa89d600640 1 -- start start 2024-05-27T00:33:41.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa89d600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa8980735d0 0x7fa898151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa89d600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8980c17d0 con 0x7fa8980731f0 2024-05-27T00:33:41.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa89d600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa898070e10 con 0x7fa8980c79f0 2024-05-27T00:33:41.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa89d600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa898071990 con 0x7fa8980735d0 2024-05-27T00:33:41.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa896a00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa8980731f0 0x7fa8980c0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56926/0 (socket says 172.21.15.98:56926) 2024-05-27T00:33:41.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa896a00640 1 -- 172.21.15.98:0/2176111308 learned_addr learned my addr 172.21.15.98:0/2176111308 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:41.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa896000640 1 --2- 172.21.15.98:0/2176111308 >> v2:172.21.15.98:3300/0 conn(0x7fa8980735d0 0x7fa898151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa895600640 1 -- 172.21.15.98:0/2176111308 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1597750236 0 0) 0x7fa8980c17d0 con 0x7fa8980731f0 2024-05-27T00:33:41.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa895600640 1 -- 172.21.15.98:0/2176111308 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa87c003530 con 0x7fa8980731f0 2024-05-27T00:33:41.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa896000640 1 -- 172.21.15.98:0/2176111308 >> v1:172.21.15.204:6789/0 conn(0x7fa8980c79f0 legacy=0x7fa898157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa896000640 1 -- 172.21.15.98:0/2176111308 >> v1:172.21.15.98:6789/0 conn(0x7fa8980731f0 legacy=0x7fa8980c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.790+0000 7fa896000640 1 -- 172.21.15.98:0/2176111308 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8980710f0 con 0x7fa8980735d0 2024-05-27T00:33:41.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa896000640 1 --2- 172.21.15.98:0/2176111308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 0x7fa898151430 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fa898071b10 tx=0x7fa88401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ffd16d9c1f2e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:41.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa895600640 1 -- 172.21.15.98:0/2176111308 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8840b0070 con 0x7fa8980735d0 2024-05-27T00:33:41.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa895600640 1 -- 172.21.15.98:0/2176111308 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8840ab600 con 0x7fa8980735d0 2024-05-27T00:33:41.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa895600640 1 -- 172.21.15.98:0/2176111308 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8840ab9d0 con 0x7fa8980735d0 2024-05-27T00:33:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa89d600640 1 -- 172.21.15.98:0/2176111308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 msgr2=0x7fa898151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa89d600640 1 --2- 172.21.15.98:0/2176111308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 0x7fa898151430 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fa898071b10 tx=0x7fa88401c4a0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa89d600640 1 -- 172.21.15.98:0/2176111308 shutdown_connections 2024-05-27T00:33:41.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa89d600640 1 --2- 172.21.15.98:0/2176111308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 0x7fa898151430 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa89d600640 1 -- 172.21.15.98:0/2176111308 >> 172.21.15.98:0/2176111308 conn(0x7fa898067530 msgr2=0x7fa89806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:41.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa89d600640 1 -- 172.21.15.98:0/2176111308 shutdown_connections 2024-05-27T00:33:41.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.794+0000 7fa89d600640 1 -- 172.21.15.98:0/2176111308 wait complete. 2024-05-27T00:33:41.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 Processor -- start 2024-05-27T00:33:41.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 -- start start 2024-05-27T00:33:41.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8980731f0 0x7fa898159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 0x7fa89814e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8980c79f0 0x7fa89815b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa8980c8ef0 con 0x7fa8980735d0 2024-05-27T00:33:41.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa898151d80 con 0x7fa8980c79f0 2024-05-27T00:33:41.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa8980c9070 con 0x7fa8980731f0 2024-05-27T00:33:41.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa896000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 0x7fa89814e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa896000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 0x7fa89814e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59800/0 (socket says 172.21.15.98:59800) 2024-05-27T00:33:41.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa896000640 1 -- 172.21.15.98:0/1603696165 learned_addr learned my addr 172.21.15.98:0/1603696165 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:41.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa897400640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8980c79f0 0x7fa89815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa896a00640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8980731f0 0x7fa898159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa896000640 1 -- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8980731f0 msgr2=0x7fa898159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa896000640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8980731f0 0x7fa898159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa896000640 1 -- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8980c79f0 msgr2=0x7fa89815b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa896000640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8980c79f0 0x7fa89815b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa896000640 1 -- 172.21.15.98:0/1603696165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa89814cb60 con 0x7fa8980735d0 2024-05-27T00:33:41.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa896000640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 0x7fa89814e0d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa8840ae040 tx=0x7fa884019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:41.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa88ea00640 1 -- 172.21.15.98:0/1603696165 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8840b0040 con 0x7fa8980735d0 2024-05-27T00:33:41.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa88ea00640 1 -- 172.21.15.98:0/1603696165 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8840b9070 con 0x7fa8980735d0 2024-05-27T00:33:41.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa88ea00640 1 -- 172.21.15.98:0/1603696165 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8840b4650 con 0x7fa8980735d0 2024-05-27T00:33:41.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 -- 172.21.15.98:0/1603696165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa89814ce40 con 0x7fa8980735d0 2024-05-27T00:33:41.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 -- 172.21.15.98:0/1603696165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa898158ad0 con 0x7fa8980735d0 2024-05-27T00:33:41.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.798+0000 7fa89d600640 1 -- 172.21.15.98:0/1603696165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa898075cb0 con 0x7fa8980735d0 2024-05-27T00:33:41.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.802+0000 7fa88ea00640 1 -- 172.21.15.98:0/1603696165 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa8840b47f0 con 0x7fa8980735d0 2024-05-27T00:33:41.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.802+0000 7fa88ea00640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa8780678c0 0x7fa878069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:41.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.802+0000 7fa88ea00640 1 -- 172.21.15.98:0/1603696165 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fa88411ff20 con 0x7fa8980735d0 2024-05-27T00:33:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.802+0000 7fa896a00640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa8780678c0 0x7fa878069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.802+0000 7fa896a00640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa8780678c0 0x7fa878069d80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa8880011f0 tx=0x7fa888009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:41.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.806+0000 7fa88ea00640 1 -- 172.21.15.98:0/1603696165 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa884016090 con 0x7fa8980735d0 2024-05-27T00:33:41.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.962+0000 7fa89d600640 1 -- 172.21.15.98:0/1603696165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa89814f510 con 0x7fa8980735d0 2024-05-27T00:33:41.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa88ea00640 1 -- 172.21.15.98:0/1603696165 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+18411 (secure 0 0 0) 0x7fa8840f0540 con 0x7fa8980735d0 2024-05-27T00:33:41.965 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:33:41.966 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:32:10.313136+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6834/3070334683":"2024-05-28T00:32:10.313051+0000","172.21.15.98:6836/1802902856":"2024-05-28T00:30:40.295945+0000","172.21.15.98:6834/2148124202":"2024-05-28T00:28:50.279743+0000","172.21.15.98:6832/2148124202":"2024-05-28T00:28:50.279743+0000","172.21.15.98:6837/2940211164":"2024-05-28T00:27:55.271086+0000","172.21.15.98:6836/2940211164":"2024-05-28T00:27:55.271086+0000","172.21.15.98:6834/1223646158":"2024-05-28T00:27:00.259848+0000","172.21.15.98:6837/518550803":"2024-05-28T00:24:50.239991+0000","172.21.15.98:6836/3824013488":"2024-05-28T00:16:25.159349+0000","172.21.15.98:6832/3070334683":"2024-05-28T00:32:10.313051+0000","172.21.15.98:6832/3247908011":"2024-05-28T00:21:25.205263+0000","172.21.15.98:6834/4290077910":"2024-05-28T00:23:40.226716+0000","172.21.15.98:6832/4290077910":"2024-05-28T00:23:40.226716+0000","172.21.15.98:6837/1802902856":"2024-05-28T00:30:40.295945+0000","172.21.15.98:6832/3903574515":"2024-05-28T00:18:25.179026+0000","172.21.15.98:6837/3824013488":"2024-05-28T00:16:25.159349+0000","172.21.15.98:6836/1781395818":"2024-05-28T00:19:35.189766+0000","172.21.15.98:6834/3903574515":"2024-05-28T00:18:25.179026+0000","172.21.15.98:6837/1781395818":"2024-05-28T00:19:35.189766+0000","172.21.15.98:6834/3247908011":"2024-05-28T00:21:25.205263+0000","172.21.15.98:6832/1223646158":"2024-05-28T00:27:00.259848+0000","172.21.15.98:6836/1698253956":"2024-05-28T00:22:20.213891+0000","172.21.15.98:6836/518550803":"2024-05-28T00:24:50.239991+0000","172.21.15.98:6837/1698253956":"2024-05-28T00:22:20.213891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:33:41.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 -- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa8780678c0 msgr2=0x7fa878069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa8780678c0 0x7fa878069d80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa8880011f0 tx=0x7fa888009040 comp rx=0 tx=0).stop 2024-05-27T00:33:41.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 -- 172.21.15.98:0/1603696165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 msgr2=0x7fa89814e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:41.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 0x7fa89814e0d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa8840ae040 tx=0x7fa884019240 comp rx=0 tx=0).stop 2024-05-27T00:33:41.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 -- 172.21.15.98:0/1603696165 shutdown_connections 2024-05-27T00:33:41.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa8780678c0 0x7fa878069d80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8980c79f0 0x7fa89815b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8980735d0 0x7fa89814e0d0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 --2- 172.21.15.98:0/1603696165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8980731f0 0x7fa898159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:41.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 -- 172.21.15.98:0/1603696165 >> 172.21.15.98:0/1603696165 conn(0x7fa898067530 msgr2=0x7fa8980c6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:41.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 -- 172.21.15.98:0/1603696165 shutdown_connections 2024-05-27T00:33:41.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:41.966+0000 7fa89d600640 1 -- 172.21.15.98:0/1603696165 wait complete. 2024-05-27T00:33:42.991 INFO:tasks.ceph:Scrubbing osd.0 2024-05-27T00:33:42.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-27T00:33:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8be00640 1 Processor -- start 2024-05-27T00:33:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8be00640 1 -- start start 2024-05-27T00:33:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8be00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fea8c1528a0 0x7fea8c0c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea8c0b3e20 0x7fea8c06f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8be00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea8c158700 con 0x7fea8c0c79c0 2024-05-27T00:33:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8be00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fea8c0c8df0 con 0x7fea8c1528a0 2024-05-27T00:33:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fea8c1589e0 con 0x7fea8c0b3e20 2024-05-27T00:33:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8b400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fea8c0c79c0 0x7fea8c06d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56940/0 (socket says 172.21.15.98:56940) 2024-05-27T00:33:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8b400640 1 -- 172.21.15.98:0/2611817697 learned_addr learned my addr 172.21.15.98:0/2611817697 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8aa00640 1 --2- 172.21.15.98:0/2611817697 >> v2:172.21.15.204:3300/0 conn(0x7fea8c1528a0 0x7fea8c0c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea83e00640 1 --2- 172.21.15.98:0/2611817697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea8c0b3e20 0x7fea8c06f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8a000640 1 -- 172.21.15.98:0/2611817697 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 595717498 0 0) 0x7fea8c158700 con 0x7fea8c0c79c0 2024-05-27T00:33:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8a000640 1 -- 172.21.15.98:0/2611817697 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea70003530 con 0x7fea8c0c79c0 2024-05-27T00:33:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8aa00640 1 -- 172.21.15.98:0/2611817697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea8c0b3e20 msgr2=0x7fea8c06f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8aa00640 1 --2- 172.21.15.98:0/2611817697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea8c0b3e20 0x7fea8c06f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8aa00640 1 -- 172.21.15.98:0/2611817697 >> v1:172.21.15.98:6789/0 conn(0x7fea8c0c79c0 legacy=0x7fea8c06d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.142+0000 7fea8aa00640 1 -- 172.21.15.98:0/2611817697 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c158cc0 con 0x7fea8c1528a0 2024-05-27T00:33:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8aa00640 1 --2- 172.21.15.98:0/2611817697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8c1528a0 0x7fea8c0c87b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0097c0 tx=0x7fea7c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef343784dd84e2f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8a000640 1 -- 172.21.15.98:0/2611817697 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7c0b5070 con 0x7fea8c1528a0 2024-05-27T00:33:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8a000640 1 -- 172.21.15.98:0/2611817697 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea7c0a8990 con 0x7fea8c1528a0 2024-05-27T00:33:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8a000640 1 -- 172.21.15.98:0/2611817697 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7c0a8d40 con 0x7fea8c1528a0 2024-05-27T00:33:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8be00640 1 -- 172.21.15.98:0/2611817697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8c1528a0 msgr2=0x7fea8c0c87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8be00640 1 --2- 172.21.15.98:0/2611817697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8c1528a0 0x7fea8c0c87b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0097c0 tx=0x7fea7c01ec90 comp rx=0 tx=0).stop 2024-05-27T00:33:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8be00640 1 -- 172.21.15.98:0/2611817697 shutdown_connections 2024-05-27T00:33:43.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8be00640 1 --2- 172.21.15.98:0/2611817697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea8c0b3e20 0x7fea8c06f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8be00640 1 --2- 172.21.15.98:0/2611817697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8c1528a0 0x7fea8c0c87b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8be00640 1 -- 172.21.15.98:0/2611817697 >> 172.21.15.98:0/2611817697 conn(0x7fea8c0ca9b0 msgr2=0x7fea8c0cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:43.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8be00640 1 -- 172.21.15.98:0/2611817697 shutdown_connections 2024-05-27T00:33:43.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.146+0000 7fea8be00640 1 -- 172.21.15.98:0/2611817697 wait complete. 2024-05-27T00:33:43.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8be00640 1 Processor -- start 2024-05-27T00:33:43.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8be00640 1 -- start start 2024-05-27T00:33:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8c0b3e20 0x7fea8c06cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea8c0c79c0 0x7fea8c1f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea8c1528a0 0x7fea8c1f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8be00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c06de90 con 0x7fea8c1528a0 2024-05-27T00:33:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c159280 con 0x7fea8c0b3e20 2024-05-27T00:33:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fea8c06e010 con 0x7fea8c0c79c0 2024-05-27T00:33:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea8c1528a0 0x7fea8c1f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8c0b3e20 0x7fea8c06cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea8c1528a0 0x7fea8c1f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59804/0 (socket says 172.21.15.98:59804) 2024-05-27T00:33:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8b400640 1 -- 172.21.15.98:0/2554557874 learned_addr learned my addr 172.21.15.98:0/2554557874 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea83e00640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea8c0c79c0 0x7fea8c1f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8b400640 1 -- 172.21.15.98:0/2554557874 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea8c0c79c0 msgr2=0x7fea8c1f21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8b400640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea8c0c79c0 0x7fea8c1f21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8b400640 1 -- 172.21.15.98:0/2554557874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8c0b3e20 msgr2=0x7fea8c06cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8b400640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8c0b3e20 0x7fea8c06cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8b400640 1 -- 172.21.15.98:0/2554557874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c15c1e0 con 0x7fea8c1528a0 2024-05-27T00:33:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8b400640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea8c1528a0 0x7fea8c1f26e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fea8400cad0 tx=0x7fea8400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea82a00640 1 -- 172.21.15.98:0/2554557874 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea84017070 con 0x7fea8c1528a0 2024-05-27T00:33:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea82a00640 1 -- 172.21.15.98:0/2554557874 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8400ac30 con 0x7fea8c1528a0 2024-05-27T00:33:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea82a00640 1 -- 172.21.15.98:0/2554557874 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea84012400 con 0x7fea8c1528a0 2024-05-27T00:33:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8c15b660 con 0x7fea8c1528a0 2024-05-27T00:33:43.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.150+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea8c15bc20 con 0x7fea8c1528a0 2024-05-27T00:33:43.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.154+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fea8c075c40 con 0x7fea8c1528a0 2024-05-27T00:33:43.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.154+0000 7fea82a00640 1 -- 172.21.15.98:0/2554557874 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fea8401a070 con 0x7fea8c1528a0 2024-05-27T00:33:43.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.154+0000 7fea82a00640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fea680678f0 0x7fea68069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.154+0000 7fea82a00640 1 -- 172.21.15.98:0/2554557874 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fea840865b0 con 0x7fea8c1528a0 2024-05-27T00:33:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.154+0000 7fea82a00640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] conn(0x7fea680704f0 0x7fea68072970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.154+0000 7fea82a00640 1 -- 172.21.15.98:0/2554557874 --> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fea68073040 con 0x7fea680704f0 2024-05-27T00:33:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.154+0000 7fea82a00640 1 -- 172.21.15.98:0/2554557874 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fea84086960 con 0x7fea8c1528a0 2024-05-27T00:33:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.154+0000 7fea8aa00640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fea680678f0 0x7fea68069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.154+0000 7fea83e00640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] conn(0x7fea680704f0 0x7fea68072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.158+0000 7fea83e00640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] conn(0x7fea680704f0 0x7fea68072970 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.158+0000 7fea8aa00640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fea680678f0 0x7fea68069db0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0b2000 tx=0x7fea7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.158+0000 7fea82a00640 1 -- 172.21.15.98:0/2554557874 <== osd.0 v2:172.21.15.98:6816/4085423032 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea68073040 con 0x7fea680704f0 2024-05-27T00:33:43.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 --> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fea8c155ea0 con 0x7fea680704f0 2024-05-27T00:33:43.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea82a00640 1 -- 172.21.15.98:0/2554557874 <== osd.0 v2:172.21.15.98:6816/4085423032 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fea8c155ea0 con 0x7fea680704f0 2024-05-27T00:33:43.171 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-27T00:33:43.172 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-27T00:33:43.172 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-27T00:33:43.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 >> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] conn(0x7fea680704f0 msgr2=0x7fea68072970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.98:6816/4085423032,v1:172.21.15.98:6818/4085423032] conn(0x7fea680704f0 0x7fea68072970 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fea680678f0 msgr2=0x7fea68069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fea680678f0 0x7fea68069db0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0b2000 tx=0x7fea7c008040 comp rx=0 tx=0).stop 2024-05-27T00:33:43.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea8c1528a0 msgr2=0x7fea8c1f26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 --2- 172.21.15.98:0/2554557874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea8c1528a0 0x7fea8c1f26e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fea8400cad0 tx=0x7fea8400cf90 comp rx=0 tx=0).stop 2024-05-27T00:33:43.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8b400640 1 -- 172.21.15.98:0/2554557874 reap_dead start 2024-05-27T00:33:43.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 shutdown_connections 2024-05-27T00:33:43.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 >> 172.21.15.98:0/2554557874 conn(0x7fea8c0ca9b0 msgr2=0x7fea8c150c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:43.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 shutdown_connections 2024-05-27T00:33:43.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.170+0000 7fea8be00640 1 -- 172.21.15.98:0/2554557874 wait complete. 2024-05-27T00:33:43.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-27T00:33:43.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6873600640 1 Processor -- start 2024-05-27T00:33:43.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6873600640 1 -- start start 2024-05-27T00:33:43.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6873600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f686c152770 0x7f686c0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6873600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f686c158920 con 0x7f686c158490 2024-05-27T00:33:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6873600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f686c158be0 con 0x7f686c0c84e0 2024-05-27T00:33:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6873600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f686c159480 con 0x7f686c152770 2024-05-27T00:33:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6863e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f686c158490 0x7f686c06e890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37260/0 (socket says 172.21.15.98:37260) 2024-05-27T00:33:43.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6863e00640 1 -- 172.21.15.98:0/3264345101 learned_addr learned my addr 172.21.15.98:0/3264345101 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:43.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6870c00640 1 --2- 172.21.15.98:0/3264345101 >> v2:172.21.15.204:3300/0 conn(0x7f686c152770 0x7f686c0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f686be00640 1 -- 172.21.15.98:0/3264345101 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3053797785 0 0) 0x7f686c158920 con 0x7f686c158490 2024-05-27T00:33:43.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f686be00640 1 -- 172.21.15.98:0/3264345101 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6850003530 con 0x7f686c158490 2024-05-27T00:33:43.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6870c00640 1 -- 172.21.15.98:0/3264345101 >> v1:172.21.15.204:6789/0 conn(0x7f686c0c84e0 legacy=0x7f686c15c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6870c00640 1 -- 172.21.15.98:0/3264345101 >> v1:172.21.15.98:6789/0 conn(0x7f686c158490 legacy=0x7f686c06e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.326+0000 7f6870c00640 1 -- 172.21.15.98:0/3264345101 --> v2:172.21.15.204:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686c158ec0 con 0x7f686c152770 2024-05-27T00:33:43.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f6870c00640 1 --2- 172.21.15.98:0/3264345101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f686c152770 0x7f686c0c7fa0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f685c0097c0 tx=0x7f685c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1f188d688da1538 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f686be00640 1 -- 172.21.15.98:0/3264345101 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c0b1070 con 0x7f686c152770 2024-05-27T00:33:43.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f686be00640 1 -- 172.21.15.98:0/3264345101 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f685c0ac5f0 con 0x7f686c152770 2024-05-27T00:33:43.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f686be00640 1 -- 172.21.15.98:0/3264345101 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c0ac9c0 con 0x7f686c152770 2024-05-27T00:33:43.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f6873600640 1 -- 172.21.15.98:0/3264345101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f686c152770 msgr2=0x7f686c0c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f6873600640 1 --2- 172.21.15.98:0/3264345101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f686c152770 0x7f686c0c7fa0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f685c0097c0 tx=0x7f685c01c4a0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f6873600640 1 -- 172.21.15.98:0/3264345101 shutdown_connections 2024-05-27T00:33:43.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f6873600640 1 --2- 172.21.15.98:0/3264345101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f686c152770 0x7f686c0c7fa0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f6873600640 1 -- 172.21.15.98:0/3264345101 >> 172.21.15.98:0/3264345101 conn(0x7f686c0cc200 msgr2=0x7f686c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:43.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f6873600640 1 -- 172.21.15.98:0/3264345101 shutdown_connections 2024-05-27T00:33:43.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f6873600640 1 -- 172.21.15.98:0/3264345101 wait complete. 2024-05-27T00:33:43.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.330+0000 7f6873600640 1 Processor -- start 2024-05-27T00:33:43.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6873600640 1 -- start start 2024-05-27T00:33:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6873600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f686c0c84e0 0x7f686c14b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6873600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f686c152770 0x7f686c14e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6873600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c0c2b10 0x7f686c1f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6873600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f686c06efa0 con 0x7f686c0c2b10 2024-05-27T00:33:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6873600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f686c070fa0 con 0x7f686c0c84e0 2024-05-27T00:33:43.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6873600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f686c06f120 con 0x7f686c152770 2024-05-27T00:33:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6871600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c0c2b10 0x7f686c1f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6871600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c0c2b10 0x7f686c1f6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55150/0 (socket says 172.21.15.98:55150) 2024-05-27T00:33:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6871600640 1 -- 172.21.15.98:0/3237645203 learned_addr learned my addr 172.21.15.98:0/3237645203 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6870c00640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f686c0c84e0 0x7f686c14b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6863e00640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f686c152770 0x7f686c14e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6871600640 1 -- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f686c152770 msgr2=0x7f686c14e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6871600640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f686c152770 0x7f686c14e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6871600640 1 -- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f686c0c84e0 msgr2=0x7f686c14b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6871600640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f686c0c84e0 0x7f686c14b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6871600640 1 -- 172.21.15.98:0/3237645203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686c0c0fc0 con 0x7f686c0c2b10 2024-05-27T00:33:43.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6863e00640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f686c152770 0x7f686c14e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:43.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6871600640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c0c2b10 0x7f686c1f6680 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f686400c6c0 tx=0x7f686400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6869600640 1 -- 172.21.15.98:0/3237645203 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864003b80 con 0x7f686c0c2b10 2024-05-27T00:33:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6869600640 1 -- 172.21.15.98:0/3237645203 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6864003d20 con 0x7f686c0c2b10 2024-05-27T00:33:43.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6869600640 1 -- 172.21.15.98:0/3237645203 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68640052f0 con 0x7f686c0c2b10 2024-05-27T00:33:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6873600640 1 -- 172.21.15.98:0/3237645203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686c0c12a0 con 0x7f686c0c2b10 2024-05-27T00:33:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6873600640 1 -- 172.21.15.98:0/3237645203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f686c0c1b40 con 0x7f686c0c2b10 2024-05-27T00:33:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.334+0000 7f6873600640 1 -- 172.21.15.98:0/3237645203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f686c075cb0 con 0x7f686c0c2b10 2024-05-27T00:33:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.338+0000 7f6869600640 1 -- 172.21.15.98:0/3237645203 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6864005490 con 0x7f686c0c2b10 2024-05-27T00:33:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.338+0000 7f6869600640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68480678c0 0x7f6848069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.338+0000 7f6869600640 1 -- 172.21.15.98:0/3237645203 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f6864085ee0 con 0x7f686c0c2b10 2024-05-27T00:33:43.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.338+0000 7f6870c00640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68480678c0 0x7f6848069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.338+0000 7f6870c00640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68480678c0 0x7f6848069d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f685c0ae000 tx=0x7f685c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.342+0000 7f6869600640 1 -- 172.21.15.98:0/3237645203 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f6864056590 con 0x7f686c0c2b10 2024-05-27T00:33:43.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.498+0000 7f6873600640 1 -- 172.21.15.98:0/3237645203 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f686c068850 con 0x7f68480678c0 2024-05-27T00:33:43.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.498+0000 7f6869600640 1 -- 172.21.15.98:0/3237645203 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f686c068850 con 0x7f68480678c0 2024-05-27T00:33:43.499 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 0 to deep-scrub 2024-05-27T00:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.498+0000 7f6873600640 1 -- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68480678c0 msgr2=0x7f6848069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.498+0000 7f6873600640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68480678c0 0x7f6848069d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f685c0ae000 tx=0x7f685c008040 comp rx=0 tx=0).stop 2024-05-27T00:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.502+0000 7f6873600640 1 -- 172.21.15.98:0/3237645203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c0c2b10 msgr2=0x7f686c1f6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.502+0000 7f6873600640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c0c2b10 0x7f686c1f6680 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f686400c6c0 tx=0x7f686400cb80 comp rx=0 tx=0).stop 2024-05-27T00:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.502+0000 7f6873600640 1 -- 172.21.15.98:0/3237645203 shutdown_connections 2024-05-27T00:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.502+0000 7f6873600640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c0c2b10 0x7f686c1f6680 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.502+0000 7f6873600640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f686c152770 0x7f686c14e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.502+0000 7f6873600640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68480678c0 0x7f6848069d80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.502+0000 7f6873600640 1 --2- 172.21.15.98:0/3237645203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f686c0c84e0 0x7f686c14b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.502+0000 7f6873600640 1 -- 172.21.15.98:0/3237645203 >> 172.21.15.98:0/3237645203 conn(0x7f686c0cc200 msgr2=0x7f686c071700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.502+0000 7f6873600640 1 -- 172.21.15.98:0/3237645203 shutdown_connections 2024-05-27T00:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.502+0000 7f6873600640 1 -- 172.21.15.98:0/3237645203 wait complete. 2024-05-27T00:33:43.521 INFO:tasks.ceph:Scrubbing osd.1 2024-05-27T00:33:43.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-27T00:33:43.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.658+0000 7f7b91600640 1 Processor -- start 2024-05-27T00:33:43.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.658+0000 7f7b91600640 1 -- start start 2024-05-27T00:33:43.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.658+0000 7f7b91600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b8c1516b0 0x7f7b8c0c1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b91600640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7b8c0b1e10 0x7f7b8c06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b91600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7b8c0c2b10 0x7f7b8c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b91600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7b8c0c8420 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b91600640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7b8c0c1df0 con 0x7f7b8c0b1e10 2024-05-27T00:33:43.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b91600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7b8c0c3000 con 0x7f7b8c1516b0 2024-05-27T00:33:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b8b400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7b8c0c2b10 0x7f7b8c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b8b400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7b8c0c2b10 0x7f7b8c06ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55164/0 (socket says 172.21.15.98:55164) 2024-05-27T00:33:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b8b400640 1 -- 172.21.15.98:0/989554156 learned_addr learned my addr 172.21.15.98:0/989554156 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b82000640 1 --2- 172.21.15.98:0/989554156 >> v2:172.21.15.204:3300/0 conn(0x7f7b8c0b1e10 0x7f7b8c06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b8aa00640 1 --2- 172.21.15.98:0/989554156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b8c1516b0 0x7f7b8c0c1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b8b400640 1 -- 172.21.15.98:0/989554156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b8c1516b0 msgr2=0x7f7b8c0c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b8b400640 1 --2- 172.21.15.98:0/989554156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b8c1516b0 0x7f7b8c0c1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b8b400640 1 -- 172.21.15.98:0/989554156 >> v2:172.21.15.204:3300/0 conn(0x7f7b8c0b1e10 msgr2=0x7f7b8c06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:43.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b8b400640 1 --2- 172.21.15.98:0/989554156 >> v2:172.21.15.204:3300/0 conn(0x7f7b8c0b1e10 0x7f7b8c06e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b8b400640 1 -- 172.21.15.98:0/989554156 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c0c7e60 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.662+0000 7f7b8b400640 1 --2- 172.21.15.98:0/989554156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b8c0c2b10 0x7f7b8c06ecc0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f7b8400d150 tx=0x7f7b8401ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d5f8b0161fb4de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b8a000640 1 -- 172.21.15.98:0/989554156 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8400b2d0 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b8a000640 1 -- 172.21.15.98:0/989554156 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b8400b470 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b8a000640 1 -- 172.21.15.98:0/989554156 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b840a7b90 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 -- 172.21.15.98:0/989554156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b8c0c2b10 msgr2=0x7f7b8c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 --2- 172.21.15.98:0/989554156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b8c0c2b10 0x7f7b8c06ecc0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f7b8400d150 tx=0x7f7b8401ebf0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 -- 172.21.15.98:0/989554156 shutdown_connections 2024-05-27T00:33:43.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 --2- 172.21.15.98:0/989554156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b8c0c2b10 0x7f7b8c06ecc0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 --2- 172.21.15.98:0/989554156 >> v2:172.21.15.204:3300/0 conn(0x7f7b8c0b1e10 0x7f7b8c06e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 --2- 172.21.15.98:0/989554156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b8c1516b0 0x7f7b8c0c1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 -- 172.21.15.98:0/989554156 >> 172.21.15.98:0/989554156 conn(0x7f7b8c0cbf00 msgr2=0x7f7b8c0cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:43.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 -- 172.21.15.98:0/989554156 shutdown_connections 2024-05-27T00:33:43.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 -- 172.21.15.98:0/989554156 wait complete. 2024-05-27T00:33:43.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 Processor -- start 2024-05-27T00:33:43.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 -- start start 2024-05-27T00:33:43.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b8c0b1e10 0x7f7b8c144f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b8c0c2b10 0x7f7b8c1f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b8c1516b0 0x7f7b8c159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c073030 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c072eb0 con 0x7f7b8c0b1e10 2024-05-27T00:33:43.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b91600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7b8c151a90 con 0x7f7b8c1516b0 2024-05-27T00:33:43.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b82000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b8c0c2b10 0x7f7b8c1f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b82000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b8c0c2b10 0x7f7b8c1f3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55166/0 (socket says 172.21.15.98:55166) 2024-05-27T00:33:43.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b82000640 1 -- 172.21.15.98:0/634237108 learned_addr learned my addr 172.21.15.98:0/634237108 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:43.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b8b400640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b8c1516b0 0x7f7b8c159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.666+0000 7f7b8aa00640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b8c0b1e10 0x7f7b8c144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b82000640 1 -- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b8c1516b0 msgr2=0x7f7b8c159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b82000640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7b8c1516b0 0x7f7b8c159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b82000640 1 -- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b8c0b1e10 msgr2=0x7f7b8c144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b82000640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b8c0b1e10 0x7f7b8c144f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b82000640 1 -- 172.21.15.98:0/634237108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c0c5950 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b82000640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b8c0c2b10 0x7f7b8c1f3f30 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400e870 tx=0x7f7b7400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b83400640 1 -- 172.21.15.98:0/634237108 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74017070 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b83400640 1 -- 172.21.15.98:0/634237108 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7400c940 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8c0c5c30 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b83400640 1 -- 172.21.15.98:0/634237108 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7400cd30 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b8c0c61f0 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b8c075cb0 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b83400640 1 -- 172.21.15.98:0/634237108 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7b7401a070 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b83400640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b680678c0 0x7f7b68069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b83400640 1 -- 172.21.15.98:0/634237108 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f7b74086070 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b83400640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] conn(0x7f7b680704c0 0x7f7b68072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b83400640 1 -- 172.21.15.98:0/634237108 --> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b7400be50 con 0x7f7b680704c0 2024-05-27T00:33:43.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b83400640 1 -- 172.21.15.98:0/634237108 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f7b7408d050 con 0x7f7b8c0c2b10 2024-05-27T00:33:43.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b8aa00640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b680678c0 0x7f7b68069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.670+0000 7f7b8b400640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] conn(0x7f7b680704c0 0x7f7b68072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.674+0000 7f7b8b400640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] conn(0x7f7b680704c0 0x7f7b68072940 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.674+0000 7f7b8aa00640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b680678c0 0x7f7b68069d80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009ae0 tx=0x7f7b7c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.674+0000 7f7b83400640 1 -- 172.21.15.98:0/634237108 <== osd.1 v2:172.21.15.98:6808/1799281170 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b7400be50 con 0x7f7b680704c0 2024-05-27T00:33:43.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 --> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7b8c147ce0 con 0x7f7b680704c0 2024-05-27T00:33:43.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b83400640 1 -- 172.21.15.98:0/634237108 <== osd.1 v2:172.21.15.98:6808/1799281170 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7b8c147ce0 con 0x7f7b680704c0 2024-05-27T00:33:43.688 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-27T00:33:43.688 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-27T00:33:43.688 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-27T00:33:43.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 >> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] conn(0x7f7b680704c0 msgr2=0x7f7b68072940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.98:6808/1799281170,v1:172.21.15.98:6809/1799281170] conn(0x7f7b680704c0 0x7f7b68072940 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b680678c0 msgr2=0x7f7b68069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7b680678c0 0x7f7b68069d80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009ae0 tx=0x7f7b7c009190 comp rx=0 tx=0).stop 2024-05-27T00:33:43.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b8c0c2b10 msgr2=0x7f7b8c1f3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 --2- 172.21.15.98:0/634237108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b8c0c2b10 0x7f7b8c1f3f30 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400e870 tx=0x7f7b7400ec70 comp rx=0 tx=0).stop 2024-05-27T00:33:43.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b8b400640 1 -- 172.21.15.98:0/634237108 reap_dead start 2024-05-27T00:33:43.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 shutdown_connections 2024-05-27T00:33:43.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 >> 172.21.15.98:0/634237108 conn(0x7f7b8c0cbf00 msgr2=0x7f7b8c14eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:43.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 shutdown_connections 2024-05-27T00:33:43.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.686+0000 7f7b91600640 1 -- 172.21.15.98:0/634237108 wait complete. 2024-05-27T00:33:43.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-27T00:33:43.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.842+0000 7f6bedc00640 1 Processor -- start 2024-05-27T00:33:43.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.842+0000 7f6bedc00640 1 -- start start 2024-05-27T00:33:43.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.842+0000 7f6bedc00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6be8154a60 0x7f6be815a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.842+0000 7f6bedc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be80c7040 0x7f6be80c7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.842+0000 7f6bedc00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6be815aee0 con 0x7f6be815e820 2024-05-27T00:33:43.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.842+0000 7f6bedc00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6be815b1c0 con 0x7f6be8154a60 2024-05-27T00:33:43.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.842+0000 7f6bedc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6be815b4a0 con 0x7f6be80c7040 2024-05-27T00:33:43.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.842+0000 7f6be6a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6be8154a60 0x7f6be815a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.842+0000 7f6be6000640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6be815e820 0x7f6be8072750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37266/0 (socket says 172.21.15.98:37266) 2024-05-27T00:33:43.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.842+0000 7f6be6000640 1 -- 172.21.15.98:0/1493249095 learned_addr learned my addr 172.21.15.98:0/1493249095 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:43.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.846+0000 7f6be7400640 1 --2- 172.21.15.98:0/1493249095 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be80c7040 0x7f6be80c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.846+0000 7f6be6a00640 1 -- 172.21.15.98:0/1493249095 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be80c7040 msgr2=0x7f6be80c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:43.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.846+0000 7f6be6a00640 1 --2- 172.21.15.98:0/1493249095 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be80c7040 0x7f6be80c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.846+0000 7f6be6a00640 1 -- 172.21.15.98:0/1493249095 >> v1:172.21.15.98:6789/0 conn(0x7f6be815e820 legacy=0x7f6be8072750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.846+0000 7f6be5600640 1 -- 172.21.15.98:0/1493249095 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2186269229 0 0) 0x7f6be815aee0 con 0x7f6be815e820 2024-05-27T00:33:43.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.846+0000 7f6be6a00640 1 -- 172.21.15.98:0/1493249095 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be815b780 con 0x7f6be8154a60 2024-05-27T00:33:43.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.846+0000 7f6be6a00640 1 --2- 172.21.15.98:0/1493249095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be8154a60 0x7f6be815a820 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8009d90 tx=0x7f6bd801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa4644aa4da829 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.846+0000 7f6be5600640 1 -- 172.21.15.98:0/1493249095 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd80b2070 con 0x7f6be8154a60 2024-05-27T00:33:43.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.846+0000 7f6be5600640 1 -- 172.21.15.98:0/1493249095 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd80a5920 con 0x7f6be8154a60 2024-05-27T00:33:43.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.846+0000 7f6be5600640 1 -- 172.21.15.98:0/1493249095 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd80a5cf0 con 0x7f6be8154a60 2024-05-27T00:33:43.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.850+0000 7f6bedc00640 1 -- 172.21.15.98:0/1493249095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be8154a60 msgr2=0x7f6be815a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.850+0000 7f6bedc00640 1 --2- 172.21.15.98:0/1493249095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be8154a60 0x7f6be815a820 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8009d90 tx=0x7f6bd801c970 comp rx=0 tx=0).stop 2024-05-27T00:33:43.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.850+0000 7f6bedc00640 1 -- 172.21.15.98:0/1493249095 shutdown_connections 2024-05-27T00:33:43.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.850+0000 7f6bedc00640 1 --2- 172.21.15.98:0/1493249095 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be80c7040 0x7f6be80c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.850+0000 7f6bedc00640 1 --2- 172.21.15.98:0/1493249095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be8154a60 0x7f6be815a820 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.850+0000 7f6bedc00640 1 -- 172.21.15.98:0/1493249095 >> 172.21.15.98:0/1493249095 conn(0x7f6be806b470 msgr2=0x7f6be806b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:43.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.850+0000 7f6bedc00640 1 -- 172.21.15.98:0/1493249095 shutdown_connections 2024-05-27T00:33:43.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.850+0000 7f6bedc00640 1 -- 172.21.15.98:0/1493249095 wait complete. 2024-05-27T00:33:43.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.850+0000 7f6bedc00640 1 Processor -- start 2024-05-27T00:33:43.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.850+0000 7f6bedc00640 1 -- start start 2024-05-27T00:33:43.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bedc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be80c7040 0x7f6be8144df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bedc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be8154a60 0x7f6be813c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bedc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6be815e820 0x7f6be813c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bedc00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6be80c8150 con 0x7f6be80c7040 2024-05-27T00:33:43.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bedc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6be80c7fd0 con 0x7f6be815e820 2024-05-27T00:33:43.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bedc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6be815ec00 con 0x7f6be8154a60 2024-05-27T00:33:43.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be80c7040 0x7f6be8144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be80c7040 0x7f6be8144df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55182/0 (socket says 172.21.15.98:55182) 2024-05-27T00:33:43.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6a00640 1 -- 172.21.15.98:0/1964450196 learned_addr learned my addr 172.21.15.98:0/1964450196 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:43.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be7400640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6be815e820 0x7f6be813c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6000640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be8154a60 0x7f6be813c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6a00640 1 -- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be8154a60 msgr2=0x7f6be813c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6a00640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be8154a60 0x7f6be813c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6a00640 1 -- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6be815e820 msgr2=0x7f6be813c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:43.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6a00640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6be815e820 0x7f6be813c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:43.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6a00640 1 -- 172.21.15.98:0/1964450196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be8151f40 con 0x7f6be80c7040 2024-05-27T00:33:43.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6000640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be8154a60 0x7f6be813c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:43.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be7400640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6be815e820 0x7f6be813c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:43.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6be6a00640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be80c7040 0x7f6be8144df0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8009d90 tx=0x7f6bd8019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bdea00640 1 -- 172.21.15.98:0/1964450196 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd80b2040 con 0x7f6be80c7040 2024-05-27T00:33:43.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bdea00640 1 -- 172.21.15.98:0/1964450196 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd80bf070 con 0x7f6be80c7040 2024-05-27T00:33:43.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bdea00640 1 -- 172.21.15.98:0/1964450196 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd80b8870 con 0x7f6be80c7040 2024-05-27T00:33:43.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bedc00640 1 -- 172.21.15.98:0/1964450196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be8152220 con 0x7f6be80c7040 2024-05-27T00:33:43.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bedc00640 1 -- 172.21.15.98:0/1964450196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be8158970 con 0x7f6be80c7040 2024-05-27T00:33:43.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.854+0000 7f6bedc00640 1 -- 172.21.15.98:0/1964450196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be8075cb0 con 0x7f6be80c7040 2024-05-27T00:33:43.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.858+0000 7f6bdea00640 1 -- 172.21.15.98:0/1964450196 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6bd80b8a10 con 0x7f6be80c7040 2024-05-27T00:33:43.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.858+0000 7f6bdea00640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6bc80678c0 0x7f6bc8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:43.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.858+0000 7f6bdea00640 1 -- 172.21.15.98:0/1964450196 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f6bd8121340 con 0x7f6be80c7040 2024-05-27T00:33:43.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.858+0000 7f6be6000640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6bc80678c0 0x7f6bc8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:43.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.858+0000 7f6be6000640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6bc80678c0 0x7f6bc8069d80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6be81541d0 tx=0x7f6bd4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:43.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:43.862+0000 7f6bdea00640 1 -- 172.21.15.98:0/1964450196 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f6bd8016560 con 0x7f6be80c7040 2024-05-27T00:33:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.038+0000 7f6bedc00640 1 -- 172.21.15.98:0/1964450196 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f6be80c7af0 con 0x7f6bc80678c0 2024-05-27T00:33:44.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bdea00640 1 -- 172.21.15.98:0/1964450196 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6be80c7af0 con 0x7f6bc80678c0 2024-05-27T00:33:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 1 to deep-scrub 2024-05-27T00:33:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 -- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6bc80678c0 msgr2=0x7f6bc8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6bc80678c0 0x7f6bc8069d80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6be81541d0 tx=0x7f6bd4009040 comp rx=0 tx=0).stop 2024-05-27T00:33:44.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 -- 172.21.15.98:0/1964450196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be80c7040 msgr2=0x7f6be8144df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be80c7040 0x7f6be8144df0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8009d90 tx=0x7f6bd8019710 comp rx=0 tx=0).stop 2024-05-27T00:33:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 -- 172.21.15.98:0/1964450196 shutdown_connections 2024-05-27T00:33:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f6bc80678c0 0x7f6bc8069d80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6be815e820 0x7f6be813c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6be8154a60 0x7f6be813c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 --2- 172.21.15.98:0/1964450196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6be80c7040 0x7f6be8144df0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 -- 172.21.15.98:0/1964450196 >> 172.21.15.98:0/1964450196 conn(0x7f6be806b470 msgr2=0x7f6be80c6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:44.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 -- 172.21.15.98:0/1964450196 shutdown_connections 2024-05-27T00:33:44.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.042+0000 7f6bedc00640 1 -- 172.21.15.98:0/1964450196 wait complete. 2024-05-27T00:33:44.063 INFO:tasks.ceph:Scrubbing osd.2 2024-05-27T00:33:44.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-27T00:33:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe46b400640 1 Processor -- start 2024-05-27T00:33:44.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe46b400640 1 -- start start 2024-05-27T00:33:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe46b400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fe464152990 0x7fe46406e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe46b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe46406ecc0 0x7fe4640c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe46b400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe46415bc40 con 0x7fe4640b1c20 2024-05-27T00:33:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe46b400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fe46415c850 con 0x7fe464152990 2024-05-27T00:33:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe46b400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe46415bf20 con 0x7fe46406ecc0 2024-05-27T00:33:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe469400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe4640b1c20 0x7fe464158640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37272/0 (socket says 172.21.15.98:37272) 2024-05-27T00:33:44.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe469400640 1 -- 172.21.15.98:0/2340412965 learned_addr learned my addr 172.21.15.98:0/2340412965 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe468a00640 1 --2- 172.21.15.98:0/2340412965 >> v2:172.21.15.204:3300/0 conn(0x7fe464152990 0x7fe46406e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.202+0000 7fe463e00640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe46406ecc0 0x7fe4640c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1407127314 0 0) 0x7fe46415bc40 con 0x7fe4640b1c20 2024-05-27T00:33:44.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe448003530 con 0x7fe4640b1c20 2024-05-27T00:33:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 752609001 0 0) 0x7fe448003530 con 0x7fe4640b1c20 2024-05-27T00:33:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe46406ecc0 msgr2=0x7fe4640c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe46406ecc0 0x7fe4640c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 >> v2:172.21.15.204:3300/0 conn(0x7fe464152990 msgr2=0x7fe46406e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 --2- 172.21.15.98:0/2340412965 >> v2:172.21.15.204:3300/0 conn(0x7fe464152990 0x7fe46406e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46415c200 con 0x7fe4640b1c20 2024-05-27T00:33:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fe45c001a00 con 0x7fe4640b1c20 2024-05-27T00:33:44.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 >> v1:172.21.15.98:6789/0 conn(0x7fe4640b1c20 legacy=0x7fe464158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4480039c0 0x7fe448003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe46406ecc0 0x7fe448004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 0x7fe4480051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe46415c850 con 0x7fe448004d50 2024-05-27T00:33:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe46415bf20 con 0x7fe4480039c0 2024-05-27T00:33:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe44800b9d0 con 0x7fe46406ecc0 2024-05-27T00:33:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe469400640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 0x7fe4480051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe468a00640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4480039c0 0x7fe448003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe463e00640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe46406ecc0 0x7fe448004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe469400640 1 -- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe46406ecc0 msgr2=0x7fe448004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe469400640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe46406ecc0 0x7fe448004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe469400640 1 -- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4480039c0 msgr2=0x7fe448003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe469400640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4480039c0 0x7fe448003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.206+0000 7fe469400640 1 -- 172.21.15.98:0/2340412965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe448003530 con 0x7fe448004d50 2024-05-27T00:33:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe469400640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 0x7fe4480051d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe45c00a870 tx=0x7fe45c01d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a28941d738dc5c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c00a4c0 con 0x7fe448004d50 2024-05-27T00:33:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe45c0072d0 con 0x7fe448004d50 2024-05-27T00:33:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe463400640 1 -- 172.21.15.98:0/2340412965 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c0076c0 con 0x7fe448004d50 2024-05-27T00:33:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 -- 172.21.15.98:0/2340412965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 msgr2=0x7fe4480051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 --2- 172.21.15.98:0/2340412965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 0x7fe4480051d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe45c00a870 tx=0x7fe45c01d3a0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe469400640 1 -- 172.21.15.98:0/2340412965 reap_dead start 2024-05-27T00:33:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 -- 172.21.15.98:0/2340412965 shutdown_connections 2024-05-27T00:33:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 -- 172.21.15.98:0/2340412965 >> 172.21.15.98:0/2340412965 conn(0x7fe4640cbf00 msgr2=0x7fe4640cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 -- 172.21.15.98:0/2340412965 shutdown_connections 2024-05-27T00:33:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 -- 172.21.15.98:0/2340412965 wait complete. 2024-05-27T00:33:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 Processor -- start 2024-05-27T00:33:44.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 -- start start 2024-05-27T00:33:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 0x7fe46414b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe46415a280 0x7fe464157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe464157960 0x7fe4641f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe464158ed0 con 0x7fe448004d50 2024-05-27T00:33:44.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe464158d50 con 0x7fe46415a280 2024-05-27T00:33:44.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe46b400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe4640c2f10 con 0x7fe464157960 2024-05-27T00:33:44.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe468a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 0x7fe46414b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe468a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 0x7fe46414b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55202/0 (socket says 172.21.15.98:55202) 2024-05-27T00:33:44.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.210+0000 7fe468a00640 1 -- 172.21.15.98:0/3051561717 learned_addr learned my addr 172.21.15.98:0/3051561717 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:44.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe463e00640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe46415a280 0x7fe464157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe469400640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe464157960 0x7fe4641f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe468a00640 1 -- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe464157960 msgr2=0x7fe4641f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe468a00640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe464157960 0x7fe4641f3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe468a00640 1 -- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe46415a280 msgr2=0x7fe464157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe468a00640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe46415a280 0x7fe464157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe468a00640 1 -- 172.21.15.98:0/3051561717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe464072280 con 0x7fe448004d50 2024-05-27T00:33:44.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe468a00640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 0x7fe46414b770 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe4540119d0 tx=0x7fe454011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe460c00640 1 -- 172.21.15.98:0/3051561717 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45400fb50 con 0x7fe448004d50 2024-05-27T00:33:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe460c00640 1 -- 172.21.15.98:0/3051561717 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe45400fcf0 con 0x7fe448004d50 2024-05-27T00:33:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4640c0f20 con 0x7fe448004d50 2024-05-27T00:33:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe460c00640 1 -- 172.21.15.98:0/3051561717 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45400fea0 con 0x7fe448004d50 2024-05-27T00:33:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4641580d0 con 0x7fe448004d50 2024-05-27T00:33:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe4641583b0 con 0x7fe448004d50 2024-05-27T00:33:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe460c00640 1 -- 172.21.15.98:0/3051561717 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe454012040 con 0x7fe448004d50 2024-05-27T00:33:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe460c00640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4400678f0 0x7fe440069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe460c00640 1 -- 172.21.15.98:0/3051561717 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fe4540868c0 con 0x7fe448004d50 2024-05-27T00:33:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe460c00640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] conn(0x7fe4400704f0 0x7fe440072970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe460c00640 1 -- 172.21.15.98:0/3051561717 --> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe454025ec0 con 0x7fe4400704f0 2024-05-27T00:33:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe460c00640 1 -- 172.21.15.98:0/3051561717 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fe45408b2a0 con 0x7fe448004d50 2024-05-27T00:33:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.214+0000 7fe463e00640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4400678f0 0x7fe440069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.218+0000 7fe469400640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] conn(0x7fe4400704f0 0x7fe440072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.218+0000 7fe469400640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] conn(0x7fe4400704f0 0x7fe440072970 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.218+0000 7fe463e00640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4400678f0 0x7fe440069db0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe464158580 tx=0x7fe44c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.218+0000 7fe460c00640 1 -- 172.21.15.98:0/3051561717 <== osd.2 v2:172.21.15.98:6800/2594065140 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe454025ec0 con 0x7fe4400704f0 2024-05-27T00:33:44.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 --> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe464075cb0 con 0x7fe4400704f0 2024-05-27T00:33:44.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe460c00640 1 -- 172.21.15.98:0/3051561717 <== osd.2 v2:172.21.15.98:6800/2594065140 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe464075cb0 con 0x7fe4400704f0 2024-05-27T00:33:44.232 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-27T00:33:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-27T00:33:44.232 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-27T00:33:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 >> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] conn(0x7fe4400704f0 msgr2=0x7fe440072970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.98:6800/2594065140,v1:172.21.15.98:6801/2594065140] conn(0x7fe4400704f0 0x7fe440072970 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4400678f0 msgr2=0x7fe440069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe4400678f0 0x7fe440069db0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe464158580 tx=0x7fe44c008040 comp rx=0 tx=0).stop 2024-05-27T00:33:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 msgr2=0x7fe46414b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 --2- 172.21.15.98:0/3051561717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe448004d50 0x7fe46414b770 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe4540119d0 tx=0x7fe454011e90 comp rx=0 tx=0).stop 2024-05-27T00:33:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe469400640 1 -- 172.21.15.98:0/3051561717 reap_dead start 2024-05-27T00:33:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 shutdown_connections 2024-05-27T00:33:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 >> 172.21.15.98:0/3051561717 conn(0x7fe4640cbf00 msgr2=0x7fe464150430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 shutdown_connections 2024-05-27T00:33:44.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.230+0000 7fe46b400640 1 -- 172.21.15.98:0/3051561717 wait complete. 2024-05-27T00:33:44.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-27T00:33:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb4800640 1 Processor -- start 2024-05-27T00:33:44.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb4800640 1 -- start start 2024-05-27T00:33:44.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb4800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fafac1519d0 0x7fafac0c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb4800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafac0c84a0 con 0x7fafac0c11a0 2024-05-27T00:33:44.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb4800640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafac070fd0 con 0x7fafac0c15f0 2024-05-27T00:33:44.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb4800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fafac070d80 con 0x7fafac1519d0 2024-05-27T00:33:44.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb1400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fafac0c11a0 0x7fafac06ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37288/0 (socket says 172.21.15.98:37288) 2024-05-27T00:33:44.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb1400640 1 -- 172.21.15.98:0/2474977062 learned_addr learned my addr 172.21.15.98:0/2474977062 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:44.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb1e00640 1 --2- 172.21.15.98:0/2474977062 >> v2:172.21.15.204:3300/0 conn(0x7fafac1519d0 0x7fafac0c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1909050112 0 0) 0x7fafac0c84a0 con 0x7fafac0c11a0 2024-05-27T00:33:44.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.386+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf94003530 con 0x7fafac0c11a0 2024-05-27T00:33:44.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862817177 0 0) 0x7fafac070fd0 con 0x7fafac0c15f0 2024-05-27T00:33:44.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafac0c84a0 con 0x7fafac0c15f0 2024-05-27T00:33:44.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 786965558 0 0) 0x7faf94003530 con 0x7fafac0c11a0 2024-05-27T00:33:44.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 >> v2:172.21.15.204:3300/0 conn(0x7fafac1519d0 msgr2=0x7fafac0c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:44.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 --2- 172.21.15.98:0/2474977062 >> v2:172.21.15.204:3300/0 conn(0x7fafac1519d0 0x7fafac0c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 >> v1:172.21.15.204:6789/0 conn(0x7fafac0c15f0 legacy=0x7fafac1578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafac0712b0 con 0x7fafac0c11a0 2024-05-27T00:33:44.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7faf9c001680 con 0x7fafac0c11a0 2024-05-27T00:33:44.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 >> v1:172.21.15.98:6789/0 conn(0x7fafac0c11a0 legacy=0x7fafac06ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf94003ba0 0x7faf94003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf94004810 0x7faf94004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf940051d0 0x7faf9400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafac070d80 con 0x7faf94003ba0 2024-05-27T00:33:44.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faf9400e2b0 con 0x7faf940051d0 2024-05-27T00:33:44.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faf9400e4b0 con 0x7faf94004810 2024-05-27T00:33:44.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb1e00640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf94003ba0 0x7faf94003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb2800640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf940051d0 0x7faf9400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb1400640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf94004810 0x7faf94004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb1e00640 1 -- 172.21.15.98:0/2474977062 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf94004810 msgr2=0x7faf94004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:44.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb1e00640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf94004810 0x7faf94004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb1e00640 1 -- 172.21.15.98:0/2474977062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf940051d0 msgr2=0x7faf9400dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:44.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb1e00640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf940051d0 0x7faf9400dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb1e00640 1 -- 172.21.15.98:0/2474977062 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf94003530 con 0x7faf94003ba0 2024-05-27T00:33:44.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb2800640 1 -- 172.21.15.98:0/2474977062 reap_dead start 2024-05-27T00:33:44.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb1400640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf94004810 0x7faf94004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:33:44.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb1e00640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf94003ba0 0x7faf94003fa0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009ae0 tx=0x7fafa001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d163b2110af3932f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.390+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa00b2070 con 0x7faf94003ba0 2024-05-27T00:33:44.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafa00a5660 con 0x7faf94003ba0 2024-05-27T00:33:44.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb0a00640 1 -- 172.21.15.98:0/2474977062 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa00096b0 con 0x7faf94003ba0 2024-05-27T00:33:44.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 -- 172.21.15.98:0/2474977062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf94003ba0 msgr2=0x7faf94003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf94003ba0 0x7faf94003fa0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009ae0 tx=0x7fafa001ca70 comp rx=0 tx=0).stop 2024-05-27T00:33:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 -- 172.21.15.98:0/2474977062 shutdown_connections 2024-05-27T00:33:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 --2- 172.21.15.98:0/2474977062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf94003ba0 0x7faf94003fa0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 -- 172.21.15.98:0/2474977062 >> 172.21.15.98:0/2474977062 conn(0x7fafac0cc200 msgr2=0x7fafac0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 -- 172.21.15.98:0/2474977062 shutdown_connections 2024-05-27T00:33:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 -- 172.21.15.98:0/2474977062 wait complete. 2024-05-27T00:33:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 Processor -- start 2024-05-27T00:33:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 -- start start 2024-05-27T00:33:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf94003ba0 0x7fafac1507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafac155cb0 0x7fafac150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c4650 0x7fafac0c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafac157c20 con 0x7fafac155cb0 2024-05-27T00:33:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fafac157aa0 con 0x7faf94003ba0 2024-05-27T00:33:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb4800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fafac157da0 con 0x7fafac0c4650 2024-05-27T00:33:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb1400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafac155cb0 0x7fafac150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb1400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafac155cb0 0x7fafac150d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55212/0 (socket says 172.21.15.98:55212) 2024-05-27T00:33:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb1400640 1 -- 172.21.15.98:0/623978994 learned_addr learned my addr 172.21.15.98:0/623978994 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb1e00640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf94003ba0 0x7fafac1507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb2800640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c4650 0x7fafac0c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb1400640 1 -- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c4650 msgr2=0x7fafac0c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb1400640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c4650 0x7fafac0c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb1400640 1 -- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf94003ba0 msgr2=0x7fafac1507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb1400640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf94003ba0 0x7fafac1507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.394+0000 7fafb1400640 1 -- 172.21.15.98:0/623978994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafac0c5da0 con 0x7fafac155cb0 2024-05-27T00:33:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafb1400640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafac155cb0 0x7fafac150d30 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7faf9c007fd0 tx=0x7faf9c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafa6000640 1 -- 172.21.15.98:0/623978994 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c019070 con 0x7fafac155cb0 2024-05-27T00:33:44.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafa6000640 1 -- 172.21.15.98:0/623978994 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf9c0072d0 con 0x7fafac155cb0 2024-05-27T00:33:44.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafa6000640 1 -- 172.21.15.98:0/623978994 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c007680 con 0x7fafac155cb0 2024-05-27T00:33:44.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafb4800640 1 -- 172.21.15.98:0/623978994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafac1f35f0 con 0x7fafac155cb0 2024-05-27T00:33:44.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafb4800640 1 -- 172.21.15.98:0/623978994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafac1f3bb0 con 0x7fafac155cb0 2024-05-27T00:33:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafb4800640 1 -- 172.21.15.98:0/623978994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafac075cb0 con 0x7fafac155cb0 2024-05-27T00:33:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafa6000640 1 -- 172.21.15.98:0/623978994 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf9c0047f0 con 0x7fafac155cb0 2024-05-27T00:33:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafa6000640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf900678f0 0x7faf90069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafa6000640 1 -- 172.21.15.98:0/623978994 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7faf9c0859a0 con 0x7fafac155cb0 2024-05-27T00:33:44.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.398+0000 7fafb1e00640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf900678f0 0x7faf90069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.402+0000 7fafb1e00640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf900678f0 0x7faf90069db0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fafa00af000 tx=0x7fafa0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.402+0000 7fafa6000640 1 -- 172.21.15.98:0/623978994 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7faf9c056050 con 0x7fafac155cb0 2024-05-27T00:33:44.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 -- 172.21.15.98:0/623978994 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fafac156440 con 0x7faf900678f0 2024-05-27T00:33:44.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafa6000640 1 -- 172.21.15.98:0/623978994 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fafac156440 con 0x7faf900678f0 2024-05-27T00:33:44.562 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 2 to deep-scrub 2024-05-27T00:33:44.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 -- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf900678f0 msgr2=0x7faf90069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf900678f0 0x7faf90069db0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fafa00af000 tx=0x7fafa0008040 comp rx=0 tx=0).stop 2024-05-27T00:33:44.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 -- 172.21.15.98:0/623978994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafac155cb0 msgr2=0x7fafac150d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafac155cb0 0x7fafac150d30 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7faf9c007fd0 tx=0x7faf9c00dc00 comp rx=0 tx=0).stop 2024-05-27T00:33:44.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 -- 172.21.15.98:0/623978994 shutdown_connections 2024-05-27T00:33:44.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c4650 0x7fafac0c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafac155cb0 0x7fafac150d30 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faf900678f0 0x7faf90069db0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 --2- 172.21.15.98:0/623978994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf94003ba0 0x7fafac1507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 -- 172.21.15.98:0/623978994 >> 172.21.15.98:0/623978994 conn(0x7fafac0cc200 msgr2=0x7fafac0c6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:44.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 -- 172.21.15.98:0/623978994 shutdown_connections 2024-05-27T00:33:44.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.562+0000 7fafb4800640 1 -- 172.21.15.98:0/623978994 wait complete. 2024-05-27T00:33:44.584 INFO:tasks.ceph:Scrubbing osd.3 2024-05-27T00:33:44.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-27T00:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.742+0000 7f27ef800640 1 Processor -- start 2024-05-27T00:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.742+0000 7f27ef800640 1 -- start start 2024-05-27T00:33:44.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27ef800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27e80b1e10 0x7f27e806e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27ef800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f27e80c2b10 0x7f27e806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27ef800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e80c8420 con 0x7f27e81516b0 2024-05-27T00:33:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27ef800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f27e80c1df0 con 0x7f27e80b1e10 2024-05-27T00:33:44.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27ef800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f27e80c3000 con 0x7f27e80c2b10 2024-05-27T00:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27ece00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f27e81516b0 0x7f27e80c1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37294/0 (socket says 172.21.15.98:37294) 2024-05-27T00:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27ece00640 1 -- 172.21.15.98:0/2423314217 learned_addr learned my addr 172.21.15.98:0/2423314217 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27dfe00640 1 --2- 172.21.15.98:0/2423314217 >> v2:172.21.15.98:3300/0 conn(0x7f27e80b1e10 0x7f27e806e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27ed800640 1 --2- 172.21.15.98:0/2423314217 >> v2:172.21.15.204:3300/0 conn(0x7f27e80c2b10 0x7f27e806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27e7e00640 1 -- 172.21.15.98:0/2423314217 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2791539319 0 0) 0x7f27e80c8420 con 0x7f27e81516b0 2024-05-27T00:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27e7e00640 1 -- 172.21.15.98:0/2423314217 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27cc003530 con 0x7f27e81516b0 2024-05-27T00:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27dfe00640 1 -- 172.21.15.98:0/2423314217 >> v2:172.21.15.204:3300/0 conn(0x7f27e80c2b10 msgr2=0x7f27e806ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27dfe00640 1 --2- 172.21.15.98:0/2423314217 >> v2:172.21.15.204:3300/0 conn(0x7f27e80c2b10 0x7f27e806ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27dfe00640 1 -- 172.21.15.98:0/2423314217 >> v1:172.21.15.98:6789/0 conn(0x7f27e81516b0 legacy=0x7f27e80c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27dfe00640 1 -- 172.21.15.98:0/2423314217 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e80c7e60 con 0x7f27e80b1e10 2024-05-27T00:33:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.746+0000 7f27ed800640 1 --2- 172.21.15.98:0/2423314217 >> v2:172.21.15.204:3300/0 conn(0x7f27e80c2b10 0x7f27e806ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27dfe00640 1 --2- 172.21.15.98:0/2423314217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27e80b1e10 0x7f27e806e780 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f27e8158590 tx=0x7f27d001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3a5d5c5ca80f11 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27e7e00640 1 -- 172.21.15.98:0/2423314217 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00b0070 con 0x7f27e80b1e10 2024-05-27T00:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27e7e00640 1 -- 172.21.15.98:0/2423314217 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27d00ab600 con 0x7f27e80b1e10 2024-05-27T00:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27e7e00640 1 -- 172.21.15.98:0/2423314217 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00ab9d0 con 0x7f27e80b1e10 2024-05-27T00:33:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 -- 172.21.15.98:0/2423314217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27e80b1e10 msgr2=0x7f27e806e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 --2- 172.21.15.98:0/2423314217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27e80b1e10 0x7f27e806e780 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f27e8158590 tx=0x7f27d001c480 comp rx=0 tx=0).stop 2024-05-27T00:33:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 -- 172.21.15.98:0/2423314217 shutdown_connections 2024-05-27T00:33:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 --2- 172.21.15.98:0/2423314217 >> v2:172.21.15.204:3300/0 conn(0x7f27e80c2b10 0x7f27e806ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 --2- 172.21.15.98:0/2423314217 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27e80b1e10 0x7f27e806e780 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 -- 172.21.15.98:0/2423314217 >> 172.21.15.98:0/2423314217 conn(0x7f27e80cbf00 msgr2=0x7f27e80cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:44.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 -- 172.21.15.98:0/2423314217 shutdown_connections 2024-05-27T00:33:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 -- 172.21.15.98:0/2423314217 wait complete. 2024-05-27T00:33:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 Processor -- start 2024-05-27T00:33:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 -- start start 2024-05-27T00:33:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27e80b1e10 0x7f27e8163fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27e80c2b10 0x7f27e81552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27e81516b0 0x7f27e8155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27e8151c10 con 0x7f27e81516b0 2024-05-27T00:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f27e8151a90 con 0x7f27e80b1e10 2024-05-27T00:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ef800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f27e8162530 con 0x7f27e80c2b10 2024-05-27T00:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.750+0000 7f27ed800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27e81516b0 0x7f27e8155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ed800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27e81516b0 0x7f27e8155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55242/0 (socket says 172.21.15.98:55242) 2024-05-27T00:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ed800640 1 -- 172.21.15.98:0/2587458010 learned_addr learned my addr 172.21.15.98:0/2587458010 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ece00640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27e80b1e10 0x7f27e8163fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27dfe00640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27e80c2b10 0x7f27e81552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ed800640 1 -- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27e80c2b10 msgr2=0x7f27e81552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ed800640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27e80c2b10 0x7f27e81552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ed800640 1 -- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27e80b1e10 msgr2=0x7f27e8163fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ed800640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27e80b1e10 0x7f27e8163fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ed800640 1 -- 172.21.15.98:0/2587458010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e8156b80 con 0x7f27e81516b0 2024-05-27T00:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27dfe00640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27e80c2b10 0x7f27e81552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ece00640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27e80b1e10 0x7f27e8163fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ed800640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27e81516b0 0x7f27e8155830 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009fd0 tx=0x7f27e000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27e5600640 1 -- 172.21.15.98:0/2587458010 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e001a070 con 0x7f27e81516b0 2024-05-27T00:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27e5600640 1 -- 172.21.15.98:0/2587458010 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e0003bc0 con 0x7f27e81516b0 2024-05-27T00:33:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e8156e20 con 0x7f27e81516b0 2024-05-27T00:33:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27e5600640 1 -- 172.21.15.98:0/2587458010 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0004030 con 0x7f27e81516b0 2024-05-27T00:33:44.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e81573e0 con 0x7f27e81516b0 2024-05-27T00:33:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f27e8075cb0 con 0x7f27e81516b0 2024-05-27T00:33:44.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27e5600640 1 -- 172.21.15.98:0/2587458010 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f27e001d070 con 0x7f27e81516b0 2024-05-27T00:33:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27e5600640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27c40678c0 0x7f27c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.754+0000 7f27e5600640 1 -- 172.21.15.98:0/2587458010 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f27e0086c90 con 0x7f27e81516b0 2024-05-27T00:33:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.758+0000 7f27e5600640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] conn(0x7f27c40704b0 0x7f27c4072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.758+0000 7f27ece00640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27c40678c0 0x7f27c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.758+0000 7f27e5600640 1 -- 172.21.15.98:0/2587458010 --> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27c4073000 con 0x7f27c40704b0 2024-05-27T00:33:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.758+0000 7f27e5600640 1 -- 172.21.15.98:0/2587458010 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f27e0087040 con 0x7f27e81516b0 2024-05-27T00:33:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.758+0000 7f27dfe00640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] conn(0x7f27c40704b0 0x7f27c4072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.758+0000 7f27dfe00640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] conn(0x7f27c40704b0 0x7f27c4072930 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.758+0000 7f27ece00640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27c40678c0 0x7f27c4069d80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f27d8005890 tx=0x7f27d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.758+0000 7f27e5600640 1 -- 172.21.15.98:0/2587458010 <== osd.3 v2:172.21.15.98:6817/1805101132 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27c4073000 con 0x7f27c40704b0 2024-05-27T00:33:44.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 --> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f27e806efd0 con 0x7f27c40704b0 2024-05-27T00:33:44.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27e5600640 1 -- 172.21.15.98:0/2587458010 <== osd.3 v2:172.21.15.98:6817/1805101132 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f27e806efd0 con 0x7f27c40704b0 2024-05-27T00:33:44.772 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-27T00:33:44.772 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-27T00:33:44.773 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-27T00:33:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 >> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] conn(0x7f27c40704b0 msgr2=0x7f27c4072930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.98:6817/1805101132,v1:172.21.15.98:6819/1805101132] conn(0x7f27c40704b0 0x7f27c4072930 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27c40678c0 msgr2=0x7f27c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f27c40678c0 0x7f27c4069d80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f27d8005890 tx=0x7f27d8009040 comp rx=0 tx=0).stop 2024-05-27T00:33:44.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27e81516b0 msgr2=0x7f27e8155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 --2- 172.21.15.98:0/2587458010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27e81516b0 0x7f27e8155830 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009fd0 tx=0x7f27e000d560 comp rx=0 tx=0).stop 2024-05-27T00:33:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ed800640 1 -- 172.21.15.98:0/2587458010 reap_dead start 2024-05-27T00:33:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 shutdown_connections 2024-05-27T00:33:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 >> 172.21.15.98:0/2587458010 conn(0x7f27e80cbf00 msgr2=0x7f27e80730c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 shutdown_connections 2024-05-27T00:33:44.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.770+0000 7f27ef800640 1 -- 172.21.15.98:0/2587458010 wait complete. 2024-05-27T00:33:44.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-27T00:33:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd18a00640 1 Processor -- start 2024-05-27T00:33:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd18a00640 1 -- start start 2024-05-27T00:33:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd18a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbd14154e90 0x7fbd140c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd18a00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fbd1406c620 0x7fbd140732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd18a00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd1406cd90 con 0x7fbd14073810 2024-05-27T00:33:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd18a00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbd1406c200 con 0x7fbd14154e90 2024-05-27T00:33:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd18a00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fbd140c97e0 con 0x7fbd1406c620 2024-05-27T00:33:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd11e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbd14154e90 0x7fbd140c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd12800640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbd14073810 0x7fbd14067130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37310/0 (socket says 172.21.15.98:37310) 2024-05-27T00:33:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd12800640 1 -- 172.21.15.98:0/2811279521 learned_addr learned my addr 172.21.15.98:0/2811279521 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd11400640 1 --2- 172.21.15.98:0/2811279521 >> v2:172.21.15.204:3300/0 conn(0x7fbd1406c620 0x7fbd140732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd11e00640 1 -- 172.21.15.98:0/2811279521 >> v2:172.21.15.204:3300/0 conn(0x7fbd1406c620 msgr2=0x7fbd140732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd11e00640 1 --2- 172.21.15.98:0/2811279521 >> v2:172.21.15.204:3300/0 conn(0x7fbd1406c620 0x7fbd140732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd11e00640 1 -- 172.21.15.98:0/2811279521 >> v1:172.21.15.98:6789/0 conn(0x7fbd14073810 legacy=0x7fbd14067130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd11e00640 1 -- 172.21.15.98:0/2811279521 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1406cab0 con 0x7fbd14154e90 2024-05-27T00:33:44.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.930+0000 7fbd10a00640 1 -- 172.21.15.98:0/2811279521 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 602096579 0 0) 0x7fbd1406cd90 con 0x7fbd14073810 2024-05-27T00:33:44.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd11e00640 1 --2- 172.21.15.98:0/2811279521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd14154e90 0x7fbd140c92a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fbd140680b0 tx=0x7fbd0401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6dc6f55a8ef781 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd10a00640 1 -- 172.21.15.98:0/2811279521 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd040b5070 con 0x7fbd14154e90 2024-05-27T00:33:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd10a00640 1 -- 172.21.15.98:0/2811279521 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd040a8990 con 0x7fbd14154e90 2024-05-27T00:33:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd10a00640 1 -- 172.21.15.98:0/2811279521 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd040a8d60 con 0x7fbd14154e90 2024-05-27T00:33:44.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd18a00640 1 -- 172.21.15.98:0/2811279521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd14154e90 msgr2=0x7fbd140c92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd18a00640 1 --2- 172.21.15.98:0/2811279521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd14154e90 0x7fbd140c92a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fbd140680b0 tx=0x7fbd0401ec90 comp rx=0 tx=0).stop 2024-05-27T00:33:44.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd18a00640 1 -- 172.21.15.98:0/2811279521 shutdown_connections 2024-05-27T00:33:44.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd18a00640 1 --2- 172.21.15.98:0/2811279521 >> v2:172.21.15.204:3300/0 conn(0x7fbd1406c620 0x7fbd140732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd18a00640 1 --2- 172.21.15.98:0/2811279521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd14154e90 0x7fbd140c92a0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd18a00640 1 -- 172.21.15.98:0/2811279521 >> 172.21.15.98:0/2811279521 conn(0x7fbd140c9b70 msgr2=0x7fbd140c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd18a00640 1 -- 172.21.15.98:0/2811279521 shutdown_connections 2024-05-27T00:33:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.934+0000 7fbd18a00640 1 -- 172.21.15.98:0/2811279521 wait complete. 2024-05-27T00:33:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd18a00640 1 Processor -- start 2024-05-27T00:33:44.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd18a00640 1 -- start start 2024-05-27T00:33:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd18a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd14073810 0x7fbd1414caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd18a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14154e90 0x7fbd140b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd18a00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd1415c050 0x7fbd1406dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd18a00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd14072db0 con 0x7fbd1415c050 2024-05-27T00:33:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd18a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1406bf20 con 0x7fbd14154e90 2024-05-27T00:33:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd18a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbd1415a930 con 0x7fbd14073810 2024-05-27T00:33:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd12800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd1415c050 0x7fbd1406dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd12800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd1415c050 0x7fbd1406dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55266/0 (socket says 172.21.15.98:55266) 2024-05-27T00:33:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd12800640 1 -- 172.21.15.98:0/3302475612 learned_addr learned my addr 172.21.15.98:0/3302475612 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:44.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd11400640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14154e90 0x7fbd140b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd11e00640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd14073810 0x7fbd1414caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd12800640 1 -- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd14073810 msgr2=0x7fbd1414caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd12800640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd14073810 0x7fbd1414caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd12800640 1 -- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14154e90 msgr2=0x7fbd140b0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:44.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd12800640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14154e90 0x7fbd140b0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:44.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd12800640 1 -- 172.21.15.98:0/3302475612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd14154340 con 0x7fbd1415c050 2024-05-27T00:33:44.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd11e00640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd14073810 0x7fbd1414caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:33:44.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd12800640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd1415c050 0x7fbd1406dbc0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00c740 tx=0x7fbd0c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd0a000640 1 -- 172.21.15.98:0/3302475612 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c018070 con 0x7fbd1415c050 2024-05-27T00:33:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd0a000640 1 -- 172.21.15.98:0/3302475612 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd0c00ce50 con 0x7fbd1415c050 2024-05-27T00:33:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd0a000640 1 -- 172.21.15.98:0/3302475612 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c00a880 con 0x7fbd1415c050 2024-05-27T00:33:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd18a00640 1 -- 172.21.15.98:0/3302475612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd1415d010 con 0x7fbd1415c050 2024-05-27T00:33:44.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.938+0000 7fbd18a00640 1 -- 172.21.15.98:0/3302475612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd1415d5d0 con 0x7fbd1415c050 2024-05-27T00:33:44.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.942+0000 7fbd18a00640 1 -- 172.21.15.98:0/3302475612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd14075cb0 con 0x7fbd1415c050 2024-05-27T00:33:44.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.942+0000 7fbd0a000640 1 -- 172.21.15.98:0/3302475612 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbd0c00aa20 con 0x7fbd1415c050 2024-05-27T00:33:44.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.942+0000 7fbd0a000640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbcf40678c0 0x7fbcf4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:44.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.942+0000 7fbd0a000640 1 -- 172.21.15.98:0/3302475612 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fbd0c0856f0 con 0x7fbd1415c050 2024-05-27T00:33:44.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.942+0000 7fbd11e00640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbcf40678c0 0x7fbcf4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:44.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.946+0000 7fbd11e00640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbcf40678c0 0x7fbcf4069d80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbd040b2000 tx=0x7fbd04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:44.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:44.946+0000 7fbd0a000640 1 -- 172.21.15.98:0/3302475612 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fbd0c0556c0 con 0x7fbd1415c050 2024-05-27T00:33:45.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.110+0000 7fbd18a00640 1 -- 172.21.15.98:0/3302475612 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fbd140b24a0 con 0x7fbcf40678c0 2024-05-27T00:33:45.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.110+0000 7fbd0a000640 1 -- 172.21.15.98:0/3302475612 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbd140b24a0 con 0x7fbcf40678c0 2024-05-27T00:33:45.112 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 3 to deep-scrub 2024-05-27T00:33:45.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 -- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbcf40678c0 msgr2=0x7fbcf4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbcf40678c0 0x7fbcf4069d80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbd040b2000 tx=0x7fbd04008040 comp rx=0 tx=0).stop 2024-05-27T00:33:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 -- 172.21.15.98:0/3302475612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd1415c050 msgr2=0x7fbd1406dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd1415c050 0x7fbd1406dbc0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00c740 tx=0x7fbd0c00cc00 comp rx=0 tx=0).stop 2024-05-27T00:33:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 -- 172.21.15.98:0/3302475612 shutdown_connections 2024-05-27T00:33:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd1415c050 0x7fbd1406dbc0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fbcf40678c0 0x7fbcf4069d80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14154e90 0x7fbd140b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 --2- 172.21.15.98:0/3302475612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd14073810 0x7fbd1414caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 -- 172.21.15.98:0/3302475612 >> 172.21.15.98:0/3302475612 conn(0x7fbd140c9b70 msgr2=0x7fbd1415a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:45.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 -- 172.21.15.98:0/3302475612 shutdown_connections 2024-05-27T00:33:45.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.114+0000 7fbd18a00640 1 -- 172.21.15.98:0/3302475612 wait complete. 2024-05-27T00:33:45.134 INFO:tasks.ceph:Scrubbing osd.4 2024-05-27T00:33:45.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-27T00:33:45.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.270+0000 7f925e400640 1 Processor -- start 2024-05-27T00:33:45.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.270+0000 7f925e400640 1 -- start start 2024-05-27T00:33:45.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925e400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92580c67b0 0x7f92580c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925e400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f925806f310 0x7f92580c1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925e400640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92580c1ab0 con 0x7f92580c6b90 2024-05-27T00:33:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925e400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f925806e300 con 0x7f92580c67b0 2024-05-27T00:33:45.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925e400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f925806ee80 con 0x7f925806f310 2024-05-27T00:33:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925d000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92580c67b0 0x7f92580c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925da00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f92580c6b90 0x7f9258154530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37320/0 (socket says 172.21.15.98:37320) 2024-05-27T00:33:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925da00640 1 -- 172.21.15.98:0/3714135634 learned_addr learned my addr 172.21.15.98:0/3714135634 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f9257e00640 1 --2- 172.21.15.98:0/3714135634 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f925806f310 0x7f92580c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925d000640 1 -- 172.21.15.98:0/3714135634 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f925806f310 msgr2=0x7f92580c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925d000640 1 --2- 172.21.15.98:0/3714135634 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f925806f310 0x7f92580c1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925d000640 1 -- 172.21.15.98:0/3714135634 >> v1:172.21.15.98:6789/0 conn(0x7f92580c6b90 legacy=0x7f9258154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f9257400640 1 -- 172.21.15.98:0/3714135634 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828604335 0 0) 0x7f92580c1ab0 con 0x7f92580c6b90 2024-05-27T00:33:45.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.274+0000 7f925d000640 1 -- 172.21.15.98:0/3714135634 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925815b570 con 0x7f92580c67b0 2024-05-27T00:33:45.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925d000640 1 --2- 172.21.15.98:0/3714135634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92580c67b0 0x7f92580c0eb0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9248009d90 tx=0x7f924801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ff293859dd3dc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f9257400640 1 -- 172.21.15.98:0/3714135634 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480b4070 con 0x7f92580c67b0 2024-05-27T00:33:45.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f9257400640 1 -- 172.21.15.98:0/3714135634 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92480af600 con 0x7f92580c67b0 2024-05-27T00:33:45.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f9257400640 1 -- 172.21.15.98:0/3714135634 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480af9d0 con 0x7f92580c67b0 2024-05-27T00:33:45.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925e400640 1 -- 172.21.15.98:0/3714135634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92580c67b0 msgr2=0x7f92580c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925e400640 1 --2- 172.21.15.98:0/3714135634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92580c67b0 0x7f92580c0eb0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9248009d90 tx=0x7f924801f150 comp rx=0 tx=0).stop 2024-05-27T00:33:45.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925e400640 1 -- 172.21.15.98:0/3714135634 shutdown_connections 2024-05-27T00:33:45.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925e400640 1 --2- 172.21.15.98:0/3714135634 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f925806f310 0x7f92580c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925e400640 1 --2- 172.21.15.98:0/3714135634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92580c67b0 0x7f92580c0eb0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925e400640 1 -- 172.21.15.98:0/3714135634 >> 172.21.15.98:0/3714135634 conn(0x7f92580c8650 msgr2=0x7f92580c8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:45.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925e400640 1 -- 172.21.15.98:0/3714135634 shutdown_connections 2024-05-27T00:33:45.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925e400640 1 -- 172.21.15.98:0/3714135634 wait complete. 2024-05-27T00:33:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925e400640 1 Processor -- start 2024-05-27T00:33:45.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.278+0000 7f925e400640 1 -- start start 2024-05-27T00:33:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925e400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f925806f310 0x7f92581e4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925e400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f92580c67b0 0x7f92581e5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925e400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f92580c6b90 0x7f92581f0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925e400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f925806eb10 con 0x7f925806f310 2024-05-27T00:33:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925e400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f925806e830 con 0x7f92580c6b90 2024-05-27T00:33:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925e400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9258154c40 con 0x7f92580c67b0 2024-05-27T00:33:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f925806f310 0x7f92581e4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f925806f310 0x7f92581e4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55282/0 (socket says 172.21.15.98:55282) 2024-05-27T00:33:45.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925d000640 1 -- 172.21.15.98:0/2920108386 learned_addr learned my addr 172.21.15.98:0/2920108386 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925da00640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f92580c6b90 0x7f92581f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f9257e00640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f92580c67b0 0x7f92581e5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925d000640 1 -- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f92580c67b0 msgr2=0x7f92581e5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925d000640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f92580c67b0 0x7f92581e5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925d000640 1 -- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f92580c6b90 msgr2=0x7f92581f0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925d000640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f92580c6b90 0x7f92581f0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925d000640 1 -- 172.21.15.98:0/2920108386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92581f41d0 con 0x7f925806f310 2024-05-27T00:33:45.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f9257e00640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f92580c67b0 0x7f92581e5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T00:33:45.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925d000640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f925806f310 0x7f92581e4b30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f9248009d90 tx=0x7f924801bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f9254c00640 1 -- 172.21.15.98:0/2920108386 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480b4040 con 0x7f925806f310 2024-05-27T00:33:45.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f9254c00640 1 -- 172.21.15.98:0/2920108386 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92480039a0 con 0x7f925806f310 2024-05-27T00:33:45.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f9254c00640 1 -- 172.21.15.98:0/2920108386 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248003d50 con 0x7f925806f310 2024-05-27T00:33:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9258155f60 con 0x7f925806f310 2024-05-27T00:33:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92581f4470 con 0x7f925806f310 2024-05-27T00:33:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9258075c40 con 0x7f925806f310 2024-05-27T00:33:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.282+0000 7f9254c00640 1 -- 172.21.15.98:0/2920108386 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f92480c4060 con 0x7f925806f310 2024-05-27T00:33:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.286+0000 7f9254c00640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92380678f0 0x7f9238069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.286+0000 7f9254c00640 1 -- 172.21.15.98:0/2920108386 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f9248123fa0 con 0x7f925806f310 2024-05-27T00:33:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.286+0000 7f9254c00640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] conn(0x7f92380704e0 0x7f9238072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.286+0000 7f9254c00640 1 -- 172.21.15.98:0/2920108386 --> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9238073030 con 0x7f92380704e0 2024-05-27T00:33:45.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.286+0000 7f9254c00640 1 -- 172.21.15.98:0/2920108386 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f92481265c0 con 0x7f925806f310 2024-05-27T00:33:45.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.286+0000 7f9257e00640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92380678f0 0x7f9238069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.286+0000 7f925da00640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] conn(0x7f92380704e0 0x7f9238072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.286+0000 7f9257e00640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92380678f0 0x7f9238069db0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f92580b1dd0 tx=0x7f9244008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.286+0000 7f925da00640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] conn(0x7f92380704e0 0x7f9238072960 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.290+0000 7f9254c00640 1 -- 172.21.15.98:0/2920108386 <== osd.4 v2:172.21.15.204:6800/4073643582 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9238073030 con 0x7f92380704e0 2024-05-27T00:33:45.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 --> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f92581580f0 con 0x7f92380704e0 2024-05-27T00:33:45.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f9254c00640 1 -- 172.21.15.98:0/2920108386 <== osd.4 v2:172.21.15.204:6800/4073643582 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f92581580f0 con 0x7f92380704e0 2024-05-27T00:33:45.303 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-27T00:33:45.303 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-27T00:33:45.303 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-27T00:33:45.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] conn(0x7f92380704e0 msgr2=0x7f9238072960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:6800/4073643582,v1:172.21.15.204:6801/4073643582] conn(0x7f92380704e0 0x7f9238072960 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92380678f0 msgr2=0x7f9238069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f92380678f0 0x7f9238069db0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f92580b1dd0 tx=0x7f9244008040 comp rx=0 tx=0).stop 2024-05-27T00:33:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f925806f310 msgr2=0x7f92581e4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 --2- 172.21.15.98:0/2920108386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f925806f310 0x7f92581e4b30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f9248009d90 tx=0x7f924801bef0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925da00640 1 -- 172.21.15.98:0/2920108386 reap_dead start 2024-05-27T00:33:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 shutdown_connections 2024-05-27T00:33:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 >> 172.21.15.98:0/2920108386 conn(0x7f92580c8650 msgr2=0x7f9258072e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 shutdown_connections 2024-05-27T00:33:45.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.302+0000 7f925e400640 1 -- 172.21.15.98:0/2920108386 wait complete. 2024-05-27T00:33:45.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-27T00:33:45.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa738800640 1 Processor -- start 2024-05-27T00:33:45.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa738800640 1 -- start start 2024-05-27T00:33:45.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa738800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa730158490 0x7fa73006e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa738800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73006edd0 0x7fa73015c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa738800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa730158920 con 0x7fa730152770 2024-05-27T00:33:45.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa738800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa730159480 con 0x7fa730158490 2024-05-27T00:33:45.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa738800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa730158be0 con 0x7fa73006edd0 2024-05-27T00:33:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa735e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa730152770 0x7fa7300c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37332/0 (socket says 172.21.15.98:37332) 2024-05-27T00:33:45.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa735e00640 1 -- 172.21.15.98:0/1868066656 learned_addr learned my addr 172.21.15.98:0/1868066656 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:45.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa735400640 1 --2- 172.21.15.98:0/1868066656 >> v2:172.21.15.204:3300/0 conn(0x7fa730158490 0x7fa73006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa736800640 1 --2- 172.21.15.98:0/1868066656 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73006edd0 0x7fa73015c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa734a00640 1 -- 172.21.15.98:0/1868066656 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1086536986 0 0) 0x7fa730158920 con 0x7fa730152770 2024-05-27T00:33:45.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa734a00640 1 -- 172.21.15.98:0/1868066656 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa718003530 con 0x7fa730152770 2024-05-27T00:33:45.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa736800640 1 -- 172.21.15.98:0/1868066656 >> v2:172.21.15.204:3300/0 conn(0x7fa730158490 msgr2=0x7fa73006e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:45.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa736800640 1 --2- 172.21.15.98:0/1868066656 >> v2:172.21.15.204:3300/0 conn(0x7fa730158490 0x7fa73006e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa736800640 1 -- 172.21.15.98:0/1868066656 >> v1:172.21.15.98:6789/0 conn(0x7fa730152770 legacy=0x7fa7300c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa734a00640 1 -- 172.21.15.98:0/1868066656 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 148823270 0 0) 0x7fa718003530 con 0x7fa730152770 2024-05-27T00:33:45.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.458+0000 7fa736800640 1 -- 172.21.15.98:0/1868066656 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa730158ec0 con 0x7fa73006edd0 2024-05-27T00:33:45.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa736800640 1 --2- 172.21.15.98:0/1868066656 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73006edd0 0x7fa73015c0d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009d90 tx=0x7fa72c01c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9eba962b5d0200d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa734a00640 1 -- 172.21.15.98:0/1868066656 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72c0ac070 con 0x7fa73006edd0 2024-05-27T00:33:45.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa734a00640 1 -- 172.21.15.98:0/1868066656 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa72c0a64a0 con 0x7fa73006edd0 2024-05-27T00:33:45.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa734a00640 1 -- 172.21.15.98:0/1868066656 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa724001680 con 0x7fa73006edd0 2024-05-27T00:33:45.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa738800640 1 -- 172.21.15.98:0/1868066656 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73006edd0 msgr2=0x7fa73015c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa738800640 1 --2- 172.21.15.98:0/1868066656 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73006edd0 0x7fa73015c0d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009d90 tx=0x7fa72c01c890 comp rx=0 tx=0).stop 2024-05-27T00:33:45.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa738800640 1 -- 172.21.15.98:0/1868066656 shutdown_connections 2024-05-27T00:33:45.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa738800640 1 --2- 172.21.15.98:0/1868066656 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73006edd0 0x7fa73015c0d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa738800640 1 --2- 172.21.15.98:0/1868066656 >> v2:172.21.15.204:3300/0 conn(0x7fa730158490 0x7fa73006e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa738800640 1 -- 172.21.15.98:0/1868066656 >> 172.21.15.98:0/1868066656 conn(0x7fa7300cc200 msgr2=0x7fa7300cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:45.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa738800640 1 -- 172.21.15.98:0/1868066656 shutdown_connections 2024-05-27T00:33:45.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.462+0000 7fa738800640 1 -- 172.21.15.98:0/1868066656 wait complete. 2024-05-27T00:33:45.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa738800640 1 Processor -- start 2024-05-27T00:33:45.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa738800640 1 -- start start 2024-05-27T00:33:45.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa738800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa73006edd0 0x7fa7300c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa738800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730152770 0x7fa73014bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa738800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73014d870 0x7fa73014a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa738800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa730070fa0 con 0x7fa73006edd0 2024-05-27T00:33:45.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa738800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa730152b50 con 0x7fa730152770 2024-05-27T00:33:45.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa738800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa730165630 con 0x7fa73014d870 2024-05-27T00:33:45.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa735e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa73006edd0 0x7fa7300c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa735e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa73006edd0 0x7fa7300c2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55296/0 (socket says 172.21.15.98:55296) 2024-05-27T00:33:45.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa735e00640 1 -- 172.21.15.98:0/2099095810 learned_addr learned my addr 172.21.15.98:0/2099095810 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:45.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa735400640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730152770 0x7fa73014bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa736800640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73014d870 0x7fa73014a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa735e00640 1 -- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73014d870 msgr2=0x7fa73014a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa735e00640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73014d870 0x7fa73014a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa735e00640 1 -- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730152770 msgr2=0x7fa73014bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa735e00640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730152770 0x7fa73014bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa735e00640 1 -- 172.21.15.98:0/2099095810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73014c050 con 0x7fa73006edd0 2024-05-27T00:33:45.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa735e00640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa73006edd0 0x7fa7300c2690 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fa72400c730 tx=0x7fa72400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa72a000640 1 -- 172.21.15.98:0/2099095810 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa724017070 con 0x7fa73006edd0 2024-05-27T00:33:45.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa72a000640 1 -- 172.21.15.98:0/2099095810 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa724005ae0 con 0x7fa73006edd0 2024-05-27T00:33:45.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa738800640 1 -- 172.21.15.98:0/2099095810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa73014a570 con 0x7fa73006edd0 2024-05-27T00:33:45.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa72a000640 1 -- 172.21.15.98:0/2099095810 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa724007070 con 0x7fa73006edd0 2024-05-27T00:33:45.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.466+0000 7fa738800640 1 -- 172.21.15.98:0/2099095810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa730148cb0 con 0x7fa73006edd0 2024-05-27T00:33:45.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.470+0000 7fa738800640 1 -- 172.21.15.98:0/2099095810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa730075cb0 con 0x7fa73006edd0 2024-05-27T00:33:45.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.470+0000 7fa72a000640 1 -- 172.21.15.98:0/2099095810 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa72401a070 con 0x7fa73006edd0 2024-05-27T00:33:45.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.470+0000 7fa72a000640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7140678c0 0x7fa714069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.470+0000 7fa72a000640 1 -- 172.21.15.98:0/2099095810 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fa724085700 con 0x7fa73006edd0 2024-05-27T00:33:45.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.470+0000 7fa735400640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7140678c0 0x7fa714069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.474+0000 7fa735400640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7140678c0 0x7fa714069d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa730161ff0 tx=0x7fa720009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.478+0000 7fa72a000640 1 -- 172.21.15.98:0/2099095810 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fa7240556d0 con 0x7fa73006edd0 2024-05-27T00:33:45.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.674+0000 7fa738800640 1 -- 172.21.15.98:0/2099095810 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fa7301481b0 con 0x7fa7140678c0 2024-05-27T00:33:45.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.674+0000 7fa72a000640 1 -- 172.21.15.98:0/2099095810 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa7301481b0 con 0x7fa7140678c0 2024-05-27T00:33:45.675 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 4 to deep-scrub 2024-05-27T00:33:45.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 -- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7140678c0 msgr2=0x7fa714069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7140678c0 0x7fa714069d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa730161ff0 tx=0x7fa720009190 comp rx=0 tx=0).stop 2024-05-27T00:33:45.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 -- 172.21.15.98:0/2099095810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa73006edd0 msgr2=0x7fa7300c2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa73006edd0 0x7fa7300c2690 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fa72400c730 tx=0x7fa72400cbf0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 -- 172.21.15.98:0/2099095810 shutdown_connections 2024-05-27T00:33:45.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa73014d870 0x7fa73014a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa7140678c0 0x7fa714069d80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730152770 0x7fa73014bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 --2- 172.21.15.98:0/2099095810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa73006edd0 0x7fa7300c2690 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 -- 172.21.15.98:0/2099095810 >> 172.21.15.98:0/2099095810 conn(0x7fa7300cc200 msgr2=0x7fa730162680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:45.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 -- 172.21.15.98:0/2099095810 shutdown_connections 2024-05-27T00:33:45.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.678+0000 7fa738800640 1 -- 172.21.15.98:0/2099095810 wait complete. 2024-05-27T00:33:45.707 INFO:tasks.ceph:Scrubbing osd.5 2024-05-27T00:33:45.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-27T00:33:45.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.850+0000 7fa069000640 1 Processor -- start 2024-05-27T00:33:45.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.850+0000 7fa069000640 1 -- start start 2024-05-27T00:33:45.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa069000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7fa0640c70b0 0x7fa064166210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa069000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0640c7be0 con 0x7fa064154ad0 2024-05-27T00:33:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa069000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06415ade0 con 0x7fa06415a880 2024-05-27T00:33:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa069000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7fa06415de70 con 0x7fa0640c70b0 2024-05-27T00:33:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa061e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa064154ad0 0x7fa06415e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37344/0 (socket says 172.21.15.98:37344) 2024-05-27T00:33:45.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa061e00640 1 -- 172.21.15.98:0/1597388520 learned_addr learned my addr 172.21.15.98:0/1597388520 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:45.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa062800640 1 --2- 172.21.15.98:0/1597388520 >> v2:172.21.15.204:3300/0 conn(0x7fa0640c70b0 0x7fa064166210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2882896948 0 0) 0x7fa0640c7be0 con 0x7fa064154ad0 2024-05-27T00:33:45.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa048003530 con 0x7fa064154ad0 2024-05-27T00:33:45.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3869734402 0 0) 0x7fa048003530 con 0x7fa064154ad0 2024-05-27T00:33:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 >> v2:172.21.15.204:3300/0 conn(0x7fa0640c70b0 msgr2=0x7fa064166210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 --2- 172.21.15.98:0/1597388520 >> v2:172.21.15.204:3300/0 conn(0x7fa0640c70b0 0x7fa064166210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 >> v1:172.21.15.204:6789/0 conn(0x7fa06415a880 legacy=0x7fa0640c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06415b0c0 con 0x7fa064154ad0 2024-05-27T00:33:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7fa054001a00 con 0x7fa064154ad0 2024-05-27T00:33:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 >> v1:172.21.15.98:6789/0 conn(0x7fa064154ad0 legacy=0x7fa06415e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0480039c0 0x7fa048003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa048004810 0x7fa048004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0480051d0 0x7fa04800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa06415de70 con 0x7fa0480039c0 2024-05-27T00:33:45.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa04800c070 con 0x7fa0480051d0 2024-05-27T00:33:45.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa04800c270 con 0x7fa048004810 2024-05-27T00:33:45.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa061e00640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0480039c0 0x7fa048003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa061400640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa048004810 0x7fa048004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa062800640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0480051d0 0x7fa04800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa061e00640 1 -- 172.21.15.98:0/1597388520 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa048004810 msgr2=0x7fa048004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa061e00640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa048004810 0x7fa048004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa061e00640 1 -- 172.21.15.98:0/1597388520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0480051d0 msgr2=0x7fa04800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa061e00640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0480051d0 0x7fa04800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa061e00640 1 -- 172.21.15.98:0/1597388520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa048003530 con 0x7fa0480039c0 2024-05-27T00:33:45.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.854+0000 7fa062800640 1 -- 172.21.15.98:0/1597388520 reap_dead start 2024-05-27T00:33:45.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa061e00640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0480039c0 0x7fa048003de0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa05400a840 tx=0x7fa05401d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57928f7f4fcd19c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05400a460 con 0x7fa0480039c0 2024-05-27T00:33:45.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0540ad650 con 0x7fa0480039c0 2024-05-27T00:33:45.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa060a00640 1 -- 172.21.15.98:0/1597388520 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0540ada80 con 0x7fa0480039c0 2024-05-27T00:33:45.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa069000640 1 -- 172.21.15.98:0/1597388520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0480039c0 msgr2=0x7fa048003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa069000640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0480039c0 0x7fa048003de0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa05400a840 tx=0x7fa05401d320 comp rx=0 tx=0).stop 2024-05-27T00:33:45.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa069000640 1 -- 172.21.15.98:0/1597388520 shutdown_connections 2024-05-27T00:33:45.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa069000640 1 --2- 172.21.15.98:0/1597388520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0480039c0 0x7fa048003de0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa069000640 1 -- 172.21.15.98:0/1597388520 >> 172.21.15.98:0/1597388520 conn(0x7fa06406dca0 msgr2=0x7fa06406b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa069000640 1 -- 172.21.15.98:0/1597388520 shutdown_connections 2024-05-27T00:33:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa069000640 1 -- 172.21.15.98:0/1597388520 wait complete. 2024-05-27T00:33:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa069000640 1 Processor -- start 2024-05-27T00:33:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.858+0000 7fa069000640 1 -- start start 2024-05-27T00:33:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa069000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0480039c0 0x7fa0641e8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa069000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0640c4ca0 0x7fa0640c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa069000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0641f3e30 0x7fa064148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa069000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa0640c9140 con 0x7fa0640c4ca0 2024-05-27T00:33:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa069000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa064154eb0 con 0x7fa0641f3e30 2024-05-27T00:33:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa069000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa0640c92c0 con 0x7fa0480039c0 2024-05-27T00:33:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa061400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0640c4ca0 0x7fa0640c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa061400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0640c4ca0 0x7fa0640c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55324/0 (socket says 172.21.15.98:55324) 2024-05-27T00:33:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa061400640 1 -- 172.21.15.98:0/29794211 learned_addr learned my addr 172.21.15.98:0/29794211 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa061e00640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0480039c0 0x7fa0641e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa062800640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0641f3e30 0x7fa064148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa061400640 1 -- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0480039c0 msgr2=0x7fa0641e8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa061400640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0480039c0 0x7fa0641e8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa061400640 1 -- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0641f3e30 msgr2=0x7fa064148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa061400640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0641f3e30 0x7fa064148040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa061400640 1 -- 172.21.15.98:0/29794211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0640ca700 con 0x7fa0640c4ca0 2024-05-27T00:33:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa061400640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0640c4ca0 0x7fa0640c43c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa05000e530 tx=0x7fa05000e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa05a000640 1 -- 172.21.15.98:0/29794211 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0500053a0 con 0x7fa0640c4ca0 2024-05-27T00:33:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa05a000640 1 -- 172.21.15.98:0/29794211 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa050018070 con 0x7fa0640c4ca0 2024-05-27T00:33:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0640ca9e0 con 0x7fa0640c4ca0 2024-05-27T00:33:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa05a000640 1 -- 172.21.15.98:0/29794211 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa050003820 con 0x7fa0640c4ca0 2024-05-27T00:33:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0640cc6a0 con 0x7fa0640c4ca0 2024-05-27T00:33:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa0640ca4b0 con 0x7fa0640c4ca0 2024-05-27T00:33:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.862+0000 7fa05a000640 1 -- 172.21.15.98:0/29794211 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa0500039c0 con 0x7fa0640c4ca0 2024-05-27T00:33:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.866+0000 7fa05a000640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0440678c0 0x7fa044069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.866+0000 7fa05a000640 1 -- 172.21.15.98:0/29794211 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fa0500866c0 con 0x7fa0640c4ca0 2024-05-27T00:33:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.866+0000 7fa05a000640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] conn(0x7fa0440704c0 0x7fa044072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.866+0000 7fa05a000640 1 -- 172.21.15.98:0/29794211 --> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa050013e50 con 0x7fa0440704c0 2024-05-27T00:33:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.866+0000 7fa05a000640 1 -- 172.21.15.98:0/29794211 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fa050086a70 con 0x7fa0640c4ca0 2024-05-27T00:33:45.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.866+0000 7fa061e00640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0440678c0 0x7fa044069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.866+0000 7fa062800640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] conn(0x7fa0440704c0 0x7fa044072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:45.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.866+0000 7fa061e00640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0440678c0 0x7fa044069d80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa0540ad000 tx=0x7fa0640c4350 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.866+0000 7fa062800640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] conn(0x7fa0440704c0 0x7fa044072940 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:45.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.866+0000 7fa05a000640 1 -- 172.21.15.98:0/29794211 <== osd.5 v2:172.21.15.204:6808/858542033 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa050013e50 con 0x7fa0440704c0 2024-05-27T00:33:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 --> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa064075cb0 con 0x7fa0440704c0 2024-05-27T00:33:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa05a000640 1 -- 172.21.15.98:0/29794211 <== osd.5 v2:172.21.15.204:6808/858542033 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa064075cb0 con 0x7fa0440704c0 2024-05-27T00:33:45.883 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-27T00:33:45.883 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-27T00:33:45.883 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-27T00:33:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] conn(0x7fa0440704c0 msgr2=0x7fa044072940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:6808/858542033,v1:172.21.15.204:6809/858542033] conn(0x7fa0440704c0 0x7fa044072940 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0440678c0 msgr2=0x7fa044069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fa0440678c0 0x7fa044069d80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa0540ad000 tx=0x7fa0640c4350 comp rx=0 tx=0).stop 2024-05-27T00:33:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0640c4ca0 msgr2=0x7fa0640c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 --2- 172.21.15.98:0/29794211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0640c4ca0 0x7fa0640c43c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa05000e530 tx=0x7fa05000e9f0 comp rx=0 tx=0).stop 2024-05-27T00:33:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa062800640 1 -- 172.21.15.98:0/29794211 reap_dead start 2024-05-27T00:33:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 shutdown_connections 2024-05-27T00:33:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 >> 172.21.15.98:0/29794211 conn(0x7fa06406dca0 msgr2=0x7fa064153b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 shutdown_connections 2024-05-27T00:33:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:45.882+0000 7fa069000640 1 -- 172.21.15.98:0/29794211 wait complete. 2024-05-27T00:33:45.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-27T00:33:46.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.070+0000 7f128d000640 1 Processor -- start 2024-05-27T00:33:46.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.070+0000 7f128d000640 1 -- start start 2024-05-27T00:33:46.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.070+0000 7f128d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288152a50 0x7f12880709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.070+0000 7f128d000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1288158e60 0x7f1288070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.070+0000 7f128d000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f128815be40 con 0x7f12880cb7e0 2024-05-27T00:33:46.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.070+0000 7f128d000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f12880c8a00 con 0x7f1288158e60 2024-05-27T00:33:46.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.070+0000 7f128d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f12880c50e0 con 0x7f1288152a50 2024-05-27T00:33:46.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.074+0000 7f1285e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288152a50 0x7f12880709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.074+0000 7f1285e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288152a50 0x7f12880709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:55652/0 (socket says 172.21.15.98:55652) 2024-05-27T00:33:46.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.074+0000 7f1285e00640 1 -- 172.21.15.98:0/602253839 learned_addr learned my addr 172.21.15.98:0/602253839 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:46.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.074+0000 7f1286800640 1 --2- 172.21.15.98:0/602253839 >> v2:172.21.15.204:3300/0 conn(0x7f1288158e60 0x7f1288070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.074+0000 7f1285e00640 1 -- 172.21.15.98:0/602253839 >> v2:172.21.15.204:3300/0 conn(0x7f1288158e60 msgr2=0x7f1288070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:46.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.074+0000 7f1285e00640 1 --2- 172.21.15.98:0/602253839 >> v2:172.21.15.204:3300/0 conn(0x7f1288158e60 0x7f1288070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.074+0000 7f1285e00640 1 -- 172.21.15.98:0/602253839 >> v1:172.21.15.204:6789/0 conn(0x7f12880cb7e0 legacy=0x7f1288158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.074+0000 7f1285e00640 1 -- 172.21.15.98:0/602253839 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12880c8c50 con 0x7f1288152a50 2024-05-27T00:33:46.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f1285e00640 1 --2- 172.21.15.98:0/602253839 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288152a50 0x7f12880709a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1278009870 tx=0x7f127801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f5ebbca8b7b8b83 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:46.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f1284a00640 1 -- 172.21.15.98:0/602253839 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12780aa600 con 0x7f1288152a50 2024-05-27T00:33:46.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f1284a00640 1 -- 172.21.15.98:0/602253839 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12780af070 con 0x7f1288152a50 2024-05-27T00:33:46.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f1284a00640 1 -- 172.21.15.98:0/602253839 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12780aa9c0 con 0x7f1288152a50 2024-05-27T00:33:46.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f128d000640 1 -- 172.21.15.98:0/602253839 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288152a50 msgr2=0x7f12880709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f128d000640 1 --2- 172.21.15.98:0/602253839 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288152a50 0x7f12880709a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1278009870 tx=0x7f127801c4a0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f128d000640 1 -- 172.21.15.98:0/602253839 shutdown_connections 2024-05-27T00:33:46.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f128d000640 1 --2- 172.21.15.98:0/602253839 >> v2:172.21.15.204:3300/0 conn(0x7f1288158e60 0x7f1288070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f128d000640 1 --2- 172.21.15.98:0/602253839 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288152a50 0x7f12880709a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f128d000640 1 -- 172.21.15.98:0/602253839 >> 172.21.15.98:0/602253839 conn(0x7f1288072f90 msgr2=0x7f12880733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:46.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f128d000640 1 -- 172.21.15.98:0/602253839 shutdown_connections 2024-05-27T00:33:46.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f128d000640 1 -- 172.21.15.98:0/602253839 wait complete. 2024-05-27T00:33:46.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f128d000640 1 Processor -- start 2024-05-27T00:33:46.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.078+0000 7f128d000640 1 -- start start 2024-05-27T00:33:46.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f128d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12880cb7e0 0x7f12881598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f128d000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1288152a50 0x7f1288147810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f128d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288158e60 0x7f1288157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f128d000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1288152e30 con 0x7f1288152a50 2024-05-27T00:33:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f128d000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f12880cb570 con 0x7f12880cb7e0 2024-05-27T00:33:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f128d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f12880c9c10 con 0x7f1288158e60 2024-05-27T00:33:46.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1285400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1288152a50 0x7f1288147810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1285400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1288152a50 0x7f1288147810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55330/0 (socket says 172.21.15.98:55330) 2024-05-27T00:33:46.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1285400640 1 -- 172.21.15.98:0/655380542 learned_addr learned my addr 172.21.15.98:0/655380542 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:46.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1285e00640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12880cb7e0 0x7f12881598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1286800640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288158e60 0x7f1288157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1285400640 1 -- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288158e60 msgr2=0x7f1288157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1285400640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288158e60 0x7f1288157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1285400640 1 -- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12880cb7e0 msgr2=0x7f12881598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1285400640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12880cb7e0 0x7f12881598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1285400640 1 -- 172.21.15.98:0/655380542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12881f39d0 con 0x7f1288152a50 2024-05-27T00:33:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f1285400640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1288152a50 0x7f1288147810 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f1274007fd0 tx=0x7f127400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f127e000640 1 -- 172.21.15.98:0/655380542 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1274017070 con 0x7f1288152a50 2024-05-27T00:33:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f127e000640 1 -- 172.21.15.98:0/655380542 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12740052d0 con 0x7f1288152a50 2024-05-27T00:33:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f127e000640 1 -- 172.21.15.98:0/655380542 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1274003840 con 0x7f1288152a50 2024-05-27T00:33:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f128d000640 1 -- 172.21.15.98:0/655380542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12881f3c70 con 0x7f1288152a50 2024-05-27T00:33:46.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f128d000640 1 -- 172.21.15.98:0/655380542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12881f4230 con 0x7f1288152a50 2024-05-27T00:33:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.082+0000 7f128d000640 1 -- 172.21.15.98:0/655380542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f128814b9c0 con 0x7f1288152a50 2024-05-27T00:33:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.086+0000 7f127e000640 1 -- 172.21.15.98:0/655380542 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f127401a070 con 0x7f1288152a50 2024-05-27T00:33:46.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.086+0000 7f127e000640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12680678c0 0x7f1268069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.086+0000 7f127e000640 1 -- 172.21.15.98:0/655380542 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f12740857e0 con 0x7f1288152a50 2024-05-27T00:33:46.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.086+0000 7f1285e00640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12680678c0 0x7f1268069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.090+0000 7f1285e00640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12680678c0 0x7f1268069d80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f12780ae000 tx=0x7f12780a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:46.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.090+0000 7f127e000640 1 -- 172.21.15.98:0/655380542 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1274021020 con 0x7f1288152a50 2024-05-27T00:33:46.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.306+0000 7f128d000640 1 -- 172.21.15.98:0/655380542 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f1288147d50 con 0x7f12680678c0 2024-05-27T00:33:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f127e000640 1 -- 172.21.15.98:0/655380542 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1288147d50 con 0x7f12680678c0 2024-05-27T00:33:46.309 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 5 to deep-scrub 2024-05-27T00:33:46.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 -- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12680678c0 msgr2=0x7f1268069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12680678c0 0x7f1268069d80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f12780ae000 tx=0x7f12780a7000 comp rx=0 tx=0).stop 2024-05-27T00:33:46.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 -- 172.21.15.98:0/655380542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1288152a50 msgr2=0x7f1288147810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1288152a50 0x7f1288147810 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f1274007fd0 tx=0x7f127400c890 comp rx=0 tx=0).stop 2024-05-27T00:33:46.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 -- 172.21.15.98:0/655380542 shutdown_connections 2024-05-27T00:33:46.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1288158e60 0x7f1288157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1288152a50 0x7f1288147810 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f12680678c0 0x7f1268069d80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 --2- 172.21.15.98:0/655380542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12880cb7e0 0x7f12881598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 -- 172.21.15.98:0/655380542 >> 172.21.15.98:0/655380542 conn(0x7f1288072f90 msgr2=0x7f1288162340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:46.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 -- 172.21.15.98:0/655380542 shutdown_connections 2024-05-27T00:33:46.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.310+0000 7f128d000640 1 -- 172.21.15.98:0/655380542 wait complete. 2024-05-27T00:33:46.336 INFO:tasks.ceph:Scrubbing osd.6 2024-05-27T00:33:46.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-27T00:33:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6fa00640 1 Processor -- start 2024-05-27T00:33:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6fa00640 1 -- start start 2024-05-27T00:33:46.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6fa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d68159060 0x7f7d680c1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6fa00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d680c7b20 0x7f7d68162c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6fa00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7d6815eb70 0x7f7d680c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6fa00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7d68162180 con 0x7f7d680c7b20 2024-05-27T00:33:46.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6fa00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f7d6815f490 con 0x7f7d6815eb70 2024-05-27T00:33:46.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6fa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7d6815f770 con 0x7f7d68159060 2024-05-27T00:33:46.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6da00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7d6815eb70 0x7f7d680c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d67e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d680c7b20 0x7f7d68162c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6da00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7d6815eb70 0x7f7d680c6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:55512/0 (socket says 172.21.15.98:55512) 2024-05-27T00:33:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d67e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7d680c7b20 0x7f7d68162c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55346/0 (socket says 172.21.15.98:55346) 2024-05-27T00:33:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6da00640 1 -- 172.21.15.98:0/2568145975 learned_addr learned my addr 172.21.15.98:0/2568145975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.518+0000 7f7d6d000640 1 --2- 172.21.15.98:0/2568145975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d68159060 0x7f7d680c1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.522+0000 7f7d67e00640 1 -- 172.21.15.98:0/2568145975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d68159060 msgr2=0x7f7d680c1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.522+0000 7f7d67e00640 1 --2- 172.21.15.98:0/2568145975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d68159060 0x7f7d680c1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.522+0000 7f7d67e00640 1 -- 172.21.15.98:0/2568145975 >> v2:172.21.15.204:3300/0 conn(0x7f7d6815eb70 msgr2=0x7f7d680c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.522+0000 7f7d67e00640 1 --2- 172.21.15.98:0/2568145975 >> v2:172.21.15.204:3300/0 conn(0x7f7d6815eb70 0x7f7d680c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.522+0000 7f7d67e00640 1 -- 172.21.15.98:0/2568145975 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d68162420 con 0x7f7d680c7b20 2024-05-27T00:33:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d67e00640 1 --2- 172.21.15.98:0/2568145975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d680c7b20 0x7f7d68162c20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f7d680b2000 tx=0x7f7d5401ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbfdb166491fdb37 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d67400640 1 -- 172.21.15.98:0/2568145975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d540b1070 con 0x7f7d680c7b20 2024-05-27T00:33:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d67400640 1 -- 172.21.15.98:0/2568145975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d540ac600 con 0x7f7d680c7b20 2024-05-27T00:33:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d67400640 1 -- 172.21.15.98:0/2568145975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d540ac9d0 con 0x7f7d680c7b20 2024-05-27T00:33:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d6fa00640 1 -- 172.21.15.98:0/2568145975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d680c7b20 msgr2=0x7f7d68162c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d6fa00640 1 --2- 172.21.15.98:0/2568145975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d680c7b20 0x7f7d68162c20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f7d680b2000 tx=0x7f7d5401ce40 comp rx=0 tx=0).stop 2024-05-27T00:33:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d6fa00640 1 -- 172.21.15.98:0/2568145975 shutdown_connections 2024-05-27T00:33:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d6fa00640 1 --2- 172.21.15.98:0/2568145975 >> v2:172.21.15.204:3300/0 conn(0x7f7d6815eb70 0x7f7d680c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d6fa00640 1 --2- 172.21.15.98:0/2568145975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d680c7b20 0x7f7d68162c20 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d6fa00640 1 --2- 172.21.15.98:0/2568145975 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d68159060 0x7f7d680c1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d6fa00640 1 -- 172.21.15.98:0/2568145975 >> 172.21.15.98:0/2568145975 conn(0x7f7d680cbf00 msgr2=0x7f7d680cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:46.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d6fa00640 1 -- 172.21.15.98:0/2568145975 shutdown_connections 2024-05-27T00:33:46.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.526+0000 7f7d6fa00640 1 -- 172.21.15.98:0/2568145975 wait complete. 2024-05-27T00:33:46.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.530+0000 7f7d6fa00640 1 Processor -- start 2024-05-27T00:33:46.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.530+0000 7f7d6fa00640 1 -- start start 2024-05-27T00:33:46.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.530+0000 7f7d6fa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d680c7b20 0x7f7d680cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.530+0000 7f7d6fa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d68159060 0x7f7d68156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.530+0000 7f7d6fa00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6815eb70 0x7f7d681566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.530+0000 7f7d6fa00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6806d970 con 0x7f7d6815eb70 2024-05-27T00:33:46.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.530+0000 7f7d6fa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6806d7f0 con 0x7f7d68159060 2024-05-27T00:33:46.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.530+0000 7f7d6fa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7d6806daf0 con 0x7f7d680c7b20 2024-05-27T00:33:46.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6da00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6815eb70 0x7f7d681566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6da00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6815eb70 0x7f7d681566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55354/0 (socket says 172.21.15.98:55354) 2024-05-27T00:33:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6da00640 1 -- 172.21.15.98:0/3376046609 learned_addr learned my addr 172.21.15.98:0/3376046609 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6d000640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d680c7b20 0x7f7d680cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d67e00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d68159060 0x7f7d68156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6da00640 1 -- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d680c7b20 msgr2=0x7f7d680cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6da00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d680c7b20 0x7f7d680cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6da00640 1 -- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d68159060 msgr2=0x7f7d68156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6da00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d68159060 0x7f7d68156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6da00640 1 -- 172.21.15.98:0/3376046609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d682015c0 con 0x7f7d6815eb70 2024-05-27T00:33:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6da00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6815eb70 0x7f7d681566d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7d60009cf0 tx=0x7f7d6000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d64c00640 1 -- 172.21.15.98:0/3376046609 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6000cd20 con 0x7f7d6815eb70 2024-05-27T00:33:46.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d64c00640 1 -- 172.21.15.98:0/3376046609 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d600039a0 con 0x7f7d6815eb70 2024-05-27T00:33:46.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d64c00640 1 -- 172.21.15.98:0/3376046609 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d60003d90 con 0x7f7d6815eb70 2024-05-27T00:33:46.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.534+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d68159ae0 con 0x7f7d6815eb70 2024-05-27T00:33:46.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.538+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6815a0a0 con 0x7f7d6815eb70 2024-05-27T00:33:46.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.538+0000 7f7d64c00640 1 -- 172.21.15.98:0/3376046609 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7d60012050 con 0x7f7d6815eb70 2024-05-27T00:33:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.538+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7d6815c360 con 0x7f7d6815eb70 2024-05-27T00:33:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.538+0000 7f7d64c00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d480678c0 0x7f7d48069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.542+0000 7f7d64c00640 1 -- 172.21.15.98:0/3376046609 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f7d60085c00 con 0x7f7d6815eb70 2024-05-27T00:33:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.542+0000 7f7d64c00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] conn(0x7f7d480704c0 0x7f7d48072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.542+0000 7f7d64c00640 1 -- 172.21.15.98:0/3376046609 --> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d6001ae70 con 0x7f7d480704c0 2024-05-27T00:33:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.542+0000 7f7d6d000640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d480678c0 0x7f7d48069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.542+0000 7f7d64c00640 1 -- 172.21.15.98:0/3376046609 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f7d60051ef0 con 0x7f7d6815eb70 2024-05-27T00:33:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.542+0000 7f7d67e00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] conn(0x7f7d480704c0 0x7f7d48072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.542+0000 7f7d6d000640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d480678c0 0x7f7d48069d80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0045f0 tx=0x7f7d5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.542+0000 7f7d67e00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] conn(0x7f7d480704c0 0x7f7d48072940 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:46.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.542+0000 7f7d64c00640 1 -- 172.21.15.98:0/3376046609 <== osd.6 v2:172.21.15.204:6824/469091443 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d6001ae70 con 0x7f7d480704c0 2024-05-27T00:33:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.562+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 --> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7d68075cb0 con 0x7f7d480704c0 2024-05-27T00:33:46.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.562+0000 7f7d64c00640 1 -- 172.21.15.98:0/3376046609 <== osd.6 v2:172.21.15.204:6824/469091443 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7d68075cb0 con 0x7f7d480704c0 2024-05-27T00:33:46.565 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-27T00:33:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-27T00:33:46.565 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-27T00:33:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] conn(0x7f7d480704c0 msgr2=0x7f7d48072940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6fa00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:6824/469091443,v1:172.21.15.204:6825/469091443] conn(0x7f7d480704c0 0x7f7d48072940 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d480678c0 msgr2=0x7f7d48069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6fa00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f7d480678c0 0x7f7d48069d80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0045f0 tx=0x7f7d5c008040 comp rx=0 tx=0).stop 2024-05-27T00:33:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6815eb70 msgr2=0x7f7d681566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6fa00640 1 --2- 172.21.15.98:0/3376046609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d6815eb70 0x7f7d681566d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7d60009cf0 tx=0x7f7d6000c490 comp rx=0 tx=0).stop 2024-05-27T00:33:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6da00640 1 -- 172.21.15.98:0/3376046609 reap_dead start 2024-05-27T00:33:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 shutdown_connections 2024-05-27T00:33:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 >> 172.21.15.98:0/3376046609 conn(0x7f7d680cbf00 msgr2=0x7f7d680727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:46.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 shutdown_connections 2024-05-27T00:33:46.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.566+0000 7f7d6fa00640 1 -- 172.21.15.98:0/3376046609 wait complete. 2024-05-27T00:33:46.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-27T00:33:46.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.786+0000 7f1b6f200640 1 Processor -- start 2024-05-27T00:33:46.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.786+0000 7f1b6f200640 1 -- start start 2024-05-27T00:33:46.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.786+0000 7f1b6f200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b70154a70 0x7f1b7015b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.786+0000 7f1b6f200640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f1b700c5870 0x7f1b700c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.786+0000 7f1b6f200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b7015b660 con 0x7f1b700c3020 2024-05-27T00:33:46.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.786+0000 7f1b6f200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1b700c5280 con 0x7f1b70154a70 2024-05-27T00:33:46.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.786+0000 7f1b6f200640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f1b700c41a0 con 0x7f1b700c5870 2024-05-27T00:33:46.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.790+0000 7f1b6de00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1b70154a70 0x7f1b7015b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.790+0000 7f1b6d400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1b700c3020 0x7f1b700c7b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37360/0 (socket says 172.21.15.98:37360) 2024-05-27T00:33:46.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.790+0000 7f1b6d400640 1 -- 172.21.15.98:0/1027261084 learned_addr learned my addr 172.21.15.98:0/1027261084 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.790+0000 7f1b6e800640 1 --2- 172.21.15.98:0/1027261084 >> v2:172.21.15.204:3300/0 conn(0x7f1b700c5870 0x7f1b700c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.790+0000 7f1b6ca00640 1 -- 172.21.15.98:0/1027261084 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4082337497 0 0) 0x7f1b5c001120 con 0x7f1b700c3020 2024-05-27T00:33:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.790+0000 7f1b6ca00640 1 -- 172.21.15.98:0/1027261084 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b54003530 con 0x7f1b700c3020 2024-05-27T00:33:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.790+0000 7f1b6de00640 1 -- 172.21.15.98:0/1027261084 >> v2:172.21.15.204:3300/0 conn(0x7f1b700c5870 msgr2=0x7f1b700c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.790+0000 7f1b6de00640 1 --2- 172.21.15.98:0/1027261084 >> v2:172.21.15.204:3300/0 conn(0x7f1b700c5870 0x7f1b700c5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.790+0000 7f1b6de00640 1 -- 172.21.15.98:0/1027261084 >> v1:172.21.15.98:6789/0 conn(0x7f1b700c3020 legacy=0x7f1b700c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.790+0000 7f1b6de00640 1 -- 172.21.15.98:0/1027261084 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b700c6220 con 0x7f1b70154a70 2024-05-27T00:33:46.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6de00640 1 --2- 172.21.15.98:0/1027261084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b70154a70 0x7f1b7015b120 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1b70072de0 tx=0x7f1b6001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570c2bb0e3e1940d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:46.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6ca00640 1 -- 172.21.15.98:0/1027261084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b600b0070 con 0x7f1b70154a70 2024-05-27T00:33:46.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6ca00640 1 -- 172.21.15.98:0/1027261084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b600ab600 con 0x7f1b70154a70 2024-05-27T00:33:46.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6ca00640 1 -- 172.21.15.98:0/1027261084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b600ab9d0 con 0x7f1b70154a70 2024-05-27T00:33:46.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6f200640 1 -- 172.21.15.98:0/1027261084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b70154a70 msgr2=0x7f1b7015b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6f200640 1 --2- 172.21.15.98:0/1027261084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b70154a70 0x7f1b7015b120 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1b70072de0 tx=0x7f1b6001c480 comp rx=0 tx=0).stop 2024-05-27T00:33:46.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6f200640 1 -- 172.21.15.98:0/1027261084 shutdown_connections 2024-05-27T00:33:46.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6f200640 1 --2- 172.21.15.98:0/1027261084 >> v2:172.21.15.204:3300/0 conn(0x7f1b700c5870 0x7f1b700c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6f200640 1 --2- 172.21.15.98:0/1027261084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b70154a70 0x7f1b7015b120 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6f200640 1 -- 172.21.15.98:0/1027261084 >> 172.21.15.98:0/1027261084 conn(0x7f1b700ce000 msgr2=0x7f1b700ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:46.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6f200640 1 -- 172.21.15.98:0/1027261084 shutdown_connections 2024-05-27T00:33:46.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.794+0000 7f1b6f200640 1 -- 172.21.15.98:0/1027261084 wait complete. 2024-05-27T00:33:46.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6f200640 1 Processor -- start 2024-05-27T00:33:46.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6f200640 1 -- start start 2024-05-27T00:33:46.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6f200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b700c3020 0x7f1b7014ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b700c5870 0x7f1b7014d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b70154a70 0x7f1b7014feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6f200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b700ca6e0 con 0x7f1b700c3020 2024-05-27T00:33:46.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1b700c4f10 con 0x7f1b70154a70 2024-05-27T00:33:46.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1b700c8270 con 0x7f1b700c5870 2024-05-27T00:33:46.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b700c3020 0x7f1b7014ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b70154a70 0x7f1b7014feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6de00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b700c3020 0x7f1b7014ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55370/0 (socket says 172.21.15.98:55370) 2024-05-27T00:33:46.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6de00640 1 -- 172.21.15.98:0/3957873461 learned_addr learned my addr 172.21.15.98:0/3957873461 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:46.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6e800640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b70154a70 0x7f1b7014feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:55538/0 (socket says 172.21.15.98:55538) 2024-05-27T00:33:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6d400640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b700c5870 0x7f1b7014d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6de00640 1 -- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b700c5870 msgr2=0x7f1b7014d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6de00640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b700c5870 0x7f1b7014d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6de00640 1 -- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b70154a70 msgr2=0x7f1b7014feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6de00640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b70154a70 0x7f1b7014feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6de00640 1 -- 172.21.15.98:0/3957873461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7014b930 con 0x7f1b700c3020 2024-05-27T00:33:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6de00640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b700c3020 0x7f1b7014ab50 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1b60019090 tx=0x7f1b60019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:46.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b66000640 1 -- 172.21.15.98:0/3957873461 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b600b0040 con 0x7f1b700c3020 2024-05-27T00:33:46.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b66000640 1 -- 172.21.15.98:0/3957873461 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b600ba070 con 0x7f1b700c3020 2024-05-27T00:33:46.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b66000640 1 -- 172.21.15.98:0/3957873461 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b600b4800 con 0x7f1b700c3020 2024-05-27T00:33:46.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6f200640 1 -- 172.21.15.98:0/3957873461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b700c9a50 con 0x7f1b700c3020 2024-05-27T00:33:46.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.798+0000 7f1b6f200640 1 -- 172.21.15.98:0/3957873461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b7014b370 con 0x7f1b700c3020 2024-05-27T00:33:46.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.802+0000 7f1b6f200640 1 -- 172.21.15.98:0/3957873461 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b7004f7e0 con 0x7f1b700c3020 2024-05-27T00:33:46.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.802+0000 7f1b66000640 1 -- 172.21.15.98:0/3957873461 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1b600b3070 con 0x7f1b700c3020 2024-05-27T00:33:46.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.802+0000 7f1b66000640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b500678c0 0x7f1b50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:46.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.802+0000 7f1b66000640 1 -- 172.21.15.98:0/3957873461 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f1b6011f880 con 0x7f1b700c3020 2024-05-27T00:33:46.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.802+0000 7f1b6d400640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b500678c0 0x7f1b50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:46.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.806+0000 7f1b6d400640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b500678c0 0x7f1b50069d80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1b7014b090 tx=0x7f1b5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:46.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.810+0000 7f1b66000640 1 -- 172.21.15.98:0/3957873461 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1b60016070 con 0x7f1b700c3020 2024-05-27T00:33:46.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.982+0000 7f1b6f200640 1 -- 172.21.15.98:0/3957873461 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f1b7006f870 con 0x7f1b500678c0 2024-05-27T00:33:46.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.982+0000 7f1b66000640 1 -- 172.21.15.98:0/3957873461 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1b7006f870 con 0x7f1b500678c0 2024-05-27T00:33:46.984 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 6 to deep-scrub 2024-05-27T00:33:46.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 -- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b500678c0 msgr2=0x7f1b50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b500678c0 0x7f1b50069d80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1b7014b090 tx=0x7f1b5c009040 comp rx=0 tx=0).stop 2024-05-27T00:33:46.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 -- 172.21.15.98:0/3957873461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b700c3020 msgr2=0x7f1b7014ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:46.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b700c3020 0x7f1b7014ab50 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1b60019090 tx=0x7f1b60019220 comp rx=0 tx=0).stop 2024-05-27T00:33:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 -- 172.21.15.98:0/3957873461 shutdown_connections 2024-05-27T00:33:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f1b500678c0 0x7f1b50069d80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b70154a70 0x7f1b7014feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b700c5870 0x7f1b7014d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 --2- 172.21.15.98:0/3957873461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b700c3020 0x7f1b7014ab50 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 -- 172.21.15.98:0/3957873461 >> 172.21.15.98:0/3957873461 conn(0x7f1b700ce000 msgr2=0x7f1b70158450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 -- 172.21.15.98:0/3957873461 shutdown_connections 2024-05-27T00:33:46.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:46.986+0000 7f1b6f200640 1 -- 172.21.15.98:0/3957873461 wait complete. 2024-05-27T00:33:47.010 INFO:tasks.ceph:Scrubbing osd.7 2024-05-27T00:33:47.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-27T00:33:47.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.198+0000 7f359ea00640 1 Processor -- start 2024-05-27T00:33:47.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.198+0000 7f359ea00640 1 -- start start 2024-05-27T00:33:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f359ea00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35980c9450 0x7f359806ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f359ea00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3598067fc0 con 0x7f359806fd60 2024-05-27T00:33:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f359ea00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35980c7a40 con 0x7f35981591d0 2024-05-27T00:33:47.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f359ea00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f359806f8d0 con 0x7f35980c9450 2024-05-27T00:33:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f359ca00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f359806fd60 0x7f35980701f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37372/0 (socket says 172.21.15.98:37372) 2024-05-27T00:33:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f3597e00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f35981591d0 0x7f3598073110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.98:55880/0 (socket says 172.21.15.98:55880) 2024-05-27T00:33:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f3597400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35980c9450 0x7f359806ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f359ca00640 1 -- 172.21.15.98:0/484058137 learned_addr learned my addr 172.21.15.98:0/484058137 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f3596a00640 1 -- 172.21.15.98:0/484058137 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1691521657 0 0) 0x7f3598067fc0 con 0x7f359806fd60 2024-05-27T00:33:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f3596a00640 1 -- 172.21.15.98:0/484058137 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f357c003530 con 0x7f359806fd60 2024-05-27T00:33:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f3596a00640 1 -- 172.21.15.98:0/484058137 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1839448853 0 0) 0x7f35980c7a40 con 0x7f35981591d0 2024-05-27T00:33:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f3596a00640 1 -- 172.21.15.98:0/484058137 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3598067fc0 con 0x7f35981591d0 2024-05-27T00:33:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f3597400640 1 -- 172.21.15.98:0/484058137 >> v1:172.21.15.204:6789/0 conn(0x7f35981591d0 legacy=0x7f3598073110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f3596a00640 1 -- 172.21.15.98:0/484058137 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 113905730 0 0) 0x7f357c003530 con 0x7f359806fd60 2024-05-27T00:33:47.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f3597400640 1 -- 172.21.15.98:0/484058137 >> v1:172.21.15.98:6789/0 conn(0x7f359806fd60 legacy=0x7f35980701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.202+0000 7f3597400640 1 -- 172.21.15.98:0/484058137 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35980c7d20 con 0x7f35980c9450 2024-05-27T00:33:47.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.206+0000 7f3597400640 1 --2- 172.21.15.98:0/484058137 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35980c9450 0x7f359806ba00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f35840097f0 tx=0x7f358401ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfcf6f0097c66b3b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.206+0000 7f3596a00640 1 -- 172.21.15.98:0/484058137 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35840b5070 con 0x7f35980c9450 2024-05-27T00:33:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.206+0000 7f3596a00640 1 -- 172.21.15.98:0/484058137 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35840a8670 con 0x7f35980c9450 2024-05-27T00:33:47.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.206+0000 7f3596a00640 1 -- 172.21.15.98:0/484058137 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3590001a00 con 0x7f35980c9450 2024-05-27T00:33:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.206+0000 7f359ea00640 1 -- 172.21.15.98:0/484058137 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35980c9450 msgr2=0x7f359806ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.206+0000 7f359ea00640 1 --2- 172.21.15.98:0/484058137 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35980c9450 0x7f359806ba00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f35840097f0 tx=0x7f358401ebf0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.210+0000 7f359ea00640 1 -- 172.21.15.98:0/484058137 shutdown_connections 2024-05-27T00:33:47.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.210+0000 7f359ea00640 1 --2- 172.21.15.98:0/484058137 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35980c9450 0x7f359806ba00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.210+0000 7f359ea00640 1 -- 172.21.15.98:0/484058137 >> 172.21.15.98:0/484058137 conn(0x7f359806e6b0 msgr2=0x7f35980ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.210+0000 7f359ea00640 1 -- 172.21.15.98:0/484058137 shutdown_connections 2024-05-27T00:33:47.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.210+0000 7f359ea00640 1 -- 172.21.15.98:0/484058137 wait complete. 2024-05-27T00:33:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.210+0000 7f359ea00640 1 Processor -- start 2024-05-27T00:33:47.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.210+0000 7f359ea00640 1 -- start start 2024-05-27T00:33:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f359806fd60 0x7f359815d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ea00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35980c9450 0x7f35980cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ea00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35981591d0 0x7f3598154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ea00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f359806cd40 con 0x7f35981591d0 2024-05-27T00:33:47.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ea00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f359806ca60 con 0x7f359806fd60 2024-05-27T00:33:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ea00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f35980c3fd0 con 0x7f35980c9450 2024-05-27T00:33:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f3597e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f359806fd60 0x7f359815d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f3597e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f359806fd60 0x7f359815d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:55540/0 (socket says 172.21.15.98:55540) 2024-05-27T00:33:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f3597e00640 1 -- 172.21.15.98:0/27829914 learned_addr learned my addr 172.21.15.98:0/27829914 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ca00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35981591d0 0x7f3598154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f3597400640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35980c9450 0x7f35980cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f3597e00640 1 -- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35980c9450 msgr2=0x7f35980cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f3597e00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35980c9450 0x7f35980cc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f3597e00640 1 -- 172.21.15.98:0/27829914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35981591d0 msgr2=0x7f3598154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f3597e00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35981591d0 0x7f3598154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f3597e00640 1 -- 172.21.15.98:0/27829914 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35981f4fe0 con 0x7f359806fd60 2024-05-27T00:33:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ca00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35981591d0 0x7f3598154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:33:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f3597e00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f359806fd60 0x7f359815d830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f358800c480 tx=0x7f358800c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f358fe00640 1 -- 172.21.15.98:0/27829914 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588018070 con 0x7f359806fd60 2024-05-27T00:33:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f358fe00640 1 -- 172.21.15.98:0/27829914 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35880074a0 con 0x7f359806fd60 2024-05-27T00:33:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f358fe00640 1 -- 172.21.15.98:0/27829914 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588004a30 con 0x7f359806fd60 2024-05-27T00:33:47.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35981f52c0 con 0x7f359806fd60 2024-05-27T00:33:47.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35981f5880 con 0x7f359806fd60 2024-05-27T00:33:47.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.214+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3598075cb0 con 0x7f359806fd60 2024-05-27T00:33:47.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.218+0000 7f358fe00640 1 -- 172.21.15.98:0/27829914 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3588004bd0 con 0x7f359806fd60 2024-05-27T00:33:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.218+0000 7f358fe00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35780678c0 0x7f3578069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.218+0000 7f358fe00640 1 -- 172.21.15.98:0/27829914 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f35880859a0 con 0x7f359806fd60 2024-05-27T00:33:47.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.218+0000 7f358fe00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] conn(0x7f35780704b0 0x7f3578072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.218+0000 7f358fe00640 1 -- 172.21.15.98:0/27829914 --> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f358801de50 con 0x7f35780704b0 2024-05-27T00:33:47.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.218+0000 7f3597400640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35780678c0 0x7f3578069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.218+0000 7f358fe00640 1 -- 172.21.15.98:0/27829914 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f3588085d50 con 0x7f359806fd60 2024-05-27T00:33:47.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.222+0000 7f359ca00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] conn(0x7f35780704b0 0x7f3578072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.222+0000 7f3597400640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35780678c0 0x7f3578069d80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f35840b2000 tx=0x7f35840aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:47.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.222+0000 7f359ca00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] conn(0x7f35780704b0 0x7f3578072930 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:47.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.222+0000 7f358fe00640 1 -- 172.21.15.98:0/27829914 <== osd.7 v2:172.21.15.204:6816/874852314 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f358801de50 con 0x7f35780704b0 2024-05-27T00:33:47.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 --> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f35981f5c60 con 0x7f35780704b0 2024-05-27T00:33:47.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f358fe00640 1 -- 172.21.15.98:0/27829914 <== osd.7 v2:172.21.15.204:6816/874852314 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f35981f5c60 con 0x7f35780704b0 2024-05-27T00:33:47.244 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-05-27T00:33:47.244 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-27T00:33:47.245 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-05-27T00:33:47.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] conn(0x7f35780704b0 msgr2=0x7f3578072930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:6816/874852314,v1:172.21.15.204:6817/874852314] conn(0x7f35780704b0 0x7f3578072930 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35780678c0 msgr2=0x7f3578069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f35780678c0 0x7f3578069d80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f35840b2000 tx=0x7f35840aa070 comp rx=0 tx=0).stop 2024-05-27T00:33:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f359806fd60 msgr2=0x7f359815d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 --2- 172.21.15.98:0/27829914 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f359806fd60 0x7f359815d830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f358800c480 tx=0x7f358800c940 comp rx=0 tx=0).stop 2024-05-27T00:33:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ca00640 1 -- 172.21.15.98:0/27829914 reap_dead start 2024-05-27T00:33:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 shutdown_connections 2024-05-27T00:33:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 >> 172.21.15.98:0/27829914 conn(0x7f359806e6b0 msgr2=0x7f3598155810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:47.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 shutdown_connections 2024-05-27T00:33:47.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.242+0000 7f359ea00640 1 -- 172.21.15.98:0/27829914 wait complete. 2024-05-27T00:33:47.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-27T00:33:47.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.450+0000 7f1411c00640 1 Processor -- start 2024-05-27T00:33:47.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.450+0000 7f1411c00640 1 -- start start 2024-05-27T00:33:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.450+0000 7f1411c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f140c158490 0x7f140c06e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.450+0000 7f1411c00640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f140c06edd0 0x7f140c15c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.450+0000 7f1411c00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f140c158920 con 0x7f140c152770 2024-05-27T00:33:47.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.450+0000 7f1411c00640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f140c159480 con 0x7f140c06edd0 2024-05-27T00:33:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.450+0000 7f1411c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f140c158be0 con 0x7f140c158490 2024-05-27T00:33:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.450+0000 7f140aa00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f140c152770 0x7f140c0c7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37378/0 (socket says 172.21.15.98:37378) 2024-05-27T00:33:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.450+0000 7f140aa00640 1 -- 172.21.15.98:0/1411586892 learned_addr learned my addr 172.21.15.98:0/1411586892 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140b400640 1 --2- 172.21.15.98:0/1411586892 >> v2:172.21.15.204:3300/0 conn(0x7f140c06edd0 0x7f140c15c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f1403e00640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f140c158490 0x7f140c06e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 30925598 0 0) 0x7f140c158920 con 0x7f140c152770 2024-05-27T00:33:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13f0003530 con 0x7f140c152770 2024-05-27T00:33:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1437918270 0 0) 0x7f13f0003530 con 0x7f140c152770 2024-05-27T00:33:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f140c158490 msgr2=0x7f140c06e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f140c158490 0x7f140c06e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 >> v2:172.21.15.204:3300/0 conn(0x7f140c06edd0 msgr2=0x7f140c15c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:47.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 --2- 172.21.15.98:0/1411586892 >> v2:172.21.15.204:3300/0 conn(0x7f140c06edd0 0x7f140c15c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f140c158ec0 con 0x7f140c152770 2024-05-27T00:33:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f13fc001680 con 0x7f140c152770 2024-05-27T00:33:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 >> v1:172.21.15.98:6789/0 conn(0x7f140c152770 legacy=0x7f140c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f00039c0 0x7f13f0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13f0004810 0x7f13f0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13f00051d0 0x7f13f000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f140c159480 con 0x7f13f00039c0 2024-05-27T00:33:47.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f140c158be0 con 0x7f13f0004810 2024-05-27T00:33:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f13f00045b0 con 0x7f13f00051d0 2024-05-27T00:33:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140aa00640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f00039c0 0x7f13f0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140b400640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13f00051d0 0x7f13f000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f1403e00640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13f0004810 0x7f13f0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140aa00640 1 -- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13f00051d0 msgr2=0x7f13f000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:47.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140aa00640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13f00051d0 0x7f13f000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140aa00640 1 -- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13f0004810 msgr2=0x7f13f0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140aa00640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13f0004810 0x7f13f0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.454+0000 7f140aa00640 1 -- 172.21.15.98:0/1411586892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f0003530 con 0x7f13f00039c0 2024-05-27T00:33:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f140aa00640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f00039c0 0x7f13f0003de0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00c270 tx=0x7f13fc01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18cb3f442c20b263 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13fc007d60 con 0x7f13f00039c0 2024-05-27T00:33:47.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13fc0a9020 con 0x7f13f00039c0 2024-05-27T00:33:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f140a000640 1 -- 172.21.15.98:0/1411586892 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13fc005ae0 con 0x7f13f00039c0 2024-05-27T00:33:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f1411c00640 1 -- 172.21.15.98:0/1411586892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f00039c0 msgr2=0x7f13f0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f1411c00640 1 --2- 172.21.15.98:0/1411586892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f00039c0 0x7f13f0003de0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00c270 tx=0x7f13fc01ccc0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f140b400640 1 -- 172.21.15.98:0/1411586892 reap_dead start 2024-05-27T00:33:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f1411c00640 1 -- 172.21.15.98:0/1411586892 shutdown_connections 2024-05-27T00:33:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f1411c00640 1 -- 172.21.15.98:0/1411586892 >> 172.21.15.98:0/1411586892 conn(0x7f140c0cc200 msgr2=0x7f140c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:47.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f1411c00640 1 -- 172.21.15.98:0/1411586892 shutdown_connections 2024-05-27T00:33:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.458+0000 7f1411c00640 1 -- 172.21.15.98:0/1411586892 wait complete. 2024-05-27T00:33:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1411c00640 1 Processor -- start 2024-05-27T00:33:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1411c00640 1 -- start start 2024-05-27T00:33:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1411c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13f00039c0 0x7f140c15b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1411c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f140c151030 0x7f140c1f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1411c00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f140c151490 0x7f140c1f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1411c00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f140c070fa0 con 0x7f140c151490 2024-05-27T00:33:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1411c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f140c152b50 con 0x7f13f00039c0 2024-05-27T00:33:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1411c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f140c163620 con 0x7f140c151030 2024-05-27T00:33:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f140b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f140c151490 0x7f140c1f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f140b400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f140c151490 0x7f140c1f2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55396/0 (socket says 172.21.15.98:55396) 2024-05-27T00:33:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f140b400640 1 -- 172.21.15.98:0/148454704 learned_addr learned my addr 172.21.15.98:0/148454704 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f140aa00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13f00039c0 0x7f140c15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1403e00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f140c151030 0x7f140c1f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f140b400640 1 -- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f140c151030 msgr2=0x7f140c1f2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f140b400640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f140c151030 0x7f140c1f2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f140b400640 1 -- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13f00039c0 msgr2=0x7f140c15b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f140b400640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13f00039c0 0x7f140c15b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f140b400640 1 -- 172.21.15.98:0/148454704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f140c15dcd0 con 0x7f140c151490 2024-05-27T00:33:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f140b400640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f140c151490 0x7f140c1f2980 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f140c070b60 tx=0x7f1404011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1402a00640 1 -- 172.21.15.98:0/148454704 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140400fbf0 con 0x7f140c151490 2024-05-27T00:33:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1402a00640 1 -- 172.21.15.98:0/148454704 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140400fd90 con 0x7f140c151490 2024-05-27T00:33:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1411c00640 1 -- 172.21.15.98:0/148454704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f140c15dfb0 con 0x7f140c151490 2024-05-27T00:33:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.462+0000 7f1402a00640 1 -- 172.21.15.98:0/148454704 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1404015670 con 0x7f140c151490 2024-05-27T00:33:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.466+0000 7f1411c00640 1 -- 172.21.15.98:0/148454704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f140c15e570 con 0x7f140c151490 2024-05-27T00:33:47.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.466+0000 7f1411c00640 1 -- 172.21.15.98:0/148454704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f140c075cb0 con 0x7f140c151490 2024-05-27T00:33:47.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.466+0000 7f1402a00640 1 -- 172.21.15.98:0/148454704 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1404015810 con 0x7f140c151490 2024-05-27T00:33:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.466+0000 7f1402a00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13e80678c0 0x7f13e8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.466+0000 7f1402a00640 1 -- 172.21.15.98:0/148454704 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f1404086ed0 con 0x7f140c151490 2024-05-27T00:33:47.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.470+0000 7f140aa00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13e80678c0 0x7f13e8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.470+0000 7f140aa00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13e80678c0 0x7f13e8069d80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00c270 tx=0x7f13fc001ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:47.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.474+0000 7f1402a00640 1 -- 172.21.15.98:0/148454704 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f1404019030 con 0x7f140c151490 2024-05-27T00:33:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.734+0000 7f1411c00640 1 -- 172.21.15.98:0/148454704 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f140c158920 con 0x7f13e80678c0 2024-05-27T00:33:47.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.738+0000 7f1402a00640 1 -- 172.21.15.98:0/148454704 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f140c158920 con 0x7f13e80678c0 2024-05-27T00:33:47.738 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 7 to deep-scrub 2024-05-27T00:33:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.738+0000 7f1411c00640 1 -- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13e80678c0 msgr2=0x7f13e8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.738+0000 7f1411c00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13e80678c0 0x7f13e8069d80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00c270 tx=0x7f13fc001ac0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.738+0000 7f1411c00640 1 -- 172.21.15.98:0/148454704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f140c151490 msgr2=0x7f140c1f2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.738+0000 7f1411c00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f140c151490 0x7f140c1f2980 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f140c070b60 tx=0x7f1404011f10 comp rx=0 tx=0).stop 2024-05-27T00:33:47.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.742+0000 7f1411c00640 1 -- 172.21.15.98:0/148454704 shutdown_connections 2024-05-27T00:33:47.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.742+0000 7f1411c00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f140c151490 0x7f140c1f2980 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.742+0000 7f1411c00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f140c151030 0x7f140c1f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.742+0000 7f1411c00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13e80678c0 0x7f13e8069d80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.742+0000 7f1411c00640 1 --2- 172.21.15.98:0/148454704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13f00039c0 0x7f140c15b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.742+0000 7f1411c00640 1 -- 172.21.15.98:0/148454704 >> 172.21.15.98:0/148454704 conn(0x7f140c0cc200 msgr2=0x7f140c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.742+0000 7f1411c00640 1 -- 172.21.15.98:0/148454704 shutdown_connections 2024-05-27T00:33:47.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.742+0000 7f1411c00640 1 -- 172.21.15.98:0/148454704 wait complete. 2024-05-27T00:33:47.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:33:47.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.962+0000 7faaa0000640 1 Processor -- start 2024-05-27T00:33:47.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.962+0000 7faaa0000640 1 -- start start 2024-05-27T00:33:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.962+0000 7faaa0000640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7faa98154e90 0x7faa9815ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.962+0000 7faaa0000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa9815a950 con 0x7faa98070c00 2024-05-27T00:33:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.962+0000 7faaa0000640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa9815e060 con 0x7faa98070fe0 2024-05-27T00:33:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.962+0000 7faaa0000640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7faa9815e340 con 0x7faa98154e90 2024-05-27T00:33:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.962+0000 7faa9d600640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7faa98070c00 0x7faa980c8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37384/0 (socket says 172.21.15.98:37384) 2024-05-27T00:33:47.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.962+0000 7faa9d600640 1 -- 172.21.15.98:0/3503091994 learned_addr learned my addr 172.21.15.98:0/3503091994 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:33:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa9e000640 1 --2- 172.21.15.98:0/3503091994 >> v2:172.21.15.204:3300/0 conn(0x7faa98154e90 0x7faa9815ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1212792875 0 0) 0x7faa9815a950 con 0x7faa98070c00 2024-05-27T00:33:47.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa80003530 con 0x7faa98070c00 2024-05-27T00:33:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2335977345 0 0) 0x7faa80003530 con 0x7faa98070c00 2024-05-27T00:33:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 >> v2:172.21.15.204:3300/0 conn(0x7faa98154e90 msgr2=0x7faa9815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 --2- 172.21.15.98:0/3503091994 >> v2:172.21.15.204:3300/0 conn(0x7faa98154e90 0x7faa9815ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 >> v1:172.21.15.204:6789/0 conn(0x7faa98070fe0 legacy=0x7faa98154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9815e620 con 0x7faa98070c00 2024-05-27T00:33:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7faa8c001680 con 0x7faa98070c00 2024-05-27T00:33:47.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 >> v1:172.21.15.98:6789/0 conn(0x7faa98070c00 legacy=0x7faa980c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa800039f0 0x7faa80003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa80004810 0x7faa80004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa800051d0 0x7faa8000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa9815e340 con 0x7faa800051d0 2024-05-27T00:33:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa8000c080 con 0x7faa800039f0 2024-05-27T00:33:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faa8000c280 con 0x7faa80004810 2024-05-27T00:33:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa9e000640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa800051d0 0x7faa8000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa9cc00640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa80004810 0x7faa80004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa9d600640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa800039f0 0x7faa80003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa9e000640 1 -- 172.21.15.98:0/3503091994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa80004810 msgr2=0x7faa80004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa9e000640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa80004810 0x7faa80004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa9e000640 1 -- 172.21.15.98:0/3503091994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa800039f0 msgr2=0x7faa80003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:33:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa9e000640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa800039f0 0x7faa80003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.966+0000 7faa9e000640 1 -- 172.21.15.98:0/3503091994 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa80003530 con 0x7faa800051d0 2024-05-27T00:33:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faa9e000640 1 -- 172.21.15.98:0/3503091994 reap_dead start 2024-05-27T00:33:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faa9e000640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa800051d0 0x7faa8000b9d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7faa98155290 tx=0x7faa9001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0a3791e9673a73 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa900b2070 con 0x7faa800051d0 2024-05-27T00:33:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa900a5c70 con 0x7faa800051d0 2024-05-27T00:33:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faa97e00640 1 -- 172.21.15.98:0/3503091994 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa90009c10 con 0x7faa800051d0 2024-05-27T00:33:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faaa0000640 1 -- 172.21.15.98:0/3503091994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa800051d0 msgr2=0x7faa8000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faaa0000640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa800051d0 0x7faa8000b9d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7faa98155290 tx=0x7faa9001cc60 comp rx=0 tx=0).stop 2024-05-27T00:33:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faaa0000640 1 -- 172.21.15.98:0/3503091994 shutdown_connections 2024-05-27T00:33:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faaa0000640 1 --2- 172.21.15.98:0/3503091994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa800051d0 0x7faa8000b9d0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faaa0000640 1 -- 172.21.15.98:0/3503091994 >> 172.21.15.98:0/3503091994 conn(0x7faa98072f90 msgr2=0x7faa980733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faaa0000640 1 -- 172.21.15.98:0/3503091994 shutdown_connections 2024-05-27T00:33:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faaa0000640 1 -- 172.21.15.98:0/3503091994 wait complete. 2024-05-27T00:33:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faaa0000640 1 Processor -- start 2024-05-27T00:33:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.970+0000 7faaa0000640 1 -- start start 2024-05-27T00:33:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faaa0000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa800051d0 0x7faa980c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faaa0000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa980c57d0 0x7faa980c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faaa0000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa980c6cb0 0x7faa9814cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faaa0000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa9815ebe0 con 0x7faa980c57d0 2024-05-27T00:33:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faaa0000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa9815dec0 con 0x7faa800051d0 2024-05-27T00:33:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faaa0000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faa980cb570 con 0x7faa980c6cb0 2024-05-27T00:33:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9cc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa980c57d0 0x7faa980c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa800051d0 0x7faa980c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa980c6cb0 0x7faa9814cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9cc00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa980c57d0 0x7faa980c6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55424/0 (socket says 172.21.15.98:55424) 2024-05-27T00:33:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9cc00640 1 -- 172.21.15.98:0/4248866840 learned_addr learned my addr 172.21.15.98:0/4248866840 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:33:47.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa800051d0 0x7faa980c4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.98:55598/0 (socket says 172.21.15.98:55598) 2024-05-27T00:33:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa980c6cb0 0x7faa9814cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.98:55734/0 (socket says 172.21.15.98:55734) 2024-05-27T00:33:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9cc00640 1 -- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa980c6cb0 msgr2=0x7faa9814cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9cc00640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa980c6cb0 0x7faa9814cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9cc00640 1 -- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa800051d0 msgr2=0x7faa980c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9cc00640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa800051d0 0x7faa980c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9cc00640 1 -- 172.21.15.98:0/4248866840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa981557e0 con 0x7faa980c57d0 2024-05-27T00:33:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa9cc00640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa980c57d0 0x7faa980c6770 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7faa8400f5c0 tx=0x7faa8400fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa95600640 1 -- 172.21.15.98:0/4248866840 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa840052e0 con 0x7faa980c57d0 2024-05-27T00:33:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa95600640 1 -- 172.21.15.98:0/4248866840 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa84018070 con 0x7faa980c57d0 2024-05-27T00:33:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa95600640 1 -- 172.21.15.98:0/4248866840 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa84003840 con 0x7faa980c57d0 2024-05-27T00:33:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faaa0000640 1 -- 172.21.15.98:0/4248866840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa981f44e0 con 0x7faa980c57d0 2024-05-27T00:33:47.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faaa0000640 1 -- 172.21.15.98:0/4248866840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa981f3720 con 0x7faa980c57d0 2024-05-27T00:33:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faaa0000640 1 -- 172.21.15.98:0/4248866840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa98075cb0 con 0x7faa980c57d0 2024-05-27T00:33:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.974+0000 7faa95600640 1 -- 172.21.15.98:0/4248866840 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faa840039e0 con 0x7faa980c57d0 2024-05-27T00:33:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.978+0000 7faa95600640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa780678c0 0x7faa78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:33:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.978+0000 7faa95600640 1 -- 172.21.15.98:0/4248866840 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7faa84085e10 con 0x7faa980c57d0 2024-05-27T00:33:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.978+0000 7faa9d600640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa780678c0 0x7faa78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:33:47.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.978+0000 7faa9d600640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa780678c0 0x7faa78069d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faa8c001060 tx=0x7faa8c007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:33:47.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:47.982+0000 7faa95600640 1 -- 172.21.15.98:0/4248866840 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7faa84089030 con 0x7faa980c57d0 2024-05-27T00:33:48.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.230+0000 7faaa0000640 1 -- 172.21.15.98:0/4248866840 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faa980c6080 con 0x7faa780678c0 2024-05-27T00:33:48.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.242+0000 7faa95600640 1 -- 172.21.15.98:0/4248866840 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+255298 (secure 0 0 0) 0x7faa8c011a10 con 0x7faa780678c0 2024-05-27T00:33:48.245 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:33:48.246 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:33:48.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 -- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa780678c0 msgr2=0x7faa78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:48.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa780678c0 0x7faa78069d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faa8c001060 tx=0x7faa8c007040 comp rx=0 tx=0).stop 2024-05-27T00:33:48.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 -- 172.21.15.98:0/4248866840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa980c57d0 msgr2=0x7faa980c6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:33:48.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa980c57d0 0x7faa980c6770 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7faa8400f5c0 tx=0x7faa8400fa80 comp rx=0 tx=0).stop 2024-05-27T00:33:48.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 -- 172.21.15.98:0/4248866840 shutdown_connections 2024-05-27T00:33:48.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa980c6cb0 0x7faa9814cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa980c57d0 0x7faa980c6770 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7faa780678c0 0x7faa78069d80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 --2- 172.21.15.98:0/4248866840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa800051d0 0x7faa980c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:33:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 -- 172.21.15.98:0/4248866840 >> 172.21.15.98:0/4248866840 conn(0x7faa98072f90 msgr2=0x7faa980c8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:33:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 -- 172.21.15.98:0/4248866840 shutdown_connections 2024-05-27T00:33:48.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:33:48.246+0000 7faaa0000640 1 -- 172.21.15.98:0/4248866840 wait complete. 2024-05-27T00:33:48.276 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":599,"stamp":"2024-05-27T00:33:47.545203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1731784342,"num_objects":41396,"num_object_clones":18317,"num_object_copies":82792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41396,"num_whiteouts":12453,"num_read":13893,"num_read_kb":6221409,"num_write":127388,"num_write_kb":3742405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1869,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6956090,"num_omap_keys":12268,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133217,"ondisk_log_size":133217,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4626680,"kb_used_data":3478200,"kb_used_omap":12,"kb_used_meta":1148403,"kb_avail":750348040,"statfs":{"total":773094113280,"available":768356392960,"internally_reserved":0,"allocated":3561676800,"data_stored":3467568900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1175965234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-217375288,"num_objects":421,"num_object_clones":66,"num_object_copies":842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":7,"num_read":5,"num_read_kb":10,"num_write":1204,"num_write_kb":47060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6956090,"num_omap_keys":12268,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004342"},"pg_stats":[{"pgid":"3.3f","version":"48'1531","reported_seq":3407,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.158904+0000","last_change":"2024-05-27T00:17:54.336701+0000","last_active":"2024-05-27T00:33:40.158904+0000","last_peered":"2024-05-27T00:33:40.158904+0000","last_clean":"2024-05-27T00:33:40.158904+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:33:40.158904+0000","last_undegraded":"2024-05-27T00:33:40.158904+0000","last_fullsized":"2024-05-27T00:33:40.158904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:39.250119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027619028,"stat_sum":{"num_bytes":17381879,"num_objects":576,"num_object_clones":268,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":195,"num_read":49,"num_read_kb":31548,"num_write":1375,"num_write_kb":18952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"48'1514","reported_seq":3413,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.168767+0000","last_change":"2024-05-27T00:17:54.490184+0000","last_active":"2024-05-27T00:33:40.168767+0000","last_peered":"2024-05-27T00:33:40.168767+0000","last_clean":"2024-05-27T00:33:40.168767+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:33:40.168767+0000","last_undegraded":"2024-05-27T00:33:40.168767+0000","last_fullsized":"2024-05-27T00:33:40.168767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1514,"log_dups_size":0,"ondisk_log_size":1514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015469564,"stat_sum":{"num_bytes":18134215,"num_objects":549,"num_object_clones":256,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":169,"num_read":59,"num_read_kb":35851,"num_write":1385,"num_write_kb":20077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"48'2159","reported_seq":4037,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2024-05-27T00:33:46.744945+0000","last_change":"2024-05-27T00:33:45.942952+0000","last_active":"2024-05-27T00:33:46.744945+0000","last_peered":"2024-05-27T00:33:46.744945+0000","last_clean":"2024-05-27T00:33:46.744945+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:33:46.744945+0000","last_undegraded":"2024-05-27T00:33:46.744945+0000","last_fullsized":"2024-05-27T00:33:46.744945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":561,"log_size":2159,"log_dups_size":0,"ondisk_log_size":2159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018507847000000001,"stat_sum":{"num_bytes":15323750,"num_objects":636,"num_object_clones":298,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":209,"num_read":53,"num_read_kb":23374,"num_write":1458,"num_write_kb":25883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"48'1633","reported_seq":3498,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.154980+0000","last_change":"2024-05-27T00:17:54.510484+0000","last_active":"2024-05-27T00:33:40.154980+0000","last_peered":"2024-05-27T00:33:40.154980+0000","last_clean":"2024-05-27T00:33:40.154980+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:33:40.154980+0000","last_undegraded":"2024-05-27T00:33:40.154980+0000","last_fullsized":"2024-05-27T00:33:40.154980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1633,"log_dups_size":0,"ondisk_log_size":1633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034338709000000002,"stat_sum":{"num_bytes":10215698,"num_objects":623,"num_object_clones":288,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":200,"num_read":56,"num_read_kb":12442,"num_write":1464,"num_write_kb":16944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"48'1713","reported_seq":3554,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.156866+0000","last_change":"2024-05-27T00:17:54.331579+0000","last_active":"2024-05-27T00:33:40.156866+0000","last_peered":"2024-05-27T00:33:40.156866+0000","last_clean":"2024-05-27T00:33:40.156866+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:33:40.156866+0000","last_undegraded":"2024-05-27T00:33:40.156866+0000","last_fullsized":"2024-05-27T00:33:40.156866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1713,"log_dups_size":0,"ondisk_log_size":1713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:35:37.983576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026116615999999999,"stat_sum":{"num_bytes":20149863,"num_objects":672,"num_object_clones":311,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":219,"num_read":58,"num_read_kb":29919,"num_write":1539,"num_write_kb":21988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"48'1470","reported_seq":3347,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.152579+0000","last_change":"2024-05-27T00:17:54.507830+0000","last_active":"2024-05-27T00:33:40.152579+0000","last_peered":"2024-05-27T00:33:40.152579+0000","last_clean":"2024-05-27T00:33:40.152579+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:33:40.152579+0000","last_undegraded":"2024-05-27T00:33:40.152579+0000","last_fullsized":"2024-05-27T00:33:40.152579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1470,"log_dups_size":0,"ondisk_log_size":1470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032595461999999999,"stat_sum":{"num_bytes":14563572,"num_objects":549,"num_object_clones":256,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":180,"num_read":38,"num_read_kb":8264,"num_write":1305,"num_write_kb":15921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"48'1585","reported_seq":3453,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.155879+0000","last_change":"2024-05-27T00:17:54.335843+0000","last_active":"2024-05-27T00:33:40.155879+0000","last_peered":"2024-05-27T00:33:40.155879+0000","last_clean":"2024-05-27T00:33:40.155879+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:33:40.155879+0000","last_undegraded":"2024-05-27T00:33:40.155879+0000","last_fullsized":"2024-05-27T00:33:40.155879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1585,"log_dups_size":0,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:27:21.913063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029344160000000001,"stat_sum":{"num_bytes":19343039,"num_objects":604,"num_object_clones":277,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":190,"num_read":51,"num_read_kb":28747,"num_write":1439,"num_write_kb":22131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"48'1530","reported_seq":3402,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.937471+0000","last_change":"2024-05-27T00:17:54.485261+0000","last_active":"2024-05-27T00:33:39.937471+0000","last_peered":"2024-05-27T00:33:39.937471+0000","last_clean":"2024-05-27T00:33:39.937471+0000","last_became_active":"2024-05-27T00:16:06.280009+0000","last_became_peered":"2024-05-27T00:16:06.280009+0000","last_unstale":"2024-05-27T00:33:39.937471+0000","last_undegraded":"2024-05-27T00:33:39.937471+0000","last_fullsized":"2024-05-27T00:33:39.937471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012011287000000001,"stat_sum":{"num_bytes":35917582,"num_objects":581,"num_object_clones":269,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":179,"num_read":52,"num_read_kb":60147,"num_write":1386,"num_write_kb":38879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.37","version":"48'1642","reported_seq":3498,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.165842+0000","last_change":"2024-05-27T00:17:54.485867+0000","last_active":"2024-05-27T00:33:40.165842+0000","last_peered":"2024-05-27T00:33:40.165842+0000","last_clean":"2024-05-27T00:33:40.165842+0000","last_became_active":"2024-05-27T00:16:06.280544+0000","last_became_peered":"2024-05-27T00:16:06.280544+0000","last_unstale":"2024-05-27T00:33:40.165842+0000","last_undegraded":"2024-05-27T00:33:40.165842+0000","last_fullsized":"2024-05-27T00:33:40.165842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096681779999999995,"stat_sum":{"num_bytes":20317810,"num_objects":636,"num_object_clones":293,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":198,"num_read":60,"num_read_kb":29133,"num_write":1480,"num_write_kb":26137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"48'1534","reported_seq":3411,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.155366+0000","last_change":"2024-05-27T00:17:54.337723+0000","last_active":"2024-05-27T00:33:40.155366+0000","last_peered":"2024-05-27T00:33:40.155366+0000","last_clean":"2024-05-27T00:33:40.155366+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:33:40.155366+0000","last_undegraded":"2024-05-27T00:33:40.155366+0000","last_fullsized":"2024-05-27T00:33:40.155366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1534,"log_dups_size":0,"ondisk_log_size":1534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:55:01.773614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027993297,"stat_sum":{"num_bytes":14791344,"num_objects":591,"num_object_clones":272,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":185,"num_read":55,"num_read_kb":22038,"num_write":1388,"num_write_kb":16506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"48'1640","reported_seq":3471,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.155783+0000","last_change":"2024-05-27T00:17:54.335171+0000","last_active":"2024-05-27T00:33:40.155783+0000","last_peered":"2024-05-27T00:33:40.155783+0000","last_clean":"2024-05-27T00:33:40.155783+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:33:40.155783+0000","last_undegraded":"2024-05-27T00:33:40.155783+0000","last_fullsized":"2024-05-27T00:33:40.155783+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1640,"log_dups_size":0,"ondisk_log_size":1640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:11:26.796581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027428447000000002,"stat_sum":{"num_bytes":10662153,"num_objects":660,"num_object_clones":308,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":660,"num_whiteouts":201,"num_read":45,"num_read_kb":12410,"num_write":1488,"num_write_kb":11561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"48'1624","reported_seq":3450,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.955899+0000","last_change":"2024-05-27T00:17:54.504751+0000","last_active":"2024-05-27T00:33:39.955899+0000","last_peered":"2024-05-27T00:33:39.955899+0000","last_clean":"2024-05-27T00:33:39.955899+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:33:39.955899+0000","last_undegraded":"2024-05-27T00:33:39.955899+0000","last_fullsized":"2024-05-27T00:33:39.955899+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021724307000000002,"stat_sum":{"num_bytes":19461089,"num_objects":642,"num_object_clones":300,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":200,"num_read":31,"num_read_kb":14211,"num_write":1477,"num_write_kb":31965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"48'1565","reported_seq":3420,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.150102+0000","last_change":"2024-05-27T00:17:54.309597+0000","last_active":"2024-05-27T00:33:40.150102+0000","last_peered":"2024-05-27T00:33:40.150102+0000","last_clean":"2024-05-27T00:33:40.150102+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:33:40.150102+0000","last_undegraded":"2024-05-27T00:33:40.150102+0000","last_fullsized":"2024-05-27T00:33:40.150102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1565,"log_dups_size":0,"ondisk_log_size":1565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089222539999999993,"stat_sum":{"num_bytes":26903344,"num_objects":622,"num_object_clones":291,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":209,"num_read":51,"num_read_kb":32873,"num_write":1377,"num_write_kb":36946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"48'1602","reported_seq":3453,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.903439+0000","last_change":"2024-05-27T00:17:54.491142+0000","last_active":"2024-05-27T00:33:39.903439+0000","last_peered":"2024-05-27T00:33:39.903439+0000","last_clean":"2024-05-27T00:33:39.903439+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:33:39.903439+0000","last_undegraded":"2024-05-27T00:33:39.903439+0000","last_fullsized":"2024-05-27T00:33:39.903439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1602,"log_dups_size":0,"ondisk_log_size":1602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016842227000000001,"stat_sum":{"num_bytes":10587319,"num_objects":616,"num_object_clones":288,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":204,"num_read":44,"num_read_kb":16487,"num_write":1434,"num_write_kb":17079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"48'1632","reported_seq":3500,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.949846+0000","last_change":"2024-05-27T00:17:54.491503+0000","last_active":"2024-05-27T00:33:39.949846+0000","last_peered":"2024-05-27T00:33:39.949846+0000","last_clean":"2024-05-27T00:33:39.949846+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:33:39.949846+0000","last_undegraded":"2024-05-27T00:33:39.949846+0000","last_fullsized":"2024-05-27T00:33:39.949846+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1632,"log_dups_size":0,"ondisk_log_size":1632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016821815,"stat_sum":{"num_bytes":23057201,"num_objects":609,"num_object_clones":283,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":212,"num_read":56,"num_read_kb":39458,"num_write":1462,"num_write_kb":27814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"48'1600","reported_seq":3470,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.942964+0000","last_change":"2024-05-27T00:17:54.506224+0000","last_active":"2024-05-27T00:33:39.942964+0000","last_peered":"2024-05-27T00:33:39.942964+0000","last_clean":"2024-05-27T00:33:39.942964+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:33:39.942964+0000","last_undegraded":"2024-05-27T00:33:39.942964+0000","last_fullsized":"2024-05-27T00:33:39.942964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1600,"log_dups_size":0,"ondisk_log_size":1600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.031092022,"stat_sum":{"num_bytes":14535849,"num_objects":605,"num_object_clones":288,"num_object_copies":1210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":605,"num_whiteouts":201,"num_read":63,"num_read_kb":16619,"num_write":1428,"num_write_kb":20418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"48'1720","reported_seq":3589,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.157432+0000","last_change":"2024-05-27T00:17:54.511207+0000","last_active":"2024-05-27T00:33:40.157432+0000","last_peered":"2024-05-27T00:33:40.157432+0000","last_clean":"2024-05-27T00:33:40.157432+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:33:40.157432+0000","last_undegraded":"2024-05-27T00:33:40.157432+0000","last_fullsized":"2024-05-27T00:33:40.157432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1720,"log_dups_size":0,"ondisk_log_size":1720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034311531999999999,"stat_sum":{"num_bytes":28217641,"num_objects":654,"num_object_clones":304,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":203,"num_read":78,"num_read_kb":63526,"num_write":1566,"num_write_kb":34081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"48'1526","reported_seq":3397,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.157506+0000","last_change":"2024-05-27T00:17:54.487572+0000","last_active":"2024-05-27T00:33:40.157506+0000","last_peered":"2024-05-27T00:33:40.157506+0000","last_clean":"2024-05-27T00:33:40.157506+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:33:40.157506+0000","last_undegraded":"2024-05-27T00:33:40.157506+0000","last_fullsized":"2024-05-27T00:33:40.157506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010480461999999999,"stat_sum":{"num_bytes":6663218,"num_objects":590,"num_object_clones":271,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":189,"num_read":47,"num_read_kb":139,"num_write":1374,"num_write_kb":8113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"48'1685","reported_seq":3523,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.156250+0000","last_change":"2024-05-27T00:17:54.337799+0000","last_active":"2024-05-27T00:33:40.156250+0000","last_peered":"2024-05-27T00:33:40.156250+0000","last_clean":"2024-05-27T00:33:40.156250+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:33:40.156250+0000","last_undegraded":"2024-05-27T00:33:40.156250+0000","last_fullsized":"2024-05-27T00:33:40.156250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1685,"log_dups_size":0,"ondisk_log_size":1685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:24:13.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029916169999999999,"stat_sum":{"num_bytes":16074820,"num_objects":657,"num_object_clones":311,"num_object_copies":1314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":208,"num_read":54,"num_read_kb":28815,"num_write":1513,"num_write_kb":21031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"48'1619","reported_seq":3503,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.958080+0000","last_change":"2024-05-27T00:17:54.335854+0000","last_active":"2024-05-27T00:33:39.958080+0000","last_peered":"2024-05-27T00:33:39.958080+0000","last_clean":"2024-05-27T00:33:39.958080+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:33:39.958080+0000","last_undegraded":"2024-05-27T00:33:39.958080+0000","last_fullsized":"2024-05-27T00:33:39.958080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1619,"log_dups_size":0,"ondisk_log_size":1619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:13:02.845565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027334256000000001,"stat_sum":{"num_bytes":31335125,"num_objects":614,"num_object_clones":281,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":187,"num_read":70,"num_read_kb":56922,"num_write":1468,"num_write_kb":40285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"48'1632","reported_seq":3498,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.156351+0000","last_change":"2024-05-27T00:17:54.493382+0000","last_active":"2024-05-27T00:33:40.156351+0000","last_peered":"2024-05-27T00:33:40.156351+0000","last_clean":"2024-05-27T00:33:40.156351+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:33:40.156351+0000","last_undegraded":"2024-05-27T00:33:40.156351+0000","last_fullsized":"2024-05-27T00:33:40.156351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1632,"log_dups_size":0,"ondisk_log_size":1632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017108005999999999,"stat_sum":{"num_bytes":14822117,"num_objects":618,"num_object_clones":289,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":193,"num_read":59,"num_read_kb":12448,"num_write":1490,"num_write_kb":20704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"48'1693","reported_seq":3528,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.156683+0000","last_change":"2024-05-27T00:17:54.509854+0000","last_active":"2024-05-27T00:33:40.156683+0000","last_peered":"2024-05-27T00:33:40.156683+0000","last_clean":"2024-05-27T00:33:40.156683+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:33:40.156683+0000","last_undegraded":"2024-05-27T00:33:40.156683+0000","last_fullsized":"2024-05-27T00:33:40.156683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1693,"log_dups_size":0,"ondisk_log_size":1693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033862588999999998,"stat_sum":{"num_bytes":19005190,"num_objects":664,"num_object_clones":315,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":221,"num_read":54,"num_read_kb":29811,"num_write":1500,"num_write_kb":24710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"48'1540","reported_seq":3417,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.949683+0000","last_change":"2024-05-27T00:17:54.309499+0000","last_active":"2024-05-27T00:33:39.949683+0000","last_peered":"2024-05-27T00:33:39.949683+0000","last_clean":"2024-05-27T00:33:39.949683+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:33:39.949683+0000","last_undegraded":"2024-05-27T00:33:39.949683+0000","last_fullsized":"2024-05-27T00:33:39.949683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1540,"log_dups_size":0,"ondisk_log_size":1540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088470660000000007,"stat_sum":{"num_bytes":9616087,"num_objects":578,"num_object_clones":267,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":178,"num_read":48,"num_read_kb":13982,"num_write":1423,"num_write_kb":15666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"48'1580","reported_seq":3475,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.940611+0000","last_change":"2024-05-27T00:17:54.509276+0000","last_active":"2024-05-27T00:33:39.940611+0000","last_peered":"2024-05-27T00:33:39.940611+0000","last_clean":"2024-05-27T00:33:39.940611+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:33:39.940611+0000","last_undegraded":"2024-05-27T00:33:39.940611+0000","last_fullsized":"2024-05-27T00:33:39.940611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1580,"log_dups_size":0,"ondisk_log_size":1580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033457270999999997,"stat_sum":{"num_bytes":30464589,"num_objects":588,"num_object_clones":269,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":175,"num_read":68,"num_read_kb":58155,"num_write":1446,"num_write_kb":31953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"48'1575","reported_seq":3435,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.896531+0000","last_change":"2024-05-27T00:17:54.310134+0000","last_active":"2024-05-27T00:33:39.896531+0000","last_peered":"2024-05-27T00:33:39.896531+0000","last_clean":"2024-05-27T00:33:39.896531+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:33:39.896531+0000","last_undegraded":"2024-05-27T00:33:39.896531+0000","last_fullsized":"2024-05-27T00:33:39.896531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1575,"log_dups_size":0,"ondisk_log_size":1575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089651499999999999,"stat_sum":{"num_bytes":14724653,"num_objects":608,"num_object_clones":285,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":194,"num_read":49,"num_read_kb":20561,"num_write":1431,"num_write_kb":17383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"48'1694","reported_seq":3556,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.969346+0000","last_change":"2024-05-27T00:17:54.337554+0000","last_active":"2024-05-27T00:33:39.969346+0000","last_peered":"2024-05-27T00:33:39.969346+0000","last_clean":"2024-05-27T00:33:39.969346+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:33:39.969346+0000","last_undegraded":"2024-05-27T00:33:39.969346+0000","last_fullsized":"2024-05-27T00:33:39.969346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1694,"log_dups_size":0,"ondisk_log_size":1694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:02:46.037448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024583906999999999,"stat_sum":{"num_bytes":23745982,"num_objects":632,"num_object_clones":288,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":195,"num_read":55,"num_read_kb":32562,"num_write":1528,"num_write_kb":26486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"48'1534","reported_seq":3408,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.893819+0000","last_change":"2024-05-27T00:17:54.491723+0000","last_active":"2024-05-27T00:33:39.893819+0000","last_peered":"2024-05-27T00:33:39.893819+0000","last_clean":"2024-05-27T00:33:39.893819+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:33:39.893819+0000","last_undegraded":"2024-05-27T00:33:39.893819+0000","last_fullsized":"2024-05-27T00:33:39.893819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1534,"log_dups_size":0,"ondisk_log_size":1534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016722006000000001,"stat_sum":{"num_bytes":10475874,"num_objects":577,"num_object_clones":269,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":180,"num_read":47,"num_read_kb":12352,"num_write":1388,"num_write_kb":12467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"48'1585","reported_seq":3445,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.152936+0000","last_change":"2024-05-27T00:17:54.309792+0000","last_active":"2024-05-27T00:33:40.152936+0000","last_peered":"2024-05-27T00:33:40.152936+0000","last_clean":"2024-05-27T00:33:40.152936+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:33:40.152936+0000","last_undegraded":"2024-05-27T00:33:40.152936+0000","last_fullsized":"2024-05-27T00:33:40.152936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1585,"log_dups_size":0,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089478649999999993,"stat_sum":{"num_bytes":10395394,"num_objects":611,"num_object_clones":285,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":192,"num_read":49,"num_read_kb":15706,"num_write":1435,"num_write_kb":15502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"48'1720","reported_seq":3539,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.149746+0000","last_change":"2024-05-27T00:17:54.494236+0000","last_active":"2024-05-27T00:33:40.149746+0000","last_peered":"2024-05-27T00:33:40.149746+0000","last_clean":"2024-05-27T00:33:40.149746+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:33:40.149746+0000","last_undegraded":"2024-05-27T00:33:40.149746+0000","last_fullsized":"2024-05-27T00:33:40.149746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1720,"log_dups_size":0,"ondisk_log_size":1720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017911486000000001,"stat_sum":{"num_bytes":10165450,"num_objects":674,"num_object_clones":317,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":212,"num_read":40,"num_read_kb":1060,"num_write":1542,"num_write_kb":12604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"48'1617","reported_seq":3486,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.949400+0000","last_change":"2024-05-27T00:17:54.510031+0000","last_active":"2024-05-27T00:33:39.949400+0000","last_peered":"2024-05-27T00:33:39.949400+0000","last_clean":"2024-05-27T00:33:39.949400+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:33:39.949400+0000","last_undegraded":"2024-05-27T00:33:39.949400+0000","last_fullsized":"2024-05-27T00:33:39.949400+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1617,"log_dups_size":0,"ondisk_log_size":1617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033387499000000001,"stat_sum":{"num_bytes":19049177,"num_objects":608,"num_object_clones":281,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":187,"num_read":54,"num_read_kb":24794,"num_write":1464,"num_write_kb":20553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"48'1672","reported_seq":3551,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.154188+0000","last_change":"2024-05-27T00:17:54.484908+0000","last_active":"2024-05-27T00:33:40.154188+0000","last_peered":"2024-05-27T00:33:40.154188+0000","last_clean":"2024-05-27T00:33:40.154188+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:33:40.154188+0000","last_undegraded":"2024-05-27T00:33:40.154188+0000","last_fullsized":"2024-05-27T00:33:40.154188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1672,"log_dups_size":0,"ondisk_log_size":1672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009527878,"stat_sum":{"num_bytes":28018878,"num_objects":606,"num_object_clones":279,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":193,"num_read":62,"num_read_kb":41489,"num_write":1518,"num_write_kb":37283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"48'1590","reported_seq":3486,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.146183+0000","last_change":"2024-05-27T00:17:54.331060+0000","last_active":"2024-05-27T00:33:40.146183+0000","last_peered":"2024-05-27T00:33:40.146183+0000","last_clean":"2024-05-27T00:33:40.146183+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:33:40.146183+0000","last_undegraded":"2024-05-27T00:33:40.146183+0000","last_fullsized":"2024-05-27T00:33:40.146183+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1590,"log_dups_size":0,"ondisk_log_size":1590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:05:48.629948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025435675000000001,"stat_sum":{"num_bytes":18002918,"num_objects":577,"num_object_clones":264,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":183,"num_read":65,"num_read_kb":34529,"num_write":1449,"num_write_kb":21047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"48'1651","reported_seq":3498,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.170743+0000","last_change":"2024-05-27T00:17:54.501574+0000","last_active":"2024-05-27T00:33:40.170743+0000","last_peered":"2024-05-27T00:33:40.170743+0000","last_clean":"2024-05-27T00:33:40.170743+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:33:40.170743+0000","last_undegraded":"2024-05-27T00:33:40.170743+0000","last_fullsized":"2024-05-27T00:33:40.170743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1651,"log_dups_size":0,"ondisk_log_size":1651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019320113,"stat_sum":{"num_bytes":27021070,"num_objects":645,"num_object_clones":298,"num_object_copies":1290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":203,"num_read":50,"num_read_kb":43416,"num_write":1510,"num_write_kb":28409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"48'1107","reported_seq":1327,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2024-05-27T00:33:46.632540+0000","last_change":"2024-05-27T00:33:46.632282+0000","last_active":"2024-05-27T00:33:46.632540+0000","last_peered":"2024-05-27T00:33:46.632540+0000","last_clean":"2024-05-27T00:33:46.632540+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:33:46.632540+0000","last_undegraded":"2024-05-27T00:33:46.632540+0000","last_fullsized":"2024-05-27T00:33:46.632540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1107,"log_dups_size":0,"ondisk_log_size":1107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":240,"num_read_kb":174989,"num_write":1204,"num_write_kb":92121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"48'1585","reported_seq":3451,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.152644+0000","last_change":"2024-05-27T00:17:54.493715+0000","last_active":"2024-05-27T00:33:40.152644+0000","last_peered":"2024-05-27T00:33:40.152644+0000","last_clean":"2024-05-27T00:33:40.152644+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:33:40.152644+0000","last_undegraded":"2024-05-27T00:33:40.152644+0000","last_fullsized":"2024-05-27T00:33:40.152644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1585,"log_dups_size":0,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017111721,"stat_sum":{"num_bytes":21987705,"num_objects":620,"num_object_clones":286,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":197,"num_read":57,"num_read_kb":45093,"num_write":1417,"num_write_kb":24321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"48'845","reported_seq":1089,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.200125+0000","last_change":"2024-05-27T00:15:24.624709+0000","last_active":"2024-05-27T00:33:40.200125+0000","last_peered":"2024-05-27T00:33:40.200125+0000","last_clean":"2024-05-27T00:33:40.200125+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:33:40.200125+0000","last_undegraded":"2024-05-27T00:33:40.200125+0000","last_fullsized":"2024-05-27T00:33:40.200125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":845,"log_dups_size":0,"ondisk_log_size":845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:09:26.665125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":305,"num_read_kb":205128,"num_write":985,"num_write_kb":76341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":68,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"48'598","reported_seq":842,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:43.983707+0000","last_change":"2024-05-27T00:15:24.621344+0000","last_active":"2024-05-27T00:33:43.983707+0000","last_peered":"2024-05-27T00:33:43.983707+0000","last_clean":"2024-05-27T00:33:43.983707+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:33:43.983707+0000","last_undegraded":"2024-05-27T00:33:43.983707+0000","last_fullsized":"2024-05-27T00:33:43.983707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":317,"num_read_kb":129314,"num_write":746,"num_write_kb":59176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":59,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"48'1803","reported_seq":3636,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.149397+0000","last_change":"2024-05-27T00:17:54.309270+0000","last_active":"2024-05-27T00:33:40.149397+0000","last_peered":"2024-05-27T00:33:40.149397+0000","last_clean":"2024-05-27T00:33:40.149397+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:33:40.149397+0000","last_undegraded":"2024-05-27T00:33:40.149397+0000","last_fullsized":"2024-05-27T00:33:40.149397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1803,"log_dups_size":0,"ondisk_log_size":1803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087892630000000003,"stat_sum":{"num_bytes":15183090,"num_objects":667,"num_object_clones":314,"num_object_copies":1334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":667,"num_whiteouts":213,"num_read":52,"num_read_kb":17293,"num_write":1639,"num_write_kb":25751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"48'1918","reported_seq":2166,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:43.987700+0000","last_change":"2024-05-27T00:15:24.619650+0000","last_active":"2024-05-27T00:33:43.987700+0000","last_peered":"2024-05-27T00:33:43.987700+0000","last_clean":"2024-05-27T00:33:43.987700+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:33:43.987700+0000","last_undegraded":"2024-05-27T00:33:43.987700+0000","last_fullsized":"2024-05-27T00:33:43.987700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1918,"log_dups_size":0,"ondisk_log_size":1918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":310,"num_read_kb":210919,"num_write":2091,"num_write_kb":80740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"48'1576","reported_seq":3453,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.146790+0000","last_change":"2024-05-27T00:17:54.489271+0000","last_active":"2024-05-27T00:33:40.146790+0000","last_peered":"2024-05-27T00:33:40.146790+0000","last_clean":"2024-05-27T00:33:40.146790+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:33:40.146790+0000","last_undegraded":"2024-05-27T00:33:40.146790+0000","last_fullsized":"2024-05-27T00:33:40.146790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1576,"log_dups_size":0,"ondisk_log_size":1576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014633893,"stat_sum":{"num_bytes":24048299,"num_objects":610,"num_object_clones":282,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":197,"num_read":64,"num_read_kb":52608,"num_write":1410,"num_write_kb":30269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"48'600","reported_seq":864,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:43.873224+0000","last_change":"2024-05-27T00:15:24.619511+0000","last_active":"2024-05-27T00:33:43.873224+0000","last_peered":"2024-05-27T00:33:43.873224+0000","last_clean":"2024-05-27T00:33:43.873224+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:33:43.873224+0000","last_undegraded":"2024-05-27T00:33:43.873224+0000","last_fullsized":"2024-05-27T00:33:43.873224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:37:54.928131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12951497,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":291,"num_read_kb":124312,"num_write":685,"num_write_kb":63327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"48'1523","reported_seq":3412,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.167464+0000","last_change":"2024-05-27T00:17:54.512612+0000","last_active":"2024-05-27T00:33:40.167464+0000","last_peered":"2024-05-27T00:33:40.167464+0000","last_clean":"2024-05-27T00:33:40.167464+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:33:40.167464+0000","last_undegraded":"2024-05-27T00:33:40.167464+0000","last_fullsized":"2024-05-27T00:33:40.167464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1523,"log_dups_size":0,"ondisk_log_size":1523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034220156000000002,"stat_sum":{"num_bytes":22913099,"num_objects":572,"num_object_clones":265,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":189,"num_read":59,"num_read_kb":32908,"num_write":1359,"num_write_kb":24035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"48'1494","reported_seq":1701,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:43.984997+0000","last_change":"2024-05-27T00:15:24.626683+0000","last_active":"2024-05-27T00:33:43.984997+0000","last_peered":"2024-05-27T00:33:43.984997+0000","last_clean":"2024-05-27T00:33:43.984997+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:33:43.984997+0000","last_undegraded":"2024-05-27T00:33:43.984997+0000","last_fullsized":"2024-05-27T00:33:43.984997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":244,"num_read_kb":161522,"num_write":1599,"num_write_kb":88170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"48'1768","reported_seq":3540,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.940910+0000","last_change":"2024-05-27T00:17:54.493629+0000","last_active":"2024-05-27T00:33:39.940910+0000","last_peered":"2024-05-27T00:33:39.940910+0000","last_clean":"2024-05-27T00:33:39.940910+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:33:39.940910+0000","last_undegraded":"2024-05-27T00:33:39.940910+0000","last_fullsized":"2024-05-27T00:33:39.940910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1768,"log_dups_size":0,"ondisk_log_size":1768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017358472,"stat_sum":{"num_bytes":15539305,"num_objects":713,"num_object_clones":382,"num_object_copies":1426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":205,"num_read":59,"num_read_kb":17601,"num_write":1512,"num_write_kb":17852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"48'1543","reported_seq":1840,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:44.608340+0000","last_change":"2024-05-27T00:15:24.617928+0000","last_active":"2024-05-27T00:33:44.608340+0000","last_peered":"2024-05-27T00:33:44.608340+0000","last_clean":"2024-05-27T00:33:44.608340+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:33:44.608340+0000","last_undegraded":"2024-05-27T00:33:44.608340+0000","last_fullsized":"2024-05-27T00:33:44.608340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1543,"log_dups_size":0,"ondisk_log_size":1543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17444547,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":428,"num_read_kb":130260,"num_write":1759,"num_write_kb":61382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"48'1600","reported_seq":3464,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.156525+0000","last_change":"2024-05-27T00:17:54.506220+0000","last_active":"2024-05-27T00:33:40.156525+0000","last_peered":"2024-05-27T00:33:40.156525+0000","last_clean":"2024-05-27T00:33:40.156525+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:33:40.156525+0000","last_undegraded":"2024-05-27T00:33:40.156525+0000","last_fullsized":"2024-05-27T00:33:40.156525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1600,"log_dups_size":0,"ondisk_log_size":1600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022583427,"stat_sum":{"num_bytes":13591445,"num_objects":611,"num_object_clones":285,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":198,"num_read":54,"num_read_kb":12130,"num_write":1438,"num_write_kb":19929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"48'1072","reported_seq":1354,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.197500+0000","last_change":"2024-05-27T00:15:24.622260+0000","last_active":"2024-05-27T00:33:40.197500+0000","last_peered":"2024-05-27T00:33:40.197500+0000","last_clean":"2024-05-27T00:33:40.197500+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:33:40.197500+0000","last_undegraded":"2024-05-27T00:33:40.197500+0000","last_fullsized":"2024-05-27T00:33:40.197500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:05:52.966509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555104,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":365,"num_read_kb":194738,"num_write":1226,"num_write_kb":105127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"48'1584","reported_seq":3439,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.156036+0000","last_change":"2024-05-27T00:17:54.336019+0000","last_active":"2024-05-27T00:33:40.156036+0000","last_peered":"2024-05-27T00:33:40.156036+0000","last_clean":"2024-05-27T00:33:40.156036+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:33:40.156036+0000","last_undegraded":"2024-05-27T00:33:40.156036+0000","last_fullsized":"2024-05-27T00:33:40.156036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:48.582600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.030558409000000002,"stat_sum":{"num_bytes":10583440,"num_objects":609,"num_object_clones":281,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":187,"num_read":42,"num_read_kb":12368,"num_write":1447,"num_write_kb":13410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"48'890","reported_seq":1155,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:44.009294+0000","last_change":"2024-05-27T00:33:44.009333+0000","last_active":"2024-05-27T00:33:44.009294+0000","last_peered":"2024-05-27T00:33:44.009294+0000","last_clean":"2024-05-27T00:33:44.009294+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:33:44.009294+0000","last_undegraded":"2024-05-27T00:33:44.009294+0000","last_fullsized":"2024-05-27T00:33:44.009294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'890","last_scrub_stamp":"2024-05-27T00:33:44.009223+0000","last_deep_scrub":"48'890","last_deep_scrub_stamp":"2024-05-27T00:33:44.009223+0000","last_clean_scrub_stamp":"2024-05-27T00:33:44.009223+0000","objects_scrubbed":65,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:48:38.583455+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":336,"num_read_kb":131285,"num_write":985,"num_write_kb":67415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":59,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706283,"num_omap_keys":1375,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"48'1723","reported_seq":3532,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.172954+0000","last_change":"2024-05-27T00:17:54.493027+0000","last_active":"2024-05-27T00:33:40.172954+0000","last_peered":"2024-05-27T00:33:40.172954+0000","last_clean":"2024-05-27T00:33:40.172954+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:33:40.172954+0000","last_undegraded":"2024-05-27T00:33:40.172954+0000","last_fullsized":"2024-05-27T00:33:40.172954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1723,"log_dups_size":0,"ondisk_log_size":1723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018271039999999999,"stat_sum":{"num_bytes":20486162,"num_objects":658,"num_object_clones":358,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":188,"num_read":72,"num_read_kb":36313,"num_write":1477,"num_write_kb":31834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"48'525","reported_seq":723,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.187316+0000","last_change":"2024-05-27T00:15:24.616098+0000","last_active":"2024-05-27T00:33:40.187316+0000","last_peered":"2024-05-27T00:33:40.187316+0000","last_clean":"2024-05-27T00:33:40.187316+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:33:40.187316+0000","last_undegraded":"2024-05-27T00:33:40.187316+0000","last_fullsized":"2024-05-27T00:33:40.187316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":232,"num_read_kb":71610,"num_write":648,"num_write_kb":51355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"48'1696","reported_seq":3565,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.156593+0000","last_change":"2024-05-27T00:17:54.334890+0000","last_active":"2024-05-27T00:33:40.156593+0000","last_peered":"2024-05-27T00:33:40.156593+0000","last_clean":"2024-05-27T00:33:40.156593+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:33:40.156593+0000","last_undegraded":"2024-05-27T00:33:40.156593+0000","last_fullsized":"2024-05-27T00:33:40.156593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1696,"log_dups_size":0,"ondisk_log_size":1696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:55.040370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028694234999999998,"stat_sum":{"num_bytes":27865757,"num_objects":639,"num_object_clones":296,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":193,"num_read":71,"num_read_kb":46547,"num_write":1540,"num_write_kb":29871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"48'368","reported_seq":566,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.171329+0000","last_change":"2024-05-27T00:15:24.626709+0000","last_active":"2024-05-27T00:33:40.171329+0000","last_peered":"2024-05-27T00:33:40.171329+0000","last_clean":"2024-05-27T00:33:40.171329+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:33:40.171329+0000","last_undegraded":"2024-05-27T00:33:40.171329+0000","last_fullsized":"2024-05-27T00:33:40.171329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:23.420927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":244,"num_read_kb":99922,"num_write":495,"num_write_kb":55215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"48'1584","reported_seq":3448,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.920151+0000","last_change":"2024-05-27T00:17:54.332802+0000","last_active":"2024-05-27T00:33:39.920151+0000","last_peered":"2024-05-27T00:33:39.920151+0000","last_clean":"2024-05-27T00:33:39.920151+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:33:39.920151+0000","last_undegraded":"2024-05-27T00:33:39.920151+0000","last_fullsized":"2024-05-27T00:33:39.920151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:41:31.328434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025579069999999999,"stat_sum":{"num_bytes":19681280,"num_objects":594,"num_object_clones":272,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":185,"num_read":43,"num_read_kb":14554,"num_write":1432,"num_write_kb":24217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"48'756","reported_seq":1043,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:43.866242+0000","last_change":"2024-05-27T00:15:24.601101+0000","last_active":"2024-05-27T00:33:43.866242+0000","last_peered":"2024-05-27T00:33:43.866242+0000","last_clean":"2024-05-27T00:33:43.866242+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:33:43.866242+0000","last_undegraded":"2024-05-27T00:33:43.866242+0000","last_fullsized":"2024-05-27T00:33:43.866242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":479,"num_read_kb":120710,"num_write":993,"num_write_kb":72772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":73,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"48'1629","reported_seq":3504,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.956844+0000","last_change":"2024-05-27T00:17:54.334829+0000","last_active":"2024-05-27T00:33:39.956844+0000","last_peered":"2024-05-27T00:33:39.956844+0000","last_clean":"2024-05-27T00:33:39.956844+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:33:39.956844+0000","last_undegraded":"2024-05-27T00:33:39.956844+0000","last_fullsized":"2024-05-27T00:33:39.956844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1629,"log_dups_size":0,"ondisk_log_size":1629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:00:47.756042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028937418999999999,"stat_sum":{"num_bytes":19558883,"num_objects":604,"num_object_clones":279,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":200,"num_read":60,"num_read_kb":39320,"num_write":1451,"num_write_kb":30058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"48'995","reported_seq":1256,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:46.064385+0000","last_change":"2024-05-27T00:33:46.064473+0000","last_active":"2024-05-27T00:33:46.064385+0000","last_peered":"2024-05-27T00:33:46.064385+0000","last_clean":"2024-05-27T00:33:46.064385+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:33:46.064385+0000","last_undegraded":"2024-05-27T00:33:46.064385+0000","last_fullsized":"2024-05-27T00:33:46.064385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'995","last_scrub_stamp":"2024-05-27T00:33:46.064195+0000","last_deep_scrub":"48'995","last_deep_scrub_stamp":"2024-05-27T00:33:46.064195+0000","last_clean_scrub_stamp":"2024-05-27T00:33:46.064195+0000","objects_scrubbed":68,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:50:49.020441+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":329,"num_read_kb":104768,"num_write":1082,"num_write_kb":59608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":831747,"num_omap_keys":1453,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"48'1612","reported_seq":3472,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.958971+0000","last_change":"2024-05-27T00:17:54.335438+0000","last_active":"2024-05-27T00:33:39.958971+0000","last_peered":"2024-05-27T00:33:39.958971+0000","last_clean":"2024-05-27T00:33:39.958971+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:33:39.958971+0000","last_undegraded":"2024-05-27T00:33:39.958971+0000","last_fullsized":"2024-05-27T00:33:39.958971+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1612,"log_dups_size":0,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:06:44.778493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029056843999999998,"stat_sum":{"num_bytes":27647082,"num_objects":631,"num_object_clones":290,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":189,"num_read":55,"num_read_kb":39571,"num_write":1465,"num_write_kb":33528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"48'901","reported_seq":1125,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:46.558889+0000","last_change":"2024-05-27T00:33:46.558932+0000","last_active":"2024-05-27T00:33:46.558889+0000","last_peered":"2024-05-27T00:33:46.558889+0000","last_clean":"2024-05-27T00:33:46.558889+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:33:46.558889+0000","last_undegraded":"2024-05-27T00:33:46.558889+0000","last_fullsized":"2024-05-27T00:33:46.558889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'901","last_scrub_stamp":"2024-05-27T00:33:46.558817+0000","last_deep_scrub":"48'901","last_deep_scrub_stamp":"2024-05-27T00:33:46.558817+0000","last_clean_scrub_stamp":"2024-05-27T00:33:46.558817+0000","objects_scrubbed":54,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:58:45.617077+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":241,"num_read_kb":194399,"num_write":978,"num_write_kb":84425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":982076,"num_omap_keys":1745,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"48'1553","reported_seq":3433,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.962590+0000","last_change":"2024-05-27T00:17:54.337688+0000","last_active":"2024-05-27T00:33:39.962590+0000","last_peered":"2024-05-27T00:33:39.962590+0000","last_clean":"2024-05-27T00:33:39.962590+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:33:39.962590+0000","last_undegraded":"2024-05-27T00:33:39.962590+0000","last_fullsized":"2024-05-27T00:33:39.962590+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1553,"log_dups_size":0,"ondisk_log_size":1553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:24:09.263437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028414170999999998,"stat_sum":{"num_bytes":22687403,"num_objects":590,"num_object_clones":272,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":186,"num_read":57,"num_read_kb":45093,"num_write":1416,"num_write_kb":28216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"48'1219","reported_seq":1468,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:45.787103+0000","last_change":"2024-05-27T00:33:45.787134+0000","last_active":"2024-05-27T00:33:45.787103+0000","last_peered":"2024-05-27T00:33:45.787103+0000","last_clean":"2024-05-27T00:33:45.787103+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:33:45.787103+0000","last_undegraded":"2024-05-27T00:33:45.787103+0000","last_fullsized":"2024-05-27T00:33:45.787103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1219","last_scrub_stamp":"2024-05-27T00:33:45.787032+0000","last_deep_scrub":"48'1219","last_deep_scrub_stamp":"2024-05-27T00:33:45.787032+0000","last_clean_scrub_stamp":"2024-05-27T00:33:45.787032+0000","objects_scrubbed":80,"log_size":1219,"log_dups_size":0,"ondisk_log_size":1219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:25:46.003419+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33559003,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":293,"num_read_kb":166112,"num_write":1307,"num_write_kb":92227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":69,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":694602,"num_omap_keys":1256,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"48'1573","reported_seq":3452,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.170007+0000","last_change":"2024-05-27T00:17:54.512510+0000","last_active":"2024-05-27T00:33:40.170007+0000","last_peered":"2024-05-27T00:33:40.170007+0000","last_clean":"2024-05-27T00:33:40.170007+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:33:40.170007+0000","last_undegraded":"2024-05-27T00:33:40.170007+0000","last_fullsized":"2024-05-27T00:33:40.170007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034804740000000001,"stat_sum":{"num_bytes":14076594,"num_objects":568,"num_object_clones":258,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":187,"num_read":41,"num_read_kb":16491,"num_write":1432,"num_write_kb":17170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"48'761","reported_seq":990,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.193866+0000","last_change":"2024-05-27T00:15:24.621522+0000","last_active":"2024-05-27T00:33:40.193866+0000","last_peered":"2024-05-27T00:33:40.193866+0000","last_clean":"2024-05-27T00:33:40.193866+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:33:40.193866+0000","last_undegraded":"2024-05-27T00:33:40.193866+0000","last_fullsized":"2024-05-27T00:33:40.193866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":292,"num_read_kb":141638,"num_write":933,"num_write_kb":64377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"48'1583","reported_seq":3437,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.949997+0000","last_change":"2024-05-27T00:17:54.333811+0000","last_active":"2024-05-27T00:33:39.949997+0000","last_peered":"2024-05-27T00:33:39.949997+0000","last_clean":"2024-05-27T00:33:39.949997+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:33:39.949997+0000","last_undegraded":"2024-05-27T00:33:39.949997+0000","last_fullsized":"2024-05-27T00:33:39.949997+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1583,"log_dups_size":0,"ondisk_log_size":1583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:11:14.148909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028015244000000002,"stat_sum":{"num_bytes":15862998,"num_objects":620,"num_object_clones":289,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":202,"num_read":48,"num_read_kb":16478,"num_write":1406,"num_write_kb":22882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":231,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:32:10.519872+0000","last_change":"2024-05-27T00:15:14.504622+0000","last_active":"2024-05-27T00:32:10.519872+0000","last_peered":"2024-05-27T00:32:10.519872+0000","last_clean":"2024-05-27T00:32:10.519872+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:32:10.519872+0000","last_undegraded":"2024-05-27T00:32:10.519872+0000","last_fullsized":"2024-05-27T00:32:10.519872+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:13.410459+0000","last_clean_scrub_stamp":"2024-05-27T00:15:13.410459+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:16:15.911161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"48'881","reported_seq":1083,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2024-05-27T00:33:46.723666+0000","last_change":"2024-05-27T00:33:46.723153+0000","last_active":"2024-05-27T00:33:46.723666+0000","last_peered":"2024-05-27T00:33:46.723666+0000","last_clean":"2024-05-27T00:33:46.723666+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:33:46.723666+0000","last_undegraded":"2024-05-27T00:33:46.723666+0000","last_fullsized":"2024-05-27T00:33:46.723666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":219,"num_read_kb":131843,"num_write":971,"num_write_kb":67539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"48'1634","reported_seq":3495,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.169631+0000","last_change":"2024-05-27T00:17:54.309203+0000","last_active":"2024-05-27T00:33:40.169631+0000","last_peered":"2024-05-27T00:33:40.169631+0000","last_clean":"2024-05-27T00:33:40.169631+0000","last_became_active":"2024-05-27T00:16:06.517816+0000","last_became_peered":"2024-05-27T00:16:06.517816+0000","last_unstale":"2024-05-27T00:33:40.169631+0000","last_undegraded":"2024-05-27T00:33:40.169631+0000","last_fullsized":"2024-05-27T00:33:40.169631+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1634,"log_dups_size":0,"ondisk_log_size":1634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085828740000000008,"stat_sum":{"num_bytes":27761524,"num_objects":639,"num_object_clones":294,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":200,"num_read":54,"num_read_kb":34546,"num_write":1480,"num_write_kb":32905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"48'436","reported_seq":673,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.196497+0000","last_change":"2024-05-27T00:15:24.627456+0000","last_active":"2024-05-27T00:33:40.196497+0000","last_peered":"2024-05-27T00:33:40.196497+0000","last_clean":"2024-05-27T00:33:40.196497+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:33:40.196497+0000","last_undegraded":"2024-05-27T00:33:40.196497+0000","last_fullsized":"2024-05-27T00:33:40.196497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:45:35.726018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":299,"num_read_kb":174228,"num_write":559,"num_write_kb":63257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"48'1533","reported_seq":3402,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.145186+0000","last_change":"2024-05-27T00:17:54.490594+0000","last_active":"2024-05-27T00:33:40.145186+0000","last_peered":"2024-05-27T00:33:40.145186+0000","last_clean":"2024-05-27T00:33:40.145186+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:33:40.145186+0000","last_undegraded":"2024-05-27T00:33:40.145186+0000","last_fullsized":"2024-05-27T00:33:40.145186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1533,"log_dups_size":0,"ondisk_log_size":1533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016778365999999999,"stat_sum":{"num_bytes":6084007,"num_objects":569,"num_object_clones":261,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":180,"num_read":34,"num_read_kb":53,"num_write":1405,"num_write_kb":7726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'1728","reported_seq":2626,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:45.565851+0000","last_change":"2024-05-27T00:33:45.565881+0000","last_active":"2024-05-27T00:33:45.565851+0000","last_peered":"2024-05-27T00:33:45.565851+0000","last_clean":"2024-05-27T00:33:45.565851+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:33:45.565851+0000","last_undegraded":"2024-05-27T00:33:45.565851+0000","last_fullsized":"2024-05-27T00:33:45.565851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1728","last_scrub_stamp":"2024-05-27T00:33:45.565774+0000","last_deep_scrub":"48'1728","last_deep_scrub_stamp":"2024-05-27T00:33:45.565774+0000","last_clean_scrub_stamp":"2024-05-27T00:33:45.565774+0000","objects_scrubbed":79,"log_size":1728,"log_dups_size":0,"ondisk_log_size":1728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:01:36.629433+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":79,"num_object_clones":0,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":1008,"num_read_kb":169683,"num_write":1833,"num_write_kb":85277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":73,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1183597,"num_omap_keys":2018,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"48'1635","reported_seq":3494,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.146109+0000","last_change":"2024-05-27T00:17:54.491784+0000","last_active":"2024-05-27T00:33:40.146109+0000","last_peered":"2024-05-27T00:33:40.146109+0000","last_clean":"2024-05-27T00:33:40.146109+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:33:40.146109+0000","last_undegraded":"2024-05-27T00:33:40.146109+0000","last_fullsized":"2024-05-27T00:33:40.146109+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1635,"log_dups_size":0,"ondisk_log_size":1635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016042048999999999,"stat_sum":{"num_bytes":11674242,"num_objects":638,"num_object_clones":289,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":201,"num_read":51,"num_read_kb":12401,"num_write":1479,"num_write_kb":14079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"48'712","reported_seq":972,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:43.973761+0000","last_change":"2024-05-27T00:15:24.616011+0000","last_active":"2024-05-27T00:33:43.973761+0000","last_peered":"2024-05-27T00:33:43.973761+0000","last_clean":"2024-05-27T00:33:43.973761+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:33:43.973761+0000","last_undegraded":"2024-05-27T00:33:43.973761+0000","last_fullsized":"2024-05-27T00:33:43.973761+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":344,"num_read_kb":168323,"num_write":932,"num_write_kb":68377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"48'1658","reported_seq":3528,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.173823+0000","last_change":"2024-05-27T00:17:54.505548+0000","last_active":"2024-05-27T00:33:40.173823+0000","last_peered":"2024-05-27T00:33:40.173823+0000","last_clean":"2024-05-27T00:33:40.173823+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:33:40.173823+0000","last_undegraded":"2024-05-27T00:33:40.173823+0000","last_fullsized":"2024-05-27T00:33:40.173823+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1658,"log_dups_size":0,"ondisk_log_size":1658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021175818999999999,"stat_sum":{"num_bytes":27400217,"num_objects":605,"num_object_clones":275,"num_object_copies":1210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":605,"num_whiteouts":182,"num_read":49,"num_read_kb":37771,"num_write":1528,"num_write_kb":29686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"48'847","reported_seq":1074,"reported_epoch":48,"state":"active+clean+scrubbing+deep","last_fresh":"2024-05-27T00:33:46.639030+0000","last_change":"2024-05-27T00:33:46.616212+0000","last_active":"2024-05-27T00:33:46.639030+0000","last_peered":"2024-05-27T00:33:46.639030+0000","last_clean":"2024-05-27T00:33:46.639030+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:33:46.639030+0000","last_undegraded":"2024-05-27T00:33:46.639030+0000","last_fullsized":"2024-05-27T00:33:46.639030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":24,"log_size":847,"log_dups_size":0,"ondisk_log_size":847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":262,"num_read_kb":165581,"num_write":945,"num_write_kb":83700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"48'1605","reported_seq":3470,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.168819+0000","last_change":"2024-05-27T00:17:54.511904+0000","last_active":"2024-05-27T00:33:40.168819+0000","last_peered":"2024-05-27T00:33:40.168819+0000","last_clean":"2024-05-27T00:33:40.168819+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:33:40.168819+0000","last_undegraded":"2024-05-27T00:33:40.168819+0000","last_fullsized":"2024-05-27T00:33:40.168819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1605,"log_dups_size":0,"ondisk_log_size":1605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034092800999999999,"stat_sum":{"num_bytes":22501150,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":211,"num_read":50,"num_read_kb":38442,"num_write":1424,"num_write_kb":28481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"48'1418","reported_seq":1658,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:43.987414+0000","last_change":"2024-05-27T00:15:24.635369+0000","last_active":"2024-05-27T00:33:43.987414+0000","last_peered":"2024-05-27T00:33:43.987414+0000","last_clean":"2024-05-27T00:33:43.987414+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:33:43.987414+0000","last_undegraded":"2024-05-27T00:33:43.987414+0000","last_fullsized":"2024-05-27T00:33:43.987414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":1418,"log_dups_size":0,"ondisk_log_size":1418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:31:18.312243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":299,"num_read_kb":161725,"num_write":1535,"num_write_kb":67545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"48'1583","reported_seq":3463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.149807+0000","last_change":"2024-05-27T00:17:54.488788+0000","last_active":"2024-05-27T00:33:40.149807+0000","last_peered":"2024-05-27T00:33:40.149807+0000","last_clean":"2024-05-27T00:33:40.149807+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:33:40.149807+0000","last_undegraded":"2024-05-27T00:33:40.149807+0000","last_fullsized":"2024-05-27T00:33:40.149807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1583,"log_dups_size":0,"ondisk_log_size":1583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014512239,"stat_sum":{"num_bytes":23141178,"num_objects":586,"num_object_clones":267,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":186,"num_read":51,"num_read_kb":31023,"num_write":1456,"num_write_kb":28383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'841","reported_seq":1079,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:43.971205+0000","last_change":"2024-05-27T00:15:24.627360+0000","last_active":"2024-05-27T00:33:43.971205+0000","last_peered":"2024-05-27T00:33:43.971205+0000","last_clean":"2024-05-27T00:33:43.971205+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:33:43.971205+0000","last_undegraded":"2024-05-27T00:33:43.971205+0000","last_fullsized":"2024-05-27T00:33:43.971205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":312,"num_read_kb":117165,"num_write":1609,"num_write_kb":59353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"48'1522","reported_seq":3397,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.172910+0000","last_change":"2024-05-27T00:17:54.310241+0000","last_active":"2024-05-27T00:33:40.172910+0000","last_peered":"2024-05-27T00:33:40.172910+0000","last_clean":"2024-05-27T00:33:40.172910+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:33:40.172910+0000","last_undegraded":"2024-05-27T00:33:40.172910+0000","last_fullsized":"2024-05-27T00:33:40.172910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1522,"log_dups_size":0,"ondisk_log_size":1522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089509419999999999,"stat_sum":{"num_bytes":13324494,"num_objects":587,"num_object_clones":270,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":182,"num_read":49,"num_read_kb":2453,"num_write":1387,"num_write_kb":19663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"48'650","reported_seq":902,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.181164+0000","last_change":"2024-05-27T00:15:24.628720+0000","last_active":"2024-05-27T00:33:40.181164+0000","last_peered":"2024-05-27T00:33:40.181164+0000","last_clean":"2024-05-27T00:33:40.181164+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:33:40.181164+0000","last_undegraded":"2024-05-27T00:33:40.181164+0000","last_fullsized":"2024-05-27T00:33:40.181164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":352,"num_read_kb":95868,"num_write":816,"num_write_kb":47576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"48'1584","reported_seq":3458,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.898539+0000","last_change":"2024-05-27T00:17:54.492705+0000","last_active":"2024-05-27T00:33:39.898539+0000","last_peered":"2024-05-27T00:33:39.898539+0000","last_clean":"2024-05-27T00:33:39.898539+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:33:39.898539+0000","last_undegraded":"2024-05-27T00:33:39.898539+0000","last_fullsized":"2024-05-27T00:33:39.898539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018225775999999999,"stat_sum":{"num_bytes":9782754,"num_objects":583,"num_object_clones":270,"num_object_copies":1166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":186,"num_read":49,"num_read_kb":4155,"num_write":1427,"num_write_kb":16778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'475","reported_seq":696,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:43.870618+0000","last_change":"2024-05-27T00:15:24.633173+0000","last_active":"2024-05-27T00:33:43.870618+0000","last_peered":"2024-05-27T00:33:43.870618+0000","last_clean":"2024-05-27T00:33:43.870618+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:33:43.870618+0000","last_undegraded":"2024-05-27T00:33:43.870618+0000","last_fullsized":"2024-05-27T00:33:43.870618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:14:01.202822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":297,"num_read_kb":59216,"num_write":639,"num_write_kb":42957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":72,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"48'1628","reported_seq":3492,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.952827+0000","last_change":"2024-05-27T00:17:54.504734+0000","last_active":"2024-05-27T00:33:39.952827+0000","last_peered":"2024-05-27T00:33:39.952827+0000","last_clean":"2024-05-27T00:33:39.952827+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:33:39.952827+0000","last_undegraded":"2024-05-27T00:33:39.952827+0000","last_fullsized":"2024-05-27T00:33:39.952827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1628,"log_dups_size":0,"ondisk_log_size":1628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022802596000000001,"stat_sum":{"num_bytes":19605226,"num_objects":622,"num_object_clones":284,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":196,"num_read":52,"num_read_kb":30708,"num_write":1474,"num_write_kb":21111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"48'786","reported_seq":957,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:44.043444+0000","last_change":"2024-05-27T00:15:24.609526+0000","last_active":"2024-05-27T00:33:44.043444+0000","last_peered":"2024-05-27T00:33:44.043444+0000","last_clean":"2024-05-27T00:33:44.043444+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:33:44.043444+0000","last_undegraded":"2024-05-27T00:33:44.043444+0000","last_fullsized":"2024-05-27T00:33:44.043444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":786,"log_dups_size":0,"ondisk_log_size":786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":176,"num_read_kb":80391,"num_write":882,"num_write_kb":51073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"48'1685","reported_seq":3546,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.170822+0000","last_change":"2024-05-27T00:17:54.308939+0000","last_active":"2024-05-27T00:33:40.170822+0000","last_peered":"2024-05-27T00:33:40.170822+0000","last_clean":"2024-05-27T00:33:40.170822+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:33:40.170822+0000","last_undegraded":"2024-05-27T00:33:40.170822+0000","last_fullsized":"2024-05-27T00:33:40.170822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1685,"log_dups_size":0,"ondisk_log_size":1685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084477519999999993,"stat_sum":{"num_bytes":19978404,"num_objects":645,"num_object_clones":297,"num_object_copies":1290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":207,"num_read":62,"num_read_kb":32956,"num_write":1519,"num_write_kb":26495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"48'1097","reported_seq":1332,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:44.664090+0000","last_change":"2024-05-27T00:33:44.664114+0000","last_active":"2024-05-27T00:33:44.664090+0000","last_peered":"2024-05-27T00:33:44.664090+0000","last_clean":"2024-05-27T00:33:44.664090+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:33:44.664090+0000","last_undegraded":"2024-05-27T00:33:44.664090+0000","last_fullsized":"2024-05-27T00:33:44.664090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1097","last_scrub_stamp":"2024-05-27T00:33:44.664017+0000","last_deep_scrub":"48'1097","last_deep_scrub_stamp":"2024-05-27T00:33:44.664017+0000","last_clean_scrub_stamp":"2024-05-27T00:33:44.664017+0000","objects_scrubbed":54,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:57:50.619399+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":269,"num_read_kb":153672,"num_write":1141,"num_write_kb":75700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":695727,"num_omap_keys":1211,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"48'1633","reported_seq":3475,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.150185+0000","last_change":"2024-05-27T00:17:54.310021+0000","last_active":"2024-05-27T00:33:40.150185+0000","last_peered":"2024-05-27T00:33:40.150185+0000","last_clean":"2024-05-27T00:33:40.150185+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:33:40.150185+0000","last_undegraded":"2024-05-27T00:33:40.150185+0000","last_fullsized":"2024-05-27T00:33:40.150185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1633,"log_dups_size":0,"ondisk_log_size":1633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091147599999999995,"stat_sum":{"num_bytes":15864263,"num_objects":631,"num_object_clones":289,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":197,"num_read":35,"num_read_kb":8228,"num_write":1469,"num_write_kb":20882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"48'701","reported_seq":954,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.201330+0000","last_change":"2024-05-27T00:15:24.628906+0000","last_active":"2024-05-27T00:33:40.201330+0000","last_peered":"2024-05-27T00:33:40.201330+0000","last_clean":"2024-05-27T00:33:40.201330+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:33:40.201330+0000","last_undegraded":"2024-05-27T00:33:40.201330+0000","last_fullsized":"2024-05-27T00:33:40.201330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":701,"log_dups_size":0,"ondisk_log_size":701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:06:19.558484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":331,"num_read_kb":161875,"num_write":846,"num_write_kb":84354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"48'1605","reported_seq":3462,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.147901+0000","last_change":"2024-05-27T00:17:54.333938+0000","last_active":"2024-05-27T00:33:40.147901+0000","last_peered":"2024-05-27T00:33:40.147901+0000","last_clean":"2024-05-27T00:33:40.147901+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:33:40.147901+0000","last_undegraded":"2024-05-27T00:33:40.147901+0000","last_fullsized":"2024-05-27T00:33:40.147901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1605,"log_dups_size":0,"ondisk_log_size":1605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:56:22.740779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026544363000000001,"stat_sum":{"num_bytes":11205938,"num_objects":584,"num_object_clones":274,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":174,"num_read":36,"num_read_kb":3645,"num_write":1476,"num_write_kb":12517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"48'1544","reported_seq":3409,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.145203+0000","last_change":"2024-05-27T00:17:54.485932+0000","last_active":"2024-05-27T00:33:40.145203+0000","last_peered":"2024-05-27T00:33:40.145203+0000","last_clean":"2024-05-27T00:33:40.145203+0000","last_became_active":"2024-05-27T00:16:06.811922+0000","last_became_peered":"2024-05-27T00:16:06.811922+0000","last_unstale":"2024-05-27T00:33:40.145203+0000","last_undegraded":"2024-05-27T00:33:40.145203+0000","last_fullsized":"2024-05-27T00:33:40.145203+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1544,"log_dups_size":0,"ondisk_log_size":1544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011085174,"stat_sum":{"num_bytes":14355433,"num_objects":589,"num_object_clones":272,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":196,"num_read":48,"num_read_kb":4199,"num_write":1384,"num_write_kb":25841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"48'494","reported_seq":707,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.190121+0000","last_change":"2024-05-27T00:15:24.627579+0000","last_active":"2024-05-27T00:33:40.190121+0000","last_peered":"2024-05-27T00:33:40.190121+0000","last_clean":"2024-05-27T00:33:40.190121+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:33:40.190121+0000","last_undegraded":"2024-05-27T00:33:40.190121+0000","last_fullsized":"2024-05-27T00:33:40.190121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:50:13.960534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":267,"num_read_kb":124158,"num_write":636,"num_write_kb":63683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"48'1674","reported_seq":3503,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.960032+0000","last_change":"2024-05-27T00:17:54.511007+0000","last_active":"2024-05-27T00:33:39.960032+0000","last_peered":"2024-05-27T00:33:39.960032+0000","last_clean":"2024-05-27T00:33:39.960032+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:33:39.960032+0000","last_undegraded":"2024-05-27T00:33:39.960032+0000","last_fullsized":"2024-05-27T00:33:39.960032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1674,"log_dups_size":0,"ondisk_log_size":1674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.034279472999999998,"stat_sum":{"num_bytes":10551410,"num_objects":648,"num_object_clones":309,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":224,"num_read":43,"num_read_kb":11171,"num_write":1487,"num_write_kb":16962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"48'862","reported_seq":1119,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:39.611112+0000","last_change":"2024-05-27T00:15:24.634435+0000","last_active":"2024-05-27T00:33:39.611112+0000","last_peered":"2024-05-27T00:33:39.611112+0000","last_clean":"2024-05-27T00:33:39.611112+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:33:39.611112+0000","last_undegraded":"2024-05-27T00:33:39.611112+0000","last_fullsized":"2024-05-27T00:33:39.611112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:23.557969+0000","last_clean_scrub_stamp":"2024-05-27T00:15:23.557969+0000","objects_scrubbed":0,"log_size":862,"log_dups_size":0,"ondisk_log_size":862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:17.142239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":337,"num_read_kb":206604,"num_write":987,"num_write_kb":108718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"48'1557","reported_seq":3428,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.166459+0000","last_change":"2024-05-27T00:17:54.493359+0000","last_active":"2024-05-27T00:33:40.166459+0000","last_peered":"2024-05-27T00:33:40.166459+0000","last_clean":"2024-05-27T00:33:40.166459+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:33:40.166459+0000","last_undegraded":"2024-05-27T00:33:40.166459+0000","last_fullsized":"2024-05-27T00:33:40.166459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017325917,"stat_sum":{"num_bytes":23014283,"num_objects":609,"num_object_clones":284,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":185,"num_read":59,"num_read_kb":40021,"num_write":1416,"num_write_kb":24406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"48'1338","reported_seq":1606,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:45.717953+0000","last_change":"2024-05-27T00:33:45.718003+0000","last_active":"2024-05-27T00:33:45.717953+0000","last_peered":"2024-05-27T00:33:45.717953+0000","last_clean":"2024-05-27T00:33:45.717953+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:33:45.717953+0000","last_undegraded":"2024-05-27T00:33:45.717953+0000","last_fullsized":"2024-05-27T00:33:45.717953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1338","last_scrub_stamp":"2024-05-27T00:33:45.717878+0000","last_deep_scrub":"48'1338","last_deep_scrub_stamp":"2024-05-27T00:33:45.717878+0000","last_clean_scrub_stamp":"2024-05-27T00:33:45.717878+0000","objects_scrubbed":67,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:49.255280+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":353,"num_read_kb":145683,"num_write":1467,"num_write_kb":63909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":983521,"num_omap_keys":1776,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"48'1573","reported_seq":3450,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:40.157638+0000","last_change":"2024-05-27T00:17:54.309879+0000","last_active":"2024-05-27T00:33:40.157638+0000","last_peered":"2024-05-27T00:33:40.157638+0000","last_clean":"2024-05-27T00:33:40.157638+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:33:40.157638+0000","last_undegraded":"2024-05-27T00:33:40.157638+0000","last_fullsized":"2024-05-27T00:33:40.157638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T00:15:24.567262+0000","last_clean_scrub_stamp":"2024-05-27T00:15:24.567262+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089714449999999998,"stat_sum":{"num_bytes":23597398,"num_objects":590,"num_object_clones":273,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":186,"num_read":53,"num_read_kb":39809,"num_write":1428,"num_write_kb":24946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"48'590","reported_seq":841,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:45.035783+0000","last_change":"2024-05-27T00:33:45.035806+0000","last_active":"2024-05-27T00:33:45.035783+0000","last_peered":"2024-05-27T00:33:45.035783+0000","last_clean":"2024-05-27T00:33:45.035783+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:33:45.035783+0000","last_undegraded":"2024-05-27T00:33:45.035783+0000","last_fullsized":"2024-05-27T00:33:45.035783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'590","last_scrub_stamp":"2024-05-27T00:33:45.035706+0000","last_deep_scrub":"48'590","last_deep_scrub_stamp":"2024-05-27T00:33:45.035706+0000","last_clean_scrub_stamp":"2024-05-27T00:33:45.035706+0000","objects_scrubbed":75,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:35:08.289189+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":327,"num_read_kb":63931,"num_write":705,"num_write_kb":51511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":878537,"num_omap_keys":1434,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1180698353,"num_objects":39309,"num_object_clones":18317,"num_object_copies":78618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39309,"num_whiteouts":12453,"num_read":3379,"num_read_kb":1679737,"num_write":93010,"num_write_kb":1481682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2455085056,"data_stored":2363411938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103652,"ondisk_log_size":103652,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":550495621,"num_objects":2085,"num_object_clones":0,"num_object_copies":4170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2085,"num_whiteouts":0,"num_read":10398,"num_read_kb":4541572,"num_write":34219,"num_write_kb":2258307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1869,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6956090,"num_omap_keys":12268,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1101045760,"data_stored":1100991242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29448,"ondisk_log_size":29448,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542370,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695204,"kb_used_data":555804,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93676636,"statfs":{"total":96636764160,"available":95924875264,"internally_reserved":0,"allocated":569143296,"data_stored":555930767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142735818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542371,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521108,"kb_used_data":391436,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93850732,"statfs":{"total":96636764160,"available":96103149568,"internally_reserved":0,"allocated":400830464,"data_stored":391821208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132774342},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542372,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699692,"kb_used_data":520036,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93672148,"statfs":{"total":96636764160,"available":95920279552,"internally_reserved":0,"allocated":532516864,"data_stored":518806809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":183957957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542370,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640112,"kb_used_data":478504,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":93731728,"statfs":{"total":96636764160,"available":95981289472,"internally_reserved":0,"allocated":489988096,"data_stored":475844387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542371,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609700,"kb_used_data":474460,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93762140,"statfs":{"total":96636764160,"available":96012431360,"internally_reserved":0,"allocated":485847040,"data_stored":471784053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138475975},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542370,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383660,"kb_used_data":276196,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93988180,"statfs":{"total":96636764160,"available":96243896320,"internally_reserved":0,"allocated":282824704,"data_stored":274404683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110033349},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542371,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513116,"kb_used_data":365396,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93858724,"statfs":{"total":96636764160,"available":96111333376,"internally_reserved":0,"allocated":374165504,"data_stored":365965657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151255495},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542370,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564088,"kb_used_data":416368,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93807752,"statfs":{"total":96636764160,"available":96059138048,"internally_reserved":0,"allocated":426360832,"data_stored":413011336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100675584,"data_stored":100663934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":160063488,"data_stored":160055454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62914560,"data_stored":62914560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":142614528,"data_stored":142607008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":117456896,"data_stored":117445139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":147841024,"data_stored":147836556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":168148992,"data_stored":168141417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":201330688,"data_stored":201327174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":324546560,"data_stored":311508911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213557248,"data_stored":205662080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219365376,"data_stored":211242000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":342687744,"data_stored":328928922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":371986432,"data_stored":358151125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":384131072,"data_stored":370722130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":232136704,"data_stored":223431668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":366673920,"data_stored":353765102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:33:48.283 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.283 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.283 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.283 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.284 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.284 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.284 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.284 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.284 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.284 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.285 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.285 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.285 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.285 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.285 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.285 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.285 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.286 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.286 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.286 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.286 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.286 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.286 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.287 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.287 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.287 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.287 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.287 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.287 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.288 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.288 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.288 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.288 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.288 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.289 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.289 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.289 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.289 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.289 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.289 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.290 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.290 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.290 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.290 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.290 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.290 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.291 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.291 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.291 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.291 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.291 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.291 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.292 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.292 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.292 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.292 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.292 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.292 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.293 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.293 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.293 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-27T00:15:13.410459+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=13, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.293 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.293 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.293 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.294 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.294 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.294 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.294 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.294 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.294 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.295 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.295 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.295 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.295 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.295 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.295 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.295 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.296 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.296 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.296 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.296 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.296 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.296 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.297 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.297 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.297 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.297 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-05-27T00:15:23.557969+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=23, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.297 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.297 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-05-27T00:15:24.567262+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=15, tm_sec=24, tm_wday=0, tm_yday=148, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=27, tm_hour=0, tm_min=33, tm_sec=41, tm_wday=0, tm_yday=148, tm_isdst=0) 2024-05-27T00:33:48.298 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-27T00:33:55.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:33:55.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:34:08.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T00:34:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.450+0000 7f314e400640 1 Processor -- start 2024-05-27T00:34:08.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.450+0000 7f314e400640 1 -- start start 2024-05-27T00:34:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.450+0000 7f314e400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31480cbe10 0x7f31481592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.450+0000 7f314e400640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f31480cc1f0 0x7f31480c1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.450+0000 7f314e400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31480c6dd0 con 0x7f3148073570 2024-05-27T00:34:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.450+0000 7f314e400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f3148159810 con 0x7f31480cbe10 2024-05-27T00:34:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.450+0000 7f314e400640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f31480c70b0 con 0x7f31480cc1f0 2024-05-27T00:34:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.450+0000 7f3147400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31480cbe10 0x7f31481592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f3147400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f31480cbe10 0x7f31481592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35260/0 (socket says 172.21.15.98:35260) 2024-05-27T00:34:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f3147400640 1 -- 172.21.15.98:0/2808039619 learned_addr learned my addr 172.21.15.98:0/2808039619 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f313ea00640 1 --2- 172.21.15.98:0/2808039619 >> v2:172.21.15.204:3300/0 conn(0x7f31480cc1f0 0x7f31480c1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f3147400640 1 -- 172.21.15.98:0/2808039619 >> v2:172.21.15.204:3300/0 conn(0x7f31480cc1f0 msgr2=0x7f31480c1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f3147400640 1 --2- 172.21.15.98:0/2808039619 >> v2:172.21.15.204:3300/0 conn(0x7f31480cc1f0 0x7f31480c1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f3147400640 1 -- 172.21.15.98:0/2808039619 >> v1:172.21.15.204:6789/0 conn(0x7f3148073570 legacy=0x7f31481559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:08.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f3147400640 1 -- 172.21.15.98:0/2808039619 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31480c7390 con 0x7f31480cbe10 2024-05-27T00:34:08.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f3147400640 1 --2- 172.21.15.98:0/2808039619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cbe10 0x7f31481592d0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f31480cc6d0 tx=0x7f313801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdda1ca17dee677 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:08.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f3146a00640 1 -- 172.21.15.98:0/2808039619 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31380b5070 con 0x7f31480cbe10 2024-05-27T00:34:08.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f3146a00640 1 -- 172.21.15.98:0/2808039619 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31380a89d0 con 0x7f31480cbe10 2024-05-27T00:34:08.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.454+0000 7f3146a00640 1 -- 172.21.15.98:0/2808039619 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31380a8da0 con 0x7f31480cbe10 2024-05-27T00:34:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 -- 172.21.15.98:0/2808039619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cbe10 msgr2=0x7f31481592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 --2- 172.21.15.98:0/2808039619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cbe10 0x7f31481592d0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f31480cc6d0 tx=0x7f313801ecd0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 -- 172.21.15.98:0/2808039619 shutdown_connections 2024-05-27T00:34:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 --2- 172.21.15.98:0/2808039619 >> v2:172.21.15.204:3300/0 conn(0x7f31480cc1f0 0x7f31480c1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 --2- 172.21.15.98:0/2808039619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cbe10 0x7f31481592d0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 -- 172.21.15.98:0/2808039619 >> 172.21.15.98:0/2808039619 conn(0x7f3148067530 msgr2=0x7f314806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 -- 172.21.15.98:0/2808039619 shutdown_connections 2024-05-27T00:34:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 -- 172.21.15.98:0/2808039619 wait complete. 2024-05-27T00:34:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 Processor -- start 2024-05-27T00:34:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 -- start start 2024-05-27T00:34:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3148073570 0x7f31480afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31480cbe10 0x7f3148071a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cc1f0 0x7f3148071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31480c79d0 con 0x7f31480cc1f0 2024-05-27T00:34:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f31480c1330 con 0x7f3148073570 2024-05-27T00:34:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f314e400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f31480c6990 con 0x7f31480cbe10 2024-05-27T00:34:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f3147e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cc1f0 0x7f3148071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f3147e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cc1f0 0x7f3148071fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35274/0 (socket says 172.21.15.98:35274) 2024-05-27T00:34:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f3147e00640 1 -- 172.21.15.98:0/3831319964 learned_addr learned my addr 172.21.15.98:0/3831319964 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f3147400640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3148073570 0x7f31480afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f313ea00640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31480cbe10 0x7f3148071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f3147e00640 1 -- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31480cbe10 msgr2=0x7f3148071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f3147e00640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31480cbe10 0x7f3148071a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f3147e00640 1 -- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3148073570 msgr2=0x7f31480afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f3147e00640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3148073570 0x7f31480afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f3147e00640 1 -- 172.21.15.98:0/3831319964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31480b0660 con 0x7f31480cc1f0 2024-05-27T00:34:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.458+0000 7f3147400640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3148073570 0x7f31480afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:34:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f3147e00640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cc1f0 0x7f3148071fd0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f314000b970 tx=0x7f314000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f313fe00640 1 -- 172.21.15.98:0/3831319964 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3140019070 con 0x7f31480cc1f0 2024-05-27T00:34:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f313fe00640 1 -- 172.21.15.98:0/3831319964 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f314000cb40 con 0x7f31480cc1f0 2024-05-27T00:34:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f313fe00640 1 -- 172.21.15.98:0/3831319964 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3140014440 con 0x7f31480cc1f0 2024-05-27T00:34:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f314e400640 1 -- 172.21.15.98:0/3831319964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31481f09c0 con 0x7f31480cc1f0 2024-05-27T00:34:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f314e400640 1 -- 172.21.15.98:0/3831319964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3148070fb0 con 0x7f31480cc1f0 2024-05-27T00:34:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f314e400640 1 -- 172.21.15.98:0/3831319964 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3148075cb0 con 0x7f31480cc1f0 2024-05-27T00:34:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f313fe00640 1 -- 172.21.15.98:0/3831319964 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f314001c070 con 0x7f31480cc1f0 2024-05-27T00:34:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f313fe00640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31240678c0 0x7f3124069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f313fe00640 1 -- 172.21.15.98:0/3831319964 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f3140085eb0 con 0x7f31480cc1f0 2024-05-27T00:34:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.462+0000 7f3147400640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31240678c0 0x7f3124069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.466+0000 7f3147400640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31240678c0 0x7f3124069d80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f31380b2000 tx=0x7f3138008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.470+0000 7f313fe00640 1 -- 172.21.15.98:0/3831319964 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f31480b0660 con 0x7f31480cc1f0 2024-05-27T00:34:08.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.626+0000 7f314e400640 1 -- 172.21.15.98:0/3831319964 --> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3148158a80 con 0x7f31240678c0 2024-05-27T00:34:08.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.646+0000 7f313fe00640 1 -- 172.21.15.98:0/3831319964 <== mgr.4105 v2:172.21.15.204:6832/3774298058 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258625 (secure 0 0 0) 0x7f3148158a80 con 0x7f31240678c0 2024-05-27T00:34:08.649 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:34:08.651 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-05-27T00:34:08.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 -- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31240678c0 msgr2=0x7f3124069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:08.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31240678c0 0x7f3124069d80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f31380b2000 tx=0x7f3138008040 comp rx=0 tx=0).stop 2024-05-27T00:34:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 -- 172.21.15.98:0/3831319964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cc1f0 msgr2=0x7f3148071fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cc1f0 0x7f3148071fd0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f314000b970 tx=0x7f314000be30 comp rx=0 tx=0).stop 2024-05-27T00:34:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 -- 172.21.15.98:0/3831319964 shutdown_connections 2024-05-27T00:34:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31480cc1f0 0x7f3148071fd0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31480cbe10 0x7f3148071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f31240678c0 0x7f3124069d80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 --2- 172.21.15.98:0/3831319964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3148073570 0x7f31480afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 -- 172.21.15.98:0/3831319964 >> 172.21.15.98:0/3831319964 conn(0x7f3148067530 msgr2=0x7f314806ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:08.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 -- 172.21.15.98:0/3831319964 shutdown_connections 2024-05-27T00:34:08.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.650+0000 7f314e400640 1 -- 172.21.15.98:0/3831319964 wait complete. 2024-05-27T00:34:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2024-05-27T00:34:07.550156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1736662782,"num_objects":41389,"num_object_clones":18317,"num_object_copies":82778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41389,"num_whiteouts":12400,"num_read":13893,"num_read_kb":6221409,"num_write":127651,"num_write_kb":3747206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1861,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26114489,"num_omap_keys":45815,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172428,"ondisk_log_size":172428,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4698672,"kb_used_data":3487728,"kb_used_omap":12,"kb_used_meta":1210867,"kb_avail":750276048,"statfs":{"total":773094113280,"available":768282673152,"internally_reserved":0,"allocated":3571433472,"data_stored":3477325780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1239928370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1353025,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":-31,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":1333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4574226,"num_omap_keys":8496,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002760"},"pg_stats":[{"pgid":"3.3f","version":"48'2101","reported_seq":4009,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:34:04.402891+0000","last_change":"2024-05-27T00:34:04.402928+0000","last_active":"2024-05-27T00:34:04.402891+0000","last_peered":"2024-05-27T00:34:04.402891+0000","last_clean":"2024-05-27T00:34:04.402891+0000","last_became_active":"2024-05-27T00:15:25.639979+0000","last_became_peered":"2024-05-27T00:15:25.639979+0000","last_unstale":"2024-05-27T00:34:04.402891+0000","last_undegraded":"2024-05-27T00:34:04.402891+0000","last_fullsized":"2024-05-27T00:34:04.402891+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2101","last_scrub_stamp":"2024-05-27T00:34:04.402810+0000","last_deep_scrub":"48'2101","last_deep_scrub_stamp":"2024-05-27T00:34:04.402810+0000","last_clean_scrub_stamp":"2024-05-27T00:34:04.402810+0000","objects_scrubbed":576,"log_size":2101,"log_dups_size":0,"ondisk_log_size":2101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:59:45.869495+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0.027619028,"stat_sum":{"num_bytes":17381879,"num_objects":576,"num_object_clones":268,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":194,"num_read":49,"num_read_kb":31548,"num_write":1379,"num_write_kb":18952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"48'2051","reported_seq":3980,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:55.932162+0000","last_change":"2024-05-27T00:33:55.932162+0000","last_active":"2024-05-27T00:33:55.932162+0000","last_peered":"2024-05-27T00:33:55.932162+0000","last_clean":"2024-05-27T00:33:55.932162+0000","last_became_active":"2024-05-27T00:15:25.595498+0000","last_became_peered":"2024-05-27T00:15:25.595498+0000","last_unstale":"2024-05-27T00:33:55.932162+0000","last_undegraded":"2024-05-27T00:33:55.932162+0000","last_fullsized":"2024-05-27T00:33:55.932162+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2050","last_scrub_stamp":"2024-05-27T00:33:55.931806+0000","last_deep_scrub":"48'2050","last_deep_scrub_stamp":"2024-05-27T00:33:55.931806+0000","last_clean_scrub_stamp":"2024-05-27T00:33:55.931806+0000","objects_scrubbed":549,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:17:06.833962+0000","scrub_duration":1247,"objects_trimmed":0,"snaptrim_duration":0.015469564,"stat_sum":{"num_bytes":18134215,"num_objects":549,"num_object_clones":256,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":168,"num_read":59,"num_read_kb":35851,"num_write":1387,"num_write_kb":20077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"48'2233","reported_seq":4119,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.150405+0000","last_change":"2024-05-27T00:33:46.869273+0000","last_active":"2024-05-27T00:33:59.150405+0000","last_peered":"2024-05-27T00:33:59.150405+0000","last_clean":"2024-05-27T00:33:59.150405+0000","last_became_active":"2024-05-27T00:15:25.593057+0000","last_became_peered":"2024-05-27T00:15:25.593057+0000","last_unstale":"2024-05-27T00:33:59.150405+0000","last_undegraded":"2024-05-27T00:33:59.150405+0000","last_fullsized":"2024-05-27T00:33:59.150405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2231","last_scrub_stamp":"2024-05-27T00:33:46.869156+0000","last_deep_scrub":"48'2231","last_deep_scrub_stamp":"2024-05-27T00:33:46.869156+0000","last_clean_scrub_stamp":"2024-05-27T00:33:46.869156+0000","objects_scrubbed":636,"log_size":2233,"log_dups_size":0,"ondisk_log_size":2233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:28:12.084232+0000","scrub_duration":923,"objects_trimmed":0,"snaptrim_duration":0.018507847000000001,"stat_sum":{"num_bytes":15323750,"num_objects":636,"num_object_clones":298,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":208,"num_read":53,"num_read_kb":23374,"num_write":1462,"num_write_kb":25883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"48'2243","reported_seq":4142,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:56.882134+0000","last_change":"2024-05-27T00:33:56.882193+0000","last_active":"2024-05-27T00:33:56.882134+0000","last_peered":"2024-05-27T00:33:56.882134+0000","last_clean":"2024-05-27T00:33:56.882134+0000","last_became_active":"2024-05-27T00:15:25.618876+0000","last_became_peered":"2024-05-27T00:15:25.618876+0000","last_unstale":"2024-05-27T00:33:56.882134+0000","last_undegraded":"2024-05-27T00:33:56.882134+0000","last_fullsized":"2024-05-27T00:33:56.882134+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2243","last_scrub_stamp":"2024-05-27T00:33:56.882038+0000","last_deep_scrub":"48'2243","last_deep_scrub_stamp":"2024-05-27T00:33:56.882038+0000","last_clean_scrub_stamp":"2024-05-27T00:33:56.882038+0000","objects_scrubbed":623,"log_size":2243,"log_dups_size":0,"ondisk_log_size":2243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:55:12.988105+0000","scrub_duration":1219,"objects_trimmed":0,"snaptrim_duration":0.034338709000000002,"stat_sum":{"num_bytes":10215698,"num_objects":623,"num_object_clones":288,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":199,"num_read":56,"num_read_kb":12442,"num_write":1466,"num_write_kb":16944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"48'2375","reported_seq":4252,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:34:03.818328+0000","last_change":"2024-05-27T00:34:03.818371+0000","last_active":"2024-05-27T00:34:03.818328+0000","last_peered":"2024-05-27T00:34:03.818328+0000","last_clean":"2024-05-27T00:34:03.818328+0000","last_became_active":"2024-05-27T00:15:25.639444+0000","last_became_peered":"2024-05-27T00:15:25.639444+0000","last_unstale":"2024-05-27T00:34:03.818328+0000","last_undegraded":"2024-05-27T00:34:03.818328+0000","last_fullsized":"2024-05-27T00:34:03.818328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2375","last_scrub_stamp":"2024-05-27T00:34:03.818210+0000","last_deep_scrub":"48'2375","last_deep_scrub_stamp":"2024-05-27T00:34:03.818210+0000","last_clean_scrub_stamp":"2024-05-27T00:34:03.818210+0000","objects_scrubbed":672,"log_size":2375,"log_dups_size":0,"ondisk_log_size":2375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:27:41.718366+0000","scrub_duration":1151,"objects_trimmed":0,"snaptrim_duration":0.026116615999999999,"stat_sum":{"num_bytes":20149863,"num_objects":672,"num_object_clones":311,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":219,"num_read":58,"num_read_kb":29919,"num_write":1539,"num_write_kb":21988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"48'2007","reported_seq":3914,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:34:00.329737+0000","last_change":"2024-05-27T00:34:00.329771+0000","last_active":"2024-05-27T00:34:00.329737+0000","last_peered":"2024-05-27T00:34:00.329737+0000","last_clean":"2024-05-27T00:34:00.329737+0000","last_became_active":"2024-05-27T00:15:25.618938+0000","last_became_peered":"2024-05-27T00:15:25.618938+0000","last_unstale":"2024-05-27T00:34:00.329737+0000","last_undegraded":"2024-05-27T00:34:00.329737+0000","last_fullsized":"2024-05-27T00:34:00.329737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2007","last_scrub_stamp":"2024-05-27T00:34:00.329654+0000","last_deep_scrub":"48'2007","last_deep_scrub_stamp":"2024-05-27T00:34:00.329654+0000","last_clean_scrub_stamp":"2024-05-27T00:34:00.329654+0000","objects_scrubbed":549,"log_size":2007,"log_dups_size":0,"ondisk_log_size":2007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:10:28.494890+0000","scrub_duration":683,"objects_trimmed":0,"snaptrim_duration":0.032595461999999999,"stat_sum":{"num_bytes":14563572,"num_objects":549,"num_object_clones":256,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":179,"num_read":38,"num_read_kb":8264,"num_write":1311,"num_write_kb":15921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"48'2181","reported_seq":4082,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.500918+0000","last_change":"2024-05-27T00:33:59.500991+0000","last_active":"2024-05-27T00:33:59.500918+0000","last_peered":"2024-05-27T00:33:59.500918+0000","last_clean":"2024-05-27T00:33:59.500918+0000","last_became_active":"2024-05-27T00:15:25.623223+0000","last_became_peered":"2024-05-27T00:15:25.623223+0000","last_unstale":"2024-05-27T00:33:59.500918+0000","last_undegraded":"2024-05-27T00:33:59.500918+0000","last_fullsized":"2024-05-27T00:33:59.500918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2181","last_scrub_stamp":"2024-05-27T00:33:59.500814+0000","last_deep_scrub":"48'2181","last_deep_scrub_stamp":"2024-05-27T00:33:59.500814+0000","last_clean_scrub_stamp":"2024-05-27T00:33:59.500814+0000","objects_scrubbed":604,"log_size":2181,"log_dups_size":0,"ondisk_log_size":2181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:54:57.882305+0000","scrub_duration":847,"objects_trimmed":0,"snaptrim_duration":0.029344160000000001,"stat_sum":{"num_bytes":19343039,"num_objects":604,"num_object_clones":277,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":190,"num_read":51,"num_read_kb":28747,"num_write":1441,"num_write_kb":22131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"48'2104","reported_seq":4008,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:53.334913+0000","last_change":"2024-05-27T00:33:53.334913+0000","last_active":"2024-05-27T00:33:53.334913+0000","last_peered":"2024-05-27T00:33:53.334913+0000","last_clean":"2024-05-27T00:33:53.334913+0000","last_became_active":"2024-05-27T00:16:06.280009+0000","last_became_peered":"2024-05-27T00:16:06.280009+0000","last_unstale":"2024-05-27T00:33:53.334913+0000","last_undegraded":"2024-05-27T00:33:53.334913+0000","last_fullsized":"2024-05-27T00:33:53.334913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2103","last_scrub_stamp":"2024-05-27T00:33:53.334800+0000","last_deep_scrub":"48'2103","last_deep_scrub_stamp":"2024-05-27T00:33:53.334800+0000","last_clean_scrub_stamp":"2024-05-27T00:33:53.334800+0000","objects_scrubbed":581,"log_size":2104,"log_dups_size":0,"ondisk_log_size":2104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:17:00.915040+0000","scrub_duration":1779,"objects_trimmed":0,"snaptrim_duration":0.012011287000000001,"stat_sum":{"num_bytes":35917582,"num_objects":581,"num_object_clones":269,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":179,"num_read":52,"num_read_kb":60147,"num_write":1388,"num_write_kb":38879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.37","version":"48'2258","reported_seq":4147,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.147421+0000","last_change":"2024-05-27T00:33:54.272752+0000","last_active":"2024-05-27T00:33:59.147421+0000","last_peered":"2024-05-27T00:33:59.147421+0000","last_clean":"2024-05-27T00:33:59.147421+0000","last_became_active":"2024-05-27T00:16:06.280544+0000","last_became_peered":"2024-05-27T00:16:06.280544+0000","last_unstale":"2024-05-27T00:33:59.147421+0000","last_undegraded":"2024-05-27T00:33:59.147421+0000","last_fullsized":"2024-05-27T00:33:59.147421+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2256","last_scrub_stamp":"2024-05-27T00:33:54.272460+0000","last_deep_scrub":"48'2256","last_deep_scrub_stamp":"2024-05-27T00:33:54.272460+0000","last_clean_scrub_stamp":"2024-05-27T00:33:54.272460+0000","objects_scrubbed":636,"log_size":2258,"log_dups_size":0,"ondisk_log_size":2258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:11:56.943437+0000","scrub_duration":1755,"objects_trimmed":0,"snaptrim_duration":0.0096681779999999995,"stat_sum":{"num_bytes":20317810,"num_objects":636,"num_object_clones":293,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":196,"num_read":60,"num_read_kb":29133,"num_write":1484,"num_write_kb":26137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"48'2115","reported_seq":4025,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:58.745845+0000","last_change":"2024-05-27T00:33:58.745845+0000","last_active":"2024-05-27T00:33:58.745845+0000","last_peered":"2024-05-27T00:33:58.745845+0000","last_clean":"2024-05-27T00:33:58.745845+0000","last_became_active":"2024-05-27T00:15:25.629143+0000","last_became_peered":"2024-05-27T00:15:25.629143+0000","last_unstale":"2024-05-27T00:33:58.745845+0000","last_undegraded":"2024-05-27T00:33:58.745845+0000","last_fullsized":"2024-05-27T00:33:58.745845+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2114","last_scrub_stamp":"2024-05-27T00:33:58.745677+0000","last_deep_scrub":"48'2114","last_deep_scrub_stamp":"2024-05-27T00:33:58.745677+0000","last_clean_scrub_stamp":"2024-05-27T00:33:58.745677+0000","objects_scrubbed":591,"log_size":2115,"log_dups_size":0,"ondisk_log_size":2115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:33:17.487083+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0.027993297,"stat_sum":{"num_bytes":14791344,"num_objects":591,"num_object_clones":272,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":185,"num_read":55,"num_read_kb":22038,"num_write":1390,"num_write_kb":16506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"48'2282","reported_seq":4148,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:56.077785+0000","last_change":"2024-05-27T00:33:56.077867+0000","last_active":"2024-05-27T00:33:56.077785+0000","last_peered":"2024-05-27T00:33:56.077785+0000","last_clean":"2024-05-27T00:33:56.077785+0000","last_became_active":"2024-05-27T00:15:25.614123+0000","last_became_peered":"2024-05-27T00:15:25.614123+0000","last_unstale":"2024-05-27T00:33:56.077785+0000","last_undegraded":"2024-05-27T00:33:56.077785+0000","last_fullsized":"2024-05-27T00:33:56.077785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2282","last_scrub_stamp":"2024-05-27T00:33:56.077237+0000","last_deep_scrub":"48'2282","last_deep_scrub_stamp":"2024-05-27T00:33:56.077237+0000","last_clean_scrub_stamp":"2024-05-27T00:33:56.077237+0000","objects_scrubbed":660,"log_size":2282,"log_dups_size":0,"ondisk_log_size":2282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:24.801666+0000","scrub_duration":1191,"objects_trimmed":0,"snaptrim_duration":0.027428447000000002,"stat_sum":{"num_bytes":10662153,"num_objects":660,"num_object_clones":308,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":660,"num_whiteouts":201,"num_read":45,"num_read_kb":12410,"num_write":1488,"num_write_kb":11561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"48'2252","reported_seq":4113,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:52.049922+0000","last_change":"2024-05-27T00:33:52.049972+0000","last_active":"2024-05-27T00:33:52.049922+0000","last_peered":"2024-05-27T00:33:52.049922+0000","last_clean":"2024-05-27T00:33:52.049922+0000","last_became_active":"2024-05-27T00:15:25.623213+0000","last_became_peered":"2024-05-27T00:15:25.623213+0000","last_unstale":"2024-05-27T00:33:52.049922+0000","last_undegraded":"2024-05-27T00:33:52.049922+0000","last_fullsized":"2024-05-27T00:33:52.049922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2252","last_scrub_stamp":"2024-05-27T00:33:52.049840+0000","last_deep_scrub":"48'2252","last_deep_scrub_stamp":"2024-05-27T00:33:52.049840+0000","last_clean_scrub_stamp":"2024-05-27T00:33:52.049840+0000","objects_scrubbed":642,"log_size":2252,"log_dups_size":0,"ondisk_log_size":2252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:49.897988+0000","scrub_duration":1055,"objects_trimmed":0,"snaptrim_duration":0.021724307000000002,"stat_sum":{"num_bytes":19461089,"num_objects":642,"num_object_clones":300,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":199,"num_read":31,"num_read_kb":14211,"num_write":1479,"num_write_kb":31965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"48'2172","reported_seq":4060,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:56.177585+0000","last_change":"2024-05-27T00:33:56.177585+0000","last_active":"2024-05-27T00:33:56.177585+0000","last_peered":"2024-05-27T00:33:56.177585+0000","last_clean":"2024-05-27T00:33:56.177585+0000","last_became_active":"2024-05-27T00:15:25.615163+0000","last_became_peered":"2024-05-27T00:15:25.615163+0000","last_unstale":"2024-05-27T00:33:56.177585+0000","last_undegraded":"2024-05-27T00:33:56.177585+0000","last_fullsized":"2024-05-27T00:33:56.177585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2171","last_scrub_stamp":"2024-05-27T00:33:56.177475+0000","last_deep_scrub":"48'2171","last_deep_scrub_stamp":"2024-05-27T00:33:56.177475+0000","last_clean_scrub_stamp":"2024-05-27T00:33:56.177475+0000","objects_scrubbed":622,"log_size":2172,"log_dups_size":0,"ondisk_log_size":2172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:22:23.425163+0000","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0.0089222539999999993,"stat_sum":{"num_bytes":26903344,"num_objects":622,"num_object_clones":291,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":208,"num_read":51,"num_read_kb":32873,"num_write":1381,"num_write_kb":36946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"48'2194","reported_seq":4079,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.151933+0000","last_change":"2024-05-27T00:33:47.955931+0000","last_active":"2024-05-27T00:33:59.151933+0000","last_peered":"2024-05-27T00:33:59.151933+0000","last_clean":"2024-05-27T00:33:59.151933+0000","last_became_active":"2024-05-27T00:15:25.593701+0000","last_became_peered":"2024-05-27T00:15:25.593701+0000","last_unstale":"2024-05-27T00:33:59.151933+0000","last_undegraded":"2024-05-27T00:33:59.151933+0000","last_fullsized":"2024-05-27T00:33:59.151933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2192","last_scrub_stamp":"2024-05-27T00:33:47.955821+0000","last_deep_scrub":"48'2192","last_deep_scrub_stamp":"2024-05-27T00:33:47.955821+0000","last_clean_scrub_stamp":"2024-05-27T00:33:47.955821+0000","objects_scrubbed":616,"log_size":2194,"log_dups_size":0,"ondisk_log_size":2194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:40:20.232809+0000","scrub_duration":967,"objects_trimmed":0,"snaptrim_duration":0.016842227000000001,"stat_sum":{"num_bytes":10587319,"num_objects":616,"num_object_clones":288,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":203,"num_read":44,"num_read_kb":16487,"num_write":1438,"num_write_kb":17079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"48'2229","reported_seq":4130,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:56.812783+0000","last_change":"2024-05-27T00:33:56.812836+0000","last_active":"2024-05-27T00:33:56.812783+0000","last_peered":"2024-05-27T00:33:56.812783+0000","last_clean":"2024-05-27T00:33:56.812783+0000","last_became_active":"2024-05-27T00:15:25.611445+0000","last_became_peered":"2024-05-27T00:15:25.611445+0000","last_unstale":"2024-05-27T00:33:56.812783+0000","last_undegraded":"2024-05-27T00:33:56.812783+0000","last_fullsized":"2024-05-27T00:33:56.812783+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2229","last_scrub_stamp":"2024-05-27T00:33:56.812711+0000","last_deep_scrub":"48'2229","last_deep_scrub_stamp":"2024-05-27T00:33:56.812711+0000","last_clean_scrub_stamp":"2024-05-27T00:33:56.812711+0000","objects_scrubbed":609,"log_size":2229,"log_dups_size":0,"ondisk_log_size":2229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:29:08.640888+0000","scrub_duration":1099,"objects_trimmed":0,"snaptrim_duration":0.016821815,"stat_sum":{"num_bytes":23057201,"num_objects":609,"num_object_clones":283,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":212,"num_read":56,"num_read_kb":39458,"num_write":1462,"num_write_kb":27814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"48'2200","reported_seq":4103,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:53.145733+0000","last_change":"2024-05-27T00:33:53.145818+0000","last_active":"2024-05-27T00:33:53.145733+0000","last_peered":"2024-05-27T00:33:53.145733+0000","last_clean":"2024-05-27T00:33:53.145733+0000","last_became_active":"2024-05-27T00:15:25.610452+0000","last_became_peered":"2024-05-27T00:15:25.610452+0000","last_unstale":"2024-05-27T00:33:53.145733+0000","last_undegraded":"2024-05-27T00:33:53.145733+0000","last_fullsized":"2024-05-27T00:33:53.145733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2200","last_scrub_stamp":"2024-05-27T00:33:53.145640+0000","last_deep_scrub":"48'2200","last_deep_scrub_stamp":"2024-05-27T00:33:53.145640+0000","last_clean_scrub_stamp":"2024-05-27T00:33:53.145640+0000","objects_scrubbed":605,"log_size":2200,"log_dups_size":0,"ondisk_log_size":2200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:05:20.681579+0000","scrub_duration":1495,"objects_trimmed":0,"snaptrim_duration":0.031092022,"stat_sum":{"num_bytes":14535849,"num_objects":605,"num_object_clones":288,"num_object_copies":1210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":605,"num_whiteouts":200,"num_read":63,"num_read_kb":16619,"num_write":1430,"num_write_kb":20418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"48'2369","reported_seq":4273,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:34:02.631820+0000","last_change":"2024-05-27T00:34:02.631859+0000","last_active":"2024-05-27T00:34:02.631820+0000","last_peered":"2024-05-27T00:34:02.631820+0000","last_clean":"2024-05-27T00:34:02.631820+0000","last_became_active":"2024-05-27T00:15:25.631276+0000","last_became_peered":"2024-05-27T00:15:25.631276+0000","last_unstale":"2024-05-27T00:34:02.631820+0000","last_undegraded":"2024-05-27T00:34:02.631820+0000","last_fullsized":"2024-05-27T00:34:02.631820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2369","last_scrub_stamp":"2024-05-27T00:34:02.631744+0000","last_deep_scrub":"48'2369","last_deep_scrub_stamp":"2024-05-27T00:34:02.631744+0000","last_clean_scrub_stamp":"2024-05-27T00:34:02.631744+0000","objects_scrubbed":654,"log_size":2369,"log_dups_size":0,"ondisk_log_size":2369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:31:55.262023+0000","scrub_duration":951,"objects_trimmed":0,"snaptrim_duration":0.034311531999999999,"stat_sum":{"num_bytes":28217641,"num_objects":654,"num_object_clones":304,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":203,"num_read":78,"num_read_kb":63526,"num_write":1566,"num_write_kb":34081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"48'2098","reported_seq":4001,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:51.756685+0000","last_change":"2024-05-27T00:33:51.756685+0000","last_active":"2024-05-27T00:33:51.756685+0000","last_peered":"2024-05-27T00:33:51.756685+0000","last_clean":"2024-05-27T00:33:51.756685+0000","last_became_active":"2024-05-27T00:15:25.615217+0000","last_became_peered":"2024-05-27T00:15:25.615217+0000","last_unstale":"2024-05-27T00:33:51.756685+0000","last_undegraded":"2024-05-27T00:33:51.756685+0000","last_fullsized":"2024-05-27T00:33:51.756685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2097","last_scrub_stamp":"2024-05-27T00:33:51.756488+0000","last_deep_scrub":"48'2097","last_deep_scrub_stamp":"2024-05-27T00:33:51.756488+0000","last_clean_scrub_stamp":"2024-05-27T00:33:51.756488+0000","objects_scrubbed":590,"log_size":2098,"log_dups_size":0,"ondisk_log_size":2098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:26:53.718081+0000","scrub_duration":1183,"objects_trimmed":0,"snaptrim_duration":0.010480461999999999,"stat_sum":{"num_bytes":6663218,"num_objects":590,"num_object_clones":271,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":187,"num_read":47,"num_read_kb":139,"num_write":1378,"num_write_kb":8113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"48'2323","reported_seq":4196,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:49.771569+0000","last_change":"2024-05-27T00:33:49.771667+0000","last_active":"2024-05-27T00:33:49.771569+0000","last_peered":"2024-05-27T00:33:49.771569+0000","last_clean":"2024-05-27T00:33:49.771569+0000","last_became_active":"2024-05-27T00:15:25.628361+0000","last_became_peered":"2024-05-27T00:15:25.628361+0000","last_unstale":"2024-05-27T00:33:49.771569+0000","last_undegraded":"2024-05-27T00:33:49.771569+0000","last_fullsized":"2024-05-27T00:33:49.771569+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2323","last_scrub_stamp":"2024-05-27T00:33:49.771460+0000","last_deep_scrub":"48'2323","last_deep_scrub_stamp":"2024-05-27T00:33:49.771460+0000","last_clean_scrub_stamp":"2024-05-27T00:33:49.771460+0000","objects_scrubbed":657,"log_size":2323,"log_dups_size":0,"ondisk_log_size":2323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:27:59.077254+0000","scrub_duration":1175,"objects_trimmed":0,"snaptrim_duration":0.029916169999999999,"stat_sum":{"num_bytes":16074820,"num_objects":657,"num_object_clones":311,"num_object_copies":1314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":208,"num_read":54,"num_read_kb":28815,"num_write":1513,"num_write_kb":21031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"48'2219","reported_seq":4138,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.158752+0000","last_change":"2024-05-27T00:33:57.032097+0000","last_active":"2024-05-27T00:33:59.158752+0000","last_peered":"2024-05-27T00:33:59.158752+0000","last_clean":"2024-05-27T00:33:59.158752+0000","last_became_active":"2024-05-27T00:15:25.618660+0000","last_became_peered":"2024-05-27T00:15:25.618660+0000","last_unstale":"2024-05-27T00:33:59.158752+0000","last_undegraded":"2024-05-27T00:33:59.158752+0000","last_fullsized":"2024-05-27T00:33:59.158752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2216","last_scrub_stamp":"2024-05-27T00:33:57.031928+0000","last_deep_scrub":"48'2216","last_deep_scrub_stamp":"2024-05-27T00:33:57.031928+0000","last_clean_scrub_stamp":"2024-05-27T00:33:57.031928+0000","objects_scrubbed":614,"log_size":2219,"log_dups_size":0,"ondisk_log_size":2219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:51:48.641364+0000","scrub_duration":1107,"objects_trimmed":0,"snaptrim_duration":0.027334256000000001,"stat_sum":{"num_bytes":31335125,"num_objects":614,"num_object_clones":281,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":185,"num_read":70,"num_read_kb":56922,"num_write":1474,"num_write_kb":40285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"48'2229","reported_seq":4128,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:55.708181+0000","last_change":"2024-05-27T00:33:55.708181+0000","last_active":"2024-05-27T00:33:55.708181+0000","last_peered":"2024-05-27T00:33:55.708181+0000","last_clean":"2024-05-27T00:33:55.708181+0000","last_became_active":"2024-05-27T00:15:25.616046+0000","last_became_peered":"2024-05-27T00:15:25.616046+0000","last_unstale":"2024-05-27T00:33:55.708181+0000","last_undegraded":"2024-05-27T00:33:55.708181+0000","last_fullsized":"2024-05-27T00:33:55.708181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2228","last_scrub_stamp":"2024-05-27T00:33:55.708023+0000","last_deep_scrub":"48'2228","last_deep_scrub_stamp":"2024-05-27T00:33:55.708023+0000","last_clean_scrub_stamp":"2024-05-27T00:33:55.708023+0000","objects_scrubbed":618,"log_size":2229,"log_dups_size":0,"ondisk_log_size":2229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:17:31.024775+0000","scrub_duration":1539,"objects_trimmed":0,"snaptrim_duration":0.017108005999999999,"stat_sum":{"num_bytes":14822117,"num_objects":618,"num_object_clones":289,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":191,"num_read":59,"num_read_kb":12448,"num_write":1494,"num_write_kb":20704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"48'2350","reported_seq":4220,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:34:01.686153+0000","last_change":"2024-05-27T00:34:01.686205+0000","last_active":"2024-05-27T00:34:01.686153+0000","last_peered":"2024-05-27T00:34:01.686153+0000","last_clean":"2024-05-27T00:34:01.686153+0000","last_became_active":"2024-05-27T00:15:25.628697+0000","last_became_peered":"2024-05-27T00:15:25.628697+0000","last_unstale":"2024-05-27T00:34:01.686153+0000","last_undegraded":"2024-05-27T00:34:01.686153+0000","last_fullsized":"2024-05-27T00:34:01.686153+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2350","last_scrub_stamp":"2024-05-27T00:34:01.686069+0000","last_deep_scrub":"48'2350","last_deep_scrub_stamp":"2024-05-27T00:34:01.686069+0000","last_clean_scrub_stamp":"2024-05-27T00:34:01.686069+0000","objects_scrubbed":664,"log_size":2350,"log_dups_size":0,"ondisk_log_size":2350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:31:00.970718+0000","scrub_duration":1015,"objects_trimmed":0,"snaptrim_duration":0.033862588999999998,"stat_sum":{"num_bytes":19005190,"num_objects":664,"num_object_clones":315,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":221,"num_read":54,"num_read_kb":29811,"num_write":1500,"num_write_kb":24710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"48'2105","reported_seq":4013,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:54.450369+0000","last_change":"2024-05-27T00:33:54.450369+0000","last_active":"2024-05-27T00:33:54.450369+0000","last_peered":"2024-05-27T00:33:54.450369+0000","last_clean":"2024-05-27T00:33:54.450369+0000","last_became_active":"2024-05-27T00:15:25.610446+0000","last_became_peered":"2024-05-27T00:15:25.610446+0000","last_unstale":"2024-05-27T00:33:54.450369+0000","last_undegraded":"2024-05-27T00:33:54.450369+0000","last_fullsized":"2024-05-27T00:33:54.450369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2104","last_scrub_stamp":"2024-05-27T00:33:54.450206+0000","last_deep_scrub":"48'2104","last_deep_scrub_stamp":"2024-05-27T00:33:54.450206+0000","last_clean_scrub_stamp":"2024-05-27T00:33:54.450206+0000","objects_scrubbed":578,"log_size":2105,"log_dups_size":0,"ondisk_log_size":2105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:58:04.668925+0000","scrub_duration":1827,"objects_trimmed":0,"snaptrim_duration":0.0088470660000000007,"stat_sum":{"num_bytes":9616087,"num_objects":578,"num_object_clones":267,"num_object_copies":1156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":178,"num_read":48,"num_read_kb":13982,"num_write":1425,"num_write_kb":15666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"48'2154","reported_seq":4080,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:54.291185+0000","last_change":"2024-05-27T00:33:54.291185+0000","last_active":"2024-05-27T00:33:54.291185+0000","last_peered":"2024-05-27T00:33:54.291185+0000","last_clean":"2024-05-27T00:33:54.291185+0000","last_became_active":"2024-05-27T00:15:25.612302+0000","last_became_peered":"2024-05-27T00:15:25.612302+0000","last_unstale":"2024-05-27T00:33:54.291185+0000","last_undegraded":"2024-05-27T00:33:54.291185+0000","last_fullsized":"2024-05-27T00:33:54.291185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2153","last_scrub_stamp":"2024-05-27T00:33:54.290997+0000","last_deep_scrub":"48'2153","last_deep_scrub_stamp":"2024-05-27T00:33:54.290997+0000","last_clean_scrub_stamp":"2024-05-27T00:33:54.290997+0000","objects_scrubbed":588,"log_size":2154,"log_dups_size":0,"ondisk_log_size":2154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:13:16.241132+0000","scrub_duration":1707,"objects_trimmed":0,"snaptrim_duration":0.033457270999999997,"stat_sum":{"num_bytes":30464589,"num_objects":588,"num_object_clones":269,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":174,"num_read":68,"num_read_kb":58155,"num_write":1448,"num_write_kb":31953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"48'2166","reported_seq":4059,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:53.001330+0000","last_change":"2024-05-27T00:33:53.001330+0000","last_active":"2024-05-27T00:33:53.001330+0000","last_peered":"2024-05-27T00:33:53.001330+0000","last_clean":"2024-05-27T00:33:53.001330+0000","last_became_active":"2024-05-27T00:15:25.609676+0000","last_became_peered":"2024-05-27T00:15:25.609676+0000","last_unstale":"2024-05-27T00:33:53.001330+0000","last_undegraded":"2024-05-27T00:33:53.001330+0000","last_fullsized":"2024-05-27T00:33:53.001330+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2165","last_scrub_stamp":"2024-05-27T00:33:53.001125+0000","last_deep_scrub":"48'2165","last_deep_scrub_stamp":"2024-05-27T00:33:53.001125+0000","last_clean_scrub_stamp":"2024-05-27T00:33:53.001125+0000","objects_scrubbed":608,"log_size":2166,"log_dups_size":0,"ondisk_log_size":2166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:24:01.833032+0000","scrub_duration":1371,"objects_trimmed":0,"snaptrim_duration":0.0089651499999999999,"stat_sum":{"num_bytes":14724653,"num_objects":608,"num_object_clones":285,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":192,"num_read":49,"num_read_kb":20561,"num_write":1437,"num_write_kb":17383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"48'2305","reported_seq":4201,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.906540+0000","last_change":"2024-05-27T00:33:59.906594+0000","last_active":"2024-05-27T00:33:59.906540+0000","last_peered":"2024-05-27T00:33:59.906540+0000","last_clean":"2024-05-27T00:33:59.906540+0000","last_became_active":"2024-05-27T00:15:25.632771+0000","last_became_peered":"2024-05-27T00:15:25.632771+0000","last_unstale":"2024-05-27T00:33:59.906540+0000","last_undegraded":"2024-05-27T00:33:59.906540+0000","last_fullsized":"2024-05-27T00:33:59.906540+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2305","last_scrub_stamp":"2024-05-27T00:33:59.906463+0000","last_deep_scrub":"48'2305","last_deep_scrub_stamp":"2024-05-27T00:33:59.906463+0000","last_clean_scrub_stamp":"2024-05-27T00:33:59.906463+0000","objects_scrubbed":632,"log_size":2305,"log_dups_size":0,"ondisk_log_size":2305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:45:15.373225+0000","scrub_duration":959,"objects_trimmed":0,"snaptrim_duration":0.024583906999999999,"stat_sum":{"num_bytes":23745982,"num_objects":632,"num_object_clones":288,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":194,"num_read":55,"num_read_kb":32562,"num_write":1530,"num_write_kb":26486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"48'2100","reported_seq":4005,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:55.308337+0000","last_change":"2024-05-27T00:33:55.308337+0000","last_active":"2024-05-27T00:33:55.308337+0000","last_peered":"2024-05-27T00:33:55.308337+0000","last_clean":"2024-05-27T00:33:55.308337+0000","last_became_active":"2024-05-27T00:15:25.597207+0000","last_became_peered":"2024-05-27T00:15:25.597207+0000","last_unstale":"2024-05-27T00:33:55.308337+0000","last_undegraded":"2024-05-27T00:33:55.308337+0000","last_fullsized":"2024-05-27T00:33:55.308337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2099","last_scrub_stamp":"2024-05-27T00:33:55.308184+0000","last_deep_scrub":"48'2099","last_deep_scrub_stamp":"2024-05-27T00:33:55.308184+0000","last_clean_scrub_stamp":"2024-05-27T00:33:55.308184+0000","objects_scrubbed":577,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:38:27.318665+0000","scrub_duration":1591,"objects_trimmed":0,"snaptrim_duration":0.016722006000000001,"stat_sum":{"num_bytes":10475874,"num_objects":577,"num_object_clones":269,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":179,"num_read":47,"num_read_kb":12352,"num_write":1390,"num_write_kb":12467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"48'2179","reported_seq":4071,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:55.445821+0000","last_change":"2024-05-27T00:33:55.445912+0000","last_active":"2024-05-27T00:33:55.445821+0000","last_peered":"2024-05-27T00:33:55.445821+0000","last_clean":"2024-05-27T00:33:55.445821+0000","last_became_active":"2024-05-27T00:15:25.610548+0000","last_became_peered":"2024-05-27T00:15:25.610548+0000","last_unstale":"2024-05-27T00:33:55.445821+0000","last_undegraded":"2024-05-27T00:33:55.445821+0000","last_fullsized":"2024-05-27T00:33:55.445821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2179","last_scrub_stamp":"2024-05-27T00:33:55.445724+0000","last_deep_scrub":"48'2179","last_deep_scrub_stamp":"2024-05-27T00:33:55.445724+0000","last_clean_scrub_stamp":"2024-05-27T00:33:55.445724+0000","objects_scrubbed":611,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:54:32.119741+0000","scrub_duration":1831,"objects_trimmed":0,"snaptrim_duration":0.0089478649999999993,"stat_sum":{"num_bytes":10395394,"num_objects":611,"num_object_clones":285,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":192,"num_read":49,"num_read_kb":15706,"num_write":1435,"num_write_kb":15502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"48'2374","reported_seq":4228,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:50.400592+0000","last_change":"2024-05-27T00:33:50.400654+0000","last_active":"2024-05-27T00:33:50.400592+0000","last_peered":"2024-05-27T00:33:50.400592+0000","last_clean":"2024-05-27T00:33:50.400592+0000","last_became_active":"2024-05-27T00:15:25.597977+0000","last_became_peered":"2024-05-27T00:15:25.597977+0000","last_unstale":"2024-05-27T00:33:50.400592+0000","last_undegraded":"2024-05-27T00:33:50.400592+0000","last_fullsized":"2024-05-27T00:33:50.400592+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2374","last_scrub_stamp":"2024-05-27T00:33:50.400487+0000","last_deep_scrub":"48'2374","last_deep_scrub_stamp":"2024-05-27T00:33:50.400487+0000","last_clean_scrub_stamp":"2024-05-27T00:33:50.400487+0000","objects_scrubbed":674,"log_size":2374,"log_dups_size":0,"ondisk_log_size":2374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:17:50.643884+0000","scrub_duration":1423,"objects_trimmed":0,"snaptrim_duration":0.017911486000000001,"stat_sum":{"num_bytes":10165450,"num_objects":674,"num_object_clones":317,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":212,"num_read":40,"num_read_kb":1060,"num_write":1542,"num_write_kb":12604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"48'2205","reported_seq":4107,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:57.499210+0000","last_change":"2024-05-27T00:33:57.499306+0000","last_active":"2024-05-27T00:33:57.499210+0000","last_peered":"2024-05-27T00:33:57.499210+0000","last_clean":"2024-05-27T00:33:57.499210+0000","last_became_active":"2024-05-27T00:15:25.612773+0000","last_became_peered":"2024-05-27T00:15:25.612773+0000","last_unstale":"2024-05-27T00:33:57.499210+0000","last_undegraded":"2024-05-27T00:33:57.499210+0000","last_fullsized":"2024-05-27T00:33:57.499210+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2205","last_scrub_stamp":"2024-05-27T00:33:57.499131+0000","last_deep_scrub":"48'2205","last_deep_scrub_stamp":"2024-05-27T00:33:57.499131+0000","last_clean_scrub_stamp":"2024-05-27T00:33:57.499131+0000","objects_scrubbed":608,"log_size":2205,"log_dups_size":0,"ondisk_log_size":2205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:13:16.436476+0000","scrub_duration":875,"objects_trimmed":0,"snaptrim_duration":0.033387499000000001,"stat_sum":{"num_bytes":19049177,"num_objects":608,"num_object_clones":281,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":187,"num_read":54,"num_read_kb":24794,"num_write":1464,"num_write_kb":20553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"48'2266","reported_seq":4177,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.148383+0000","last_change":"2024-05-27T00:33:50.992495+0000","last_active":"2024-05-27T00:33:59.148383+0000","last_peered":"2024-05-27T00:33:59.148383+0000","last_clean":"2024-05-27T00:33:59.148383+0000","last_became_active":"2024-05-27T00:15:25.605606+0000","last_became_peered":"2024-05-27T00:15:25.605606+0000","last_unstale":"2024-05-27T00:33:59.148383+0000","last_undegraded":"2024-05-27T00:33:59.148383+0000","last_fullsized":"2024-05-27T00:33:59.148383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2264","last_scrub_stamp":"2024-05-27T00:33:50.992331+0000","last_deep_scrub":"48'2264","last_deep_scrub_stamp":"2024-05-27T00:33:50.992331+0000","last_clean_scrub_stamp":"2024-05-27T00:33:50.992331+0000","objects_scrubbed":606,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:30:11.884881+0000","scrub_duration":1407,"objects_trimmed":0,"snaptrim_duration":0.009527878,"stat_sum":{"num_bytes":28018878,"num_objects":606,"num_object_clones":279,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":192,"num_read":62,"num_read_kb":41489,"num_write":1524,"num_write_kb":37283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"48'2150","reported_seq":4078,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:54.556737+0000","last_change":"2024-05-27T00:33:54.556785+0000","last_active":"2024-05-27T00:33:54.556737+0000","last_peered":"2024-05-27T00:33:54.556737+0000","last_clean":"2024-05-27T00:33:54.556737+0000","last_became_active":"2024-05-27T00:15:25.606408+0000","last_became_peered":"2024-05-27T00:15:25.606408+0000","last_unstale":"2024-05-27T00:33:54.556737+0000","last_undegraded":"2024-05-27T00:33:54.556737+0000","last_fullsized":"2024-05-27T00:33:54.556737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2150","last_scrub_stamp":"2024-05-27T00:33:54.556672+0000","last_deep_scrub":"48'2150","last_deep_scrub_stamp":"2024-05-27T00:33:54.556672+0000","last_clean_scrub_stamp":"2024-05-27T00:33:54.556672+0000","objects_scrubbed":577,"log_size":2150,"log_dups_size":0,"ondisk_log_size":2150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:26:11.028242+0000","scrub_duration":1707,"objects_trimmed":0,"snaptrim_duration":0.025435675000000001,"stat_sum":{"num_bytes":18002918,"num_objects":577,"num_object_clones":264,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":183,"num_read":65,"num_read_kb":34529,"num_write":1449,"num_write_kb":21047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"48'2273","reported_seq":4154,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:50.315662+0000","last_change":"2024-05-27T00:33:50.315709+0000","last_active":"2024-05-27T00:33:50.315662+0000","last_peered":"2024-05-27T00:33:50.315662+0000","last_clean":"2024-05-27T00:33:50.315662+0000","last_became_active":"2024-05-27T00:15:25.610932+0000","last_became_peered":"2024-05-27T00:15:25.610932+0000","last_unstale":"2024-05-27T00:33:50.315662+0000","last_undegraded":"2024-05-27T00:33:50.315662+0000","last_fullsized":"2024-05-27T00:33:50.315662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2273","last_scrub_stamp":"2024-05-27T00:33:50.315566+0000","last_deep_scrub":"48'2273","last_deep_scrub_stamp":"2024-05-27T00:33:50.315566+0000","last_clean_scrub_stamp":"2024-05-27T00:33:50.315566+0000","objects_scrubbed":645,"log_size":2273,"log_dups_size":0,"ondisk_log_size":2273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:27:27.605306+0000","scrub_duration":1315,"objects_trimmed":0,"snaptrim_duration":0.019320113,"stat_sum":{"num_bytes":27021070,"num_objects":645,"num_object_clones":298,"num_object_copies":1290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":203,"num_read":50,"num_read_kb":43416,"num_write":1510,"num_write_kb":28409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"48'1158","reported_seq":1385,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:46.821700+0000","last_change":"2024-05-27T00:33:46.821751+0000","last_active":"2024-05-27T00:33:46.821700+0000","last_peered":"2024-05-27T00:33:46.821700+0000","last_clean":"2024-05-27T00:33:46.821700+0000","last_became_active":"2024-05-27T00:15:24.610805+0000","last_became_peered":"2024-05-27T00:15:24.610805+0000","last_unstale":"2024-05-27T00:33:46.821700+0000","last_undegraded":"2024-05-27T00:33:46.821700+0000","last_fullsized":"2024-05-27T00:33:46.821700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1158","last_scrub_stamp":"2024-05-27T00:33:46.821286+0000","last_deep_scrub":"48'1158","last_deep_scrub_stamp":"2024-05-27T00:33:46.821286+0000","last_clean_scrub_stamp":"2024-05-27T00:33:46.821286+0000","objects_scrubbed":65,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:54:12.845783+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":240,"num_read_kb":174989,"num_write":1204,"num_write_kb":92121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":696499,"num_omap_keys":1222,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"48'2194","reported_seq":4093,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:54.979144+0000","last_change":"2024-05-27T00:33:54.979204+0000","last_active":"2024-05-27T00:33:54.979144+0000","last_peered":"2024-05-27T00:33:54.979144+0000","last_clean":"2024-05-27T00:33:54.979144+0000","last_became_active":"2024-05-27T00:15:25.613841+0000","last_became_peered":"2024-05-27T00:15:25.613841+0000","last_unstale":"2024-05-27T00:33:54.979144+0000","last_undegraded":"2024-05-27T00:33:54.979144+0000","last_fullsized":"2024-05-27T00:33:54.979144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2194","last_scrub_stamp":"2024-05-27T00:33:54.979055+0000","last_deep_scrub":"48'2194","last_deep_scrub_stamp":"2024-05-27T00:33:54.979055+0000","last_clean_scrub_stamp":"2024-05-27T00:33:54.979055+0000","objects_scrubbed":620,"log_size":2194,"log_dups_size":0,"ondisk_log_size":2194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:35:06.673081+0000","scrub_duration":1851,"objects_trimmed":0,"snaptrim_duration":0.017111721,"stat_sum":{"num_bytes":21987705,"num_objects":620,"num_object_clones":286,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":197,"num_read":57,"num_read_kb":45093,"num_write":1417,"num_write_kb":24321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"48'911","reported_seq":1166,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:50.083126+0000","last_change":"2024-05-27T00:33:50.083231+0000","last_active":"2024-05-27T00:33:50.083126+0000","last_peered":"2024-05-27T00:33:50.083126+0000","last_clean":"2024-05-27T00:33:50.083126+0000","last_became_active":"2024-05-27T00:15:24.622876+0000","last_became_peered":"2024-05-27T00:15:24.622876+0000","last_unstale":"2024-05-27T00:33:50.083126+0000","last_undegraded":"2024-05-27T00:33:50.083126+0000","last_fullsized":"2024-05-27T00:33:50.083126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'911","last_scrub_stamp":"2024-05-27T00:33:50.083000+0000","last_deep_scrub":"48'911","last_deep_scrub_stamp":"2024-05-27T00:33:50.083000+0000","last_clean_scrub_stamp":"2024-05-27T00:33:50.083000+0000","objects_scrubbed":73,"log_size":911,"log_dups_size":0,"ondisk_log_size":911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:07:04.649211+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":73,"num_object_clones":0,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":305,"num_read_kb":205128,"num_write":987,"num_write_kb":76341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":68,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":978965,"num_omap_keys":1772,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"48'655","reported_seq":909,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:47.759123+0000","last_change":"2024-05-27T00:33:47.759123+0000","last_active":"2024-05-27T00:33:47.759123+0000","last_peered":"2024-05-27T00:33:47.759123+0000","last_clean":"2024-05-27T00:33:47.759123+0000","last_became_active":"2024-05-27T00:15:24.621038+0000","last_became_peered":"2024-05-27T00:15:24.621038+0000","last_unstale":"2024-05-27T00:33:47.759123+0000","last_undegraded":"2024-05-27T00:33:47.759123+0000","last_fullsized":"2024-05-27T00:33:47.759123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'654","last_scrub_stamp":"2024-05-27T00:33:47.758941+0000","last_deep_scrub":"48'654","last_deep_scrub_stamp":"2024-05-27T00:33:47.758941+0000","last_clean_scrub_stamp":"2024-05-27T00:33:47.758941+0000","objects_scrubbed":63,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:42:50.594958+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":317,"num_read_kb":129314,"num_write":747,"num_write_kb":59176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":904751,"num_omap_keys":1704,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"48'2449","reported_seq":4317,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:50.130096+0000","last_change":"2024-05-27T00:33:50.130131+0000","last_active":"2024-05-27T00:33:50.130096+0000","last_peered":"2024-05-27T00:33:50.130096+0000","last_clean":"2024-05-27T00:33:50.130096+0000","last_became_active":"2024-05-27T00:15:25.596238+0000","last_became_peered":"2024-05-27T00:15:25.596238+0000","last_unstale":"2024-05-27T00:33:50.130096+0000","last_undegraded":"2024-05-27T00:33:50.130096+0000","last_fullsized":"2024-05-27T00:33:50.130096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2449","last_scrub_stamp":"2024-05-27T00:33:50.129964+0000","last_deep_scrub":"48'2449","last_deep_scrub_stamp":"2024-05-27T00:33:50.129964+0000","last_clean_scrub_stamp":"2024-05-27T00:33:50.129964+0000","objects_scrubbed":667,"log_size":2449,"log_dups_size":0,"ondisk_log_size":2449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:51.790932+0000","scrub_duration":1487,"objects_trimmed":0,"snaptrim_duration":0.0087892630000000003,"stat_sum":{"num_bytes":15183090,"num_objects":667,"num_object_clones":314,"num_object_copies":1334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":667,"num_whiteouts":213,"num_read":52,"num_read_kb":17293,"num_write":1639,"num_write_kb":25751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"48'1977","reported_seq":2235,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:49.815013+0000","last_change":"2024-05-27T00:33:49.815080+0000","last_active":"2024-05-27T00:33:49.815013+0000","last_peered":"2024-05-27T00:33:49.815013+0000","last_clean":"2024-05-27T00:33:49.815013+0000","last_became_active":"2024-05-27T00:15:24.618860+0000","last_became_peered":"2024-05-27T00:15:24.618860+0000","last_unstale":"2024-05-27T00:33:49.815013+0000","last_undegraded":"2024-05-27T00:33:49.815013+0000","last_fullsized":"2024-05-27T00:33:49.815013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1977","last_scrub_stamp":"2024-05-27T00:33:49.814913+0000","last_deep_scrub":"48'1977","last_deep_scrub_stamp":"2024-05-27T00:33:49.814913+0000","last_clean_scrub_stamp":"2024-05-27T00:33:49.814913+0000","objects_scrubbed":64,"log_size":1977,"log_dups_size":0,"ondisk_log_size":1977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:32:31.167183+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":310,"num_read_kb":210919,"num_write":2093,"num_write_kb":80740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":56,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":664632,"num_omap_keys":1134,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"48'2167","reported_seq":4077,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:53.451669+0000","last_change":"2024-05-27T00:33:53.451669+0000","last_active":"2024-05-27T00:33:53.451669+0000","last_peered":"2024-05-27T00:33:53.451669+0000","last_clean":"2024-05-27T00:33:53.451669+0000","last_became_active":"2024-05-27T00:15:25.592840+0000","last_became_peered":"2024-05-27T00:15:25.592840+0000","last_unstale":"2024-05-27T00:33:53.451669+0000","last_undegraded":"2024-05-27T00:33:53.451669+0000","last_fullsized":"2024-05-27T00:33:53.451669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2166","last_scrub_stamp":"2024-05-27T00:33:53.451533+0000","last_deep_scrub":"48'2166","last_deep_scrub_stamp":"2024-05-27T00:33:53.451533+0000","last_clean_scrub_stamp":"2024-05-27T00:33:53.451533+0000","objects_scrubbed":610,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:26:09.923102+0000","scrub_duration":1811,"objects_trimmed":0,"snaptrim_duration":0.014633893,"stat_sum":{"num_bytes":24048299,"num_objects":610,"num_object_clones":282,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":196,"num_read":64,"num_read_kb":52608,"num_write":1412,"num_write_kb":30269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"48'654","reported_seq":929,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:48.061245+0000","last_change":"2024-05-27T00:33:48.061245+0000","last_active":"2024-05-27T00:33:48.061245+0000","last_peered":"2024-05-27T00:33:48.061245+0000","last_clean":"2024-05-27T00:33:48.061245+0000","last_became_active":"2024-05-27T00:15:24.618492+0000","last_became_peered":"2024-05-27T00:15:24.618492+0000","last_unstale":"2024-05-27T00:33:48.061245+0000","last_undegraded":"2024-05-27T00:33:48.061245+0000","last_fullsized":"2024-05-27T00:33:48.061245+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'653","last_scrub_stamp":"2024-05-27T00:33:48.061109+0000","last_deep_scrub":"48'653","last_deep_scrub_stamp":"2024-05-27T00:33:48.061109+0000","last_clean_scrub_stamp":"2024-05-27T00:33:48.061109+0000","objects_scrubbed":65,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:26:51.786910+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12951497,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":291,"num_read_kb":124312,"num_write":686,"num_write_kb":63327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":59,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":655421,"num_omap_keys":1089,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"48'2080","reported_seq":4000,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:54.710209+0000","last_change":"2024-05-27T00:33:54.710268+0000","last_active":"2024-05-27T00:33:54.710209+0000","last_peered":"2024-05-27T00:33:54.710209+0000","last_clean":"2024-05-27T00:33:54.710209+0000","last_became_active":"2024-05-27T00:15:25.613652+0000","last_became_peered":"2024-05-27T00:15:25.613652+0000","last_unstale":"2024-05-27T00:33:54.710209+0000","last_undegraded":"2024-05-27T00:33:54.710209+0000","last_fullsized":"2024-05-27T00:33:54.710209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2080","last_scrub_stamp":"2024-05-27T00:33:54.710105+0000","last_deep_scrub":"48'2080","last_deep_scrub_stamp":"2024-05-27T00:33:54.710105+0000","last_clean_scrub_stamp":"2024-05-27T00:33:54.710105+0000","objects_scrubbed":572,"log_size":2080,"log_dups_size":0,"ondisk_log_size":2080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:54:53.091596+0000","scrub_duration":1127,"objects_trimmed":0,"snaptrim_duration":0.034220156000000002,"stat_sum":{"num_bytes":22913099,"num_objects":572,"num_object_clones":265,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":188,"num_read":59,"num_read_kb":32908,"num_write":1361,"num_write_kb":24035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"48'1543","reported_seq":1760,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:50.914198+0000","last_change":"2024-05-27T00:33:50.914259+0000","last_active":"2024-05-27T00:33:50.914198+0000","last_peered":"2024-05-27T00:33:50.914198+0000","last_clean":"2024-05-27T00:33:50.914198+0000","last_became_active":"2024-05-27T00:15:24.625929+0000","last_became_peered":"2024-05-27T00:15:24.625929+0000","last_unstale":"2024-05-27T00:33:50.914198+0000","last_undegraded":"2024-05-27T00:33:50.914198+0000","last_fullsized":"2024-05-27T00:33:50.914198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1543","last_scrub_stamp":"2024-05-27T00:33:50.914088+0000","last_deep_scrub":"48'1543","last_deep_scrub_stamp":"2024-05-27T00:33:50.914088+0000","last_clean_scrub_stamp":"2024-05-27T00:33:50.914088+0000","objects_scrubbed":56,"log_size":1543,"log_dups_size":0,"ondisk_log_size":1543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:16:37.782603+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":244,"num_read_kb":161522,"num_write":1600,"num_write_kb":88170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":550070,"num_omap_keys":855,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"48'2462","reported_seq":4271,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:53.982326+0000","last_change":"2024-05-27T00:33:53.982326+0000","last_active":"2024-05-27T00:33:53.982326+0000","last_peered":"2024-05-27T00:33:53.982326+0000","last_clean":"2024-05-27T00:33:53.982326+0000","last_became_active":"2024-05-27T00:15:25.612704+0000","last_became_peered":"2024-05-27T00:15:25.612704+0000","last_unstale":"2024-05-27T00:33:53.982326+0000","last_undegraded":"2024-05-27T00:33:53.982326+0000","last_fullsized":"2024-05-27T00:33:53.982326+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2461","last_scrub_stamp":"2024-05-27T00:33:53.982086+0000","last_deep_scrub":"48'2461","last_deep_scrub_stamp":"2024-05-27T00:33:53.982086+0000","last_clean_scrub_stamp":"2024-05-27T00:33:53.982086+0000","objects_scrubbed":713,"log_size":2462,"log_dups_size":0,"ondisk_log_size":2462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:54:25.101499+0000","scrub_duration":1903,"objects_trimmed":0,"snaptrim_duration":0.017358472,"stat_sum":{"num_bytes":15539305,"num_objects":713,"num_object_clones":382,"num_object_copies":1426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":204,"num_read":59,"num_read_kb":17601,"num_write":1514,"num_write_kb":17852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"48'1668","reported_seq":1977,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:53.358336+0000","last_change":"2024-05-27T00:33:48.872864+0000","last_active":"2024-05-27T00:33:53.358336+0000","last_peered":"2024-05-27T00:33:53.358336+0000","last_clean":"2024-05-27T00:33:53.358336+0000","last_became_active":"2024-05-27T00:15:24.617719+0000","last_became_peered":"2024-05-27T00:15:24.617719+0000","last_unstale":"2024-05-27T00:33:53.358336+0000","last_undegraded":"2024-05-27T00:33:53.358336+0000","last_fullsized":"2024-05-27T00:33:53.358336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1660","last_scrub_stamp":"2024-05-27T00:33:48.872688+0000","last_deep_scrub":"48'1660","last_deep_scrub_stamp":"2024-05-27T00:33:48.872688+0000","last_clean_scrub_stamp":"2024-05-27T00:33:48.872688+0000","objects_scrubbed":73,"log_size":1668,"log_dups_size":0,"ondisk_log_size":1668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:07:03.438898+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":74,"num_object_clones":0,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":428,"num_read_kb":130260,"num_write":1820,"num_write_kb":64852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1108977,"num_omap_keys":1855,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"48'2199","reported_seq":4096,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:51.257009+0000","last_change":"2024-05-27T00:33:51.257009+0000","last_active":"2024-05-27T00:33:51.257009+0000","last_peered":"2024-05-27T00:33:51.257009+0000","last_clean":"2024-05-27T00:33:51.257009+0000","last_became_active":"2024-05-27T00:15:25.621392+0000","last_became_peered":"2024-05-27T00:15:25.621392+0000","last_unstale":"2024-05-27T00:33:51.257009+0000","last_undegraded":"2024-05-27T00:33:51.257009+0000","last_fullsized":"2024-05-27T00:33:51.257009+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2199","last_scrub_stamp":"2024-05-27T00:33:51.256886+0000","last_deep_scrub":"48'2199","last_deep_scrub_stamp":"2024-05-27T00:33:51.256886+0000","last_clean_scrub_stamp":"2024-05-27T00:33:51.256886+0000","objects_scrubbed":611,"log_size":2199,"log_dups_size":0,"ondisk_log_size":2199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:01:20.103358+0000","scrub_duration":1255,"objects_trimmed":0,"snaptrim_duration":0.022583427,"stat_sum":{"num_bytes":13591445,"num_objects":611,"num_object_clones":285,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":198,"num_read":54,"num_read_kb":12130,"num_write":1438,"num_write_kb":19929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"48'1137","reported_seq":1431,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:49.113784+0000","last_change":"2024-05-27T00:33:49.113784+0000","last_active":"2024-05-27T00:33:49.113784+0000","last_peered":"2024-05-27T00:33:49.113784+0000","last_clean":"2024-05-27T00:33:49.113784+0000","last_became_active":"2024-05-27T00:15:24.620685+0000","last_became_peered":"2024-05-27T00:15:24.620685+0000","last_unstale":"2024-05-27T00:33:49.113784+0000","last_undegraded":"2024-05-27T00:33:49.113784+0000","last_fullsized":"2024-05-27T00:33:49.113784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1136","last_scrub_stamp":"2024-05-27T00:33:49.113615+0000","last_deep_scrub":"48'1136","last_deep_scrub_stamp":"2024-05-27T00:33:49.113615+0000","last_clean_scrub_stamp":"2024-05-27T00:33:49.113615+0000","objects_scrubbed":78,"log_size":1137,"log_dups_size":0,"ondisk_log_size":1137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:24:32.661052+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555104,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":365,"num_read_kb":194738,"num_write":1227,"num_write_kb":105128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1029602,"num_omap_keys":1894,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"48'2170","reported_seq":4058,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:58.556915+0000","last_change":"2024-05-27T00:33:58.556915+0000","last_active":"2024-05-27T00:33:58.556915+0000","last_peered":"2024-05-27T00:33:58.556915+0000","last_clean":"2024-05-27T00:33:58.556915+0000","last_became_active":"2024-05-27T00:15:25.623364+0000","last_became_peered":"2024-05-27T00:15:25.623364+0000","last_unstale":"2024-05-27T00:33:58.556915+0000","last_undegraded":"2024-05-27T00:33:58.556915+0000","last_fullsized":"2024-05-27T00:33:58.556915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2169","last_scrub_stamp":"2024-05-27T00:33:58.556708+0000","last_deep_scrub":"48'2169","last_deep_scrub_stamp":"2024-05-27T00:33:58.556708+0000","last_clean_scrub_stamp":"2024-05-27T00:33:58.556708+0000","objects_scrubbed":609,"log_size":2170,"log_dups_size":0,"ondisk_log_size":2170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:13:20.506843+0000","scrub_duration":915,"objects_trimmed":0,"snaptrim_duration":0.030558409000000002,"stat_sum":{"num_bytes":10583440,"num_objects":609,"num_object_clones":281,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":185,"num_read":42,"num_read_kb":12368,"num_write":1451,"num_write_kb":13410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"48'892","reported_seq":1158,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:49.193807+0000","last_change":"2024-05-27T00:33:44.009333+0000","last_active":"2024-05-27T00:33:49.193807+0000","last_peered":"2024-05-27T00:33:49.193807+0000","last_clean":"2024-05-27T00:33:49.193807+0000","last_became_active":"2024-05-27T00:15:24.605689+0000","last_became_peered":"2024-05-27T00:15:24.605689+0000","last_unstale":"2024-05-27T00:33:49.193807+0000","last_undegraded":"2024-05-27T00:33:49.193807+0000","last_fullsized":"2024-05-27T00:33:49.193807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'890","last_scrub_stamp":"2024-05-27T00:33:44.009223+0000","last_deep_scrub":"48'890","last_deep_scrub_stamp":"2024-05-27T00:33:44.009223+0000","last_clean_scrub_stamp":"2024-05-27T00:33:44.009223+0000","objects_scrubbed":65,"log_size":892,"log_dups_size":0,"ondisk_log_size":892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:48:38.583455+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":336,"num_read_kb":131285,"num_write":987,"num_write_kb":67415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":59,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706283,"num_omap_keys":1375,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"48'2368","reported_seq":4212,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.152105+0000","last_change":"2024-05-27T00:33:56.498844+0000","last_active":"2024-05-27T00:33:59.152105+0000","last_peered":"2024-05-27T00:33:59.152105+0000","last_clean":"2024-05-27T00:33:59.152105+0000","last_became_active":"2024-05-27T00:15:25.623756+0000","last_became_peered":"2024-05-27T00:15:25.623756+0000","last_unstale":"2024-05-27T00:33:59.152105+0000","last_undegraded":"2024-05-27T00:33:59.152105+0000","last_fullsized":"2024-05-27T00:33:59.152105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2366","last_scrub_stamp":"2024-05-27T00:33:56.498689+0000","last_deep_scrub":"48'2366","last_deep_scrub_stamp":"2024-05-27T00:33:56.498689+0000","last_clean_scrub_stamp":"2024-05-27T00:33:56.498689+0000","objects_scrubbed":658,"log_size":2368,"log_dups_size":0,"ondisk_log_size":2368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:12:11.356723+0000","scrub_duration":1347,"objects_trimmed":0,"snaptrim_duration":0.018271039999999999,"stat_sum":{"num_bytes":20486162,"num_objects":658,"num_object_clones":358,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":186,"num_read":72,"num_read_kb":36313,"num_write":1481,"num_write_kb":31834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"48'577","reported_seq":786,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:48.850537+0000","last_change":"2024-05-27T00:33:48.850537+0000","last_active":"2024-05-27T00:33:48.850537+0000","last_peered":"2024-05-27T00:33:48.850537+0000","last_clean":"2024-05-27T00:33:48.850537+0000","last_became_active":"2024-05-27T00:15:24.615141+0000","last_became_peered":"2024-05-27T00:15:24.615141+0000","last_unstale":"2024-05-27T00:33:48.850537+0000","last_undegraded":"2024-05-27T00:33:48.850537+0000","last_fullsized":"2024-05-27T00:33:48.850537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'577","last_scrub_stamp":"2024-05-27T00:33:48.850333+0000","last_deep_scrub":"48'577","last_deep_scrub_stamp":"2024-05-27T00:33:48.850333+0000","last_clean_scrub_stamp":"2024-05-27T00:33:48.850333+0000","objects_scrubbed":56,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:17:24.167085+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":232,"num_read_kb":71610,"num_write":648,"num_write_kb":51355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":670005,"num_omap_keys":1070,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"48'2330","reported_seq":4234,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:34:05.761206+0000","last_change":"2024-05-27T00:34:05.761206+0000","last_active":"2024-05-27T00:34:05.761206+0000","last_peered":"2024-05-27T00:34:05.761206+0000","last_clean":"2024-05-27T00:34:05.761206+0000","last_became_active":"2024-05-27T00:15:25.640607+0000","last_became_peered":"2024-05-27T00:15:25.640607+0000","last_unstale":"2024-05-27T00:34:05.761206+0000","last_undegraded":"2024-05-27T00:34:05.761206+0000","last_fullsized":"2024-05-27T00:34:05.761206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2330","last_scrub_stamp":"2024-05-27T00:34:05.761030+0000","last_deep_scrub":"48'2330","last_deep_scrub_stamp":"2024-05-27T00:34:05.761030+0000","last_clean_scrub_stamp":"2024-05-27T00:34:05.761030+0000","objects_scrubbed":639,"log_size":2330,"log_dups_size":0,"ondisk_log_size":2330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:54:52.740726+0000","scrub_duration":1039,"objects_trimmed":0,"snaptrim_duration":0.028694234999999998,"stat_sum":{"num_bytes":27865757,"num_objects":639,"num_object_clones":296,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":192,"num_read":71,"num_read_kb":46547,"num_write":1544,"num_write_kb":29871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"48'422","reported_seq":632,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:58.359556+0000","last_change":"2024-05-27T00:33:50.987736+0000","last_active":"2024-05-27T00:33:58.359556+0000","last_peered":"2024-05-27T00:33:58.359556+0000","last_clean":"2024-05-27T00:33:58.359556+0000","last_became_active":"2024-05-27T00:15:24.625113+0000","last_became_peered":"2024-05-27T00:15:24.625113+0000","last_unstale":"2024-05-27T00:33:58.359556+0000","last_undegraded":"2024-05-27T00:33:58.359556+0000","last_fullsized":"2024-05-27T00:33:58.359556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'421","last_scrub_stamp":"2024-05-27T00:33:50.987562+0000","last_deep_scrub":"48'421","last_deep_scrub_stamp":"2024-05-27T00:33:50.987562+0000","last_clean_scrub_stamp":"2024-05-27T00:33:50.987562+0000","objects_scrubbed":57,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:22:34.518025+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":244,"num_read_kb":99922,"num_write":495,"num_write_kb":55215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":456147,"num_omap_keys":846,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"48'2159","reported_seq":4057,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.148213+0000","last_change":"2024-05-27T00:33:55.404396+0000","last_active":"2024-05-27T00:33:59.148213+0000","last_peered":"2024-05-27T00:33:59.148213+0000","last_clean":"2024-05-27T00:33:59.148213+0000","last_became_active":"2024-05-27T00:15:25.607677+0000","last_became_peered":"2024-05-27T00:15:25.607677+0000","last_unstale":"2024-05-27T00:33:59.148213+0000","last_undegraded":"2024-05-27T00:33:59.148213+0000","last_fullsized":"2024-05-27T00:33:59.148213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2158","last_scrub_stamp":"2024-05-27T00:33:55.404303+0000","last_deep_scrub":"48'2158","last_deep_scrub_stamp":"2024-05-27T00:33:55.404303+0000","last_clean_scrub_stamp":"2024-05-27T00:33:55.404303+0000","objects_scrubbed":594,"log_size":2159,"log_dups_size":0,"ondisk_log_size":2159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:10:58.568312+0000","scrub_duration":1511,"objects_trimmed":0,"snaptrim_duration":0.025579069999999999,"stat_sum":{"num_bytes":19681280,"num_objects":594,"num_object_clones":272,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":185,"num_read":43,"num_read_kb":14554,"num_write":1434,"num_write_kb":24217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"48'817","reported_seq":1116,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:47.741773+0000","last_change":"2024-05-27T00:33:47.741773+0000","last_active":"2024-05-27T00:33:47.741773+0000","last_peered":"2024-05-27T00:33:47.741773+0000","last_clean":"2024-05-27T00:33:47.741773+0000","last_became_active":"2024-05-27T00:15:24.599815+0000","last_became_peered":"2024-05-27T00:15:24.599815+0000","last_unstale":"2024-05-27T00:33:47.741773+0000","last_undegraded":"2024-05-27T00:33:47.741773+0000","last_fullsized":"2024-05-27T00:33:47.741773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'817","last_scrub_stamp":"2024-05-27T00:33:47.741666+0000","last_deep_scrub":"48'817","last_deep_scrub_stamp":"2024-05-27T00:33:47.741666+0000","last_clean_scrub_stamp":"2024-05-27T00:33:47.741666+0000","objects_scrubbed":80,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:05:45.589814+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":80,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":479,"num_read_kb":120710,"num_write":993,"num_write_kb":72772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":73,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1266227,"num_omap_keys":1963,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"48'2226","reported_seq":4135,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:34:00.469262+0000","last_change":"2024-05-27T00:34:00.469262+0000","last_active":"2024-05-27T00:34:00.469262+0000","last_peered":"2024-05-27T00:34:00.469262+0000","last_clean":"2024-05-27T00:34:00.469262+0000","last_became_active":"2024-05-27T00:15:25.632454+0000","last_became_peered":"2024-05-27T00:15:25.632454+0000","last_unstale":"2024-05-27T00:34:00.469262+0000","last_undegraded":"2024-05-27T00:34:00.469262+0000","last_fullsized":"2024-05-27T00:34:00.469262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2226","last_scrub_stamp":"2024-05-27T00:34:00.469147+0000","last_deep_scrub":"48'2226","last_deep_scrub_stamp":"2024-05-27T00:34:00.469147+0000","last_clean_scrub_stamp":"2024-05-27T00:34:00.469147+0000","objects_scrubbed":604,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:59.155017+0000","scrub_duration":799,"objects_trimmed":0,"snaptrim_duration":0.028937418999999999,"stat_sum":{"num_bytes":19558883,"num_objects":604,"num_object_clones":279,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":198,"num_read":60,"num_read_kb":39320,"num_write":1455,"num_write_kb":30058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"48'995","reported_seq":1256,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:46.064385+0000","last_change":"2024-05-27T00:33:46.064473+0000","last_active":"2024-05-27T00:33:46.064385+0000","last_peered":"2024-05-27T00:33:46.064385+0000","last_clean":"2024-05-27T00:33:46.064385+0000","last_became_active":"2024-05-27T00:15:24.599507+0000","last_became_peered":"2024-05-27T00:15:24.599507+0000","last_unstale":"2024-05-27T00:33:46.064385+0000","last_undegraded":"2024-05-27T00:33:46.064385+0000","last_fullsized":"2024-05-27T00:33:46.064385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'995","last_scrub_stamp":"2024-05-27T00:33:46.064195+0000","last_deep_scrub":"48'995","last_deep_scrub_stamp":"2024-05-27T00:33:46.064195+0000","last_clean_scrub_stamp":"2024-05-27T00:33:46.064195+0000","objects_scrubbed":68,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:50:49.020441+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777238,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":329,"num_read_kb":104768,"num_write":1082,"num_write_kb":59608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":831747,"num_omap_keys":1453,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"48'2240","reported_seq":4134,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:34:01.671486+0000","last_change":"2024-05-27T00:34:01.671596+0000","last_active":"2024-05-27T00:34:01.671486+0000","last_peered":"2024-05-27T00:34:01.671486+0000","last_clean":"2024-05-27T00:34:01.671486+0000","last_became_active":"2024-05-27T00:15:25.632540+0000","last_became_peered":"2024-05-27T00:15:25.632540+0000","last_unstale":"2024-05-27T00:34:01.671486+0000","last_undegraded":"2024-05-27T00:34:01.671486+0000","last_fullsized":"2024-05-27T00:34:01.671486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2240","last_scrub_stamp":"2024-05-27T00:34:01.671398+0000","last_deep_scrub":"48'2240","last_deep_scrub_stamp":"2024-05-27T00:34:01.671398+0000","last_clean_scrub_stamp":"2024-05-27T00:34:01.671398+0000","objects_scrubbed":631,"log_size":2240,"log_dups_size":0,"ondisk_log_size":2240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:55:17.777466+0000","scrub_duration":963,"objects_trimmed":0,"snaptrim_duration":0.029056843999999998,"stat_sum":{"num_bytes":27647082,"num_objects":631,"num_object_clones":290,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":189,"num_read":55,"num_read_kb":39571,"num_write":1465,"num_write_kb":33528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"48'902","reported_seq":1126,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:46.558932+0000","last_change":"2024-05-27T00:33:46.558932+0000","last_active":"2024-05-27T00:33:46.558932+0000","last_peered":"2024-05-27T00:33:46.558932+0000","last_clean":"2024-05-27T00:33:46.558932+0000","last_became_active":"2024-05-27T00:15:24.599913+0000","last_became_peered":"2024-05-27T00:15:24.599913+0000","last_unstale":"2024-05-27T00:33:46.558932+0000","last_undegraded":"2024-05-27T00:33:46.558932+0000","last_fullsized":"2024-05-27T00:33:46.558932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'901","last_scrub_stamp":"2024-05-27T00:33:46.558817+0000","last_deep_scrub":"48'901","last_deep_scrub_stamp":"2024-05-27T00:33:46.558817+0000","last_clean_scrub_stamp":"2024-05-27T00:33:46.558817+0000","objects_scrubbed":54,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:58:45.617077+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":241,"num_read_kb":194399,"num_write":979,"num_write_kb":84425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":982076,"num_omap_keys":1745,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"48'2126","reported_seq":4039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:50.894557+0000","last_change":"2024-05-27T00:33:50.894594+0000","last_active":"2024-05-27T00:33:50.894557+0000","last_peered":"2024-05-27T00:33:50.894557+0000","last_clean":"2024-05-27T00:33:50.894557+0000","last_became_active":"2024-05-27T00:15:25.643401+0000","last_became_peered":"2024-05-27T00:15:25.643401+0000","last_unstale":"2024-05-27T00:33:50.894557+0000","last_undegraded":"2024-05-27T00:33:50.894557+0000","last_fullsized":"2024-05-27T00:33:50.894557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2126","last_scrub_stamp":"2024-05-27T00:33:50.894455+0000","last_deep_scrub":"48'2126","last_deep_scrub_stamp":"2024-05-27T00:33:50.894455+0000","last_clean_scrub_stamp":"2024-05-27T00:33:50.894455+0000","objects_scrubbed":590,"log_size":2126,"log_dups_size":0,"ondisk_log_size":2126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:25:50.488110+0000","scrub_duration":1251,"objects_trimmed":0,"snaptrim_duration":0.028414170999999998,"stat_sum":{"num_bytes":22687403,"num_objects":590,"num_object_clones":272,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":185,"num_read":57,"num_read_kb":45093,"num_write":1418,"num_write_kb":28216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"48'1220","reported_seq":1469,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:45.787134+0000","last_change":"2024-05-27T00:33:45.787134+0000","last_active":"2024-05-27T00:33:45.787134+0000","last_peered":"2024-05-27T00:33:45.787134+0000","last_clean":"2024-05-27T00:33:45.787134+0000","last_became_active":"2024-05-27T00:15:24.603196+0000","last_became_peered":"2024-05-27T00:15:24.603196+0000","last_unstale":"2024-05-27T00:33:45.787134+0000","last_undegraded":"2024-05-27T00:33:45.787134+0000","last_fullsized":"2024-05-27T00:33:45.787134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1219","last_scrub_stamp":"2024-05-27T00:33:45.787032+0000","last_deep_scrub":"48'1219","last_deep_scrub_stamp":"2024-05-27T00:33:45.787032+0000","last_clean_scrub_stamp":"2024-05-27T00:33:45.787032+0000","objects_scrubbed":80,"log_size":1220,"log_dups_size":0,"ondisk_log_size":1220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:25:46.003419+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33559003,"num_objects":79,"num_object_clones":0,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":293,"num_read_kb":166112,"num_write":1308,"num_write_kb":92227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":68,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":694602,"num_omap_keys":1256,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"48'2122","reported_seq":4031,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:56.180262+0000","last_change":"2024-05-27T00:33:56.180339+0000","last_active":"2024-05-27T00:33:56.180262+0000","last_peered":"2024-05-27T00:33:56.180262+0000","last_clean":"2024-05-27T00:33:56.180262+0000","last_became_active":"2024-05-27T00:15:25.613691+0000","last_became_peered":"2024-05-27T00:15:25.613691+0000","last_unstale":"2024-05-27T00:33:56.180262+0000","last_undegraded":"2024-05-27T00:33:56.180262+0000","last_fullsized":"2024-05-27T00:33:56.180262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2122","last_scrub_stamp":"2024-05-27T00:33:56.180156+0000","last_deep_scrub":"48'2122","last_deep_scrub_stamp":"2024-05-27T00:33:56.180156+0000","last_clean_scrub_stamp":"2024-05-27T00:33:56.180156+0000","objects_scrubbed":568,"log_size":2122,"log_dups_size":0,"ondisk_log_size":2122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:19:54.866025+0000","scrub_duration":1579,"objects_trimmed":0,"snaptrim_duration":0.034804740000000001,"stat_sum":{"num_bytes":14076594,"num_objects":568,"num_object_clones":258,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":185,"num_read":41,"num_read_kb":16491,"num_write":1436,"num_write_kb":17170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"48'818","reported_seq":1057,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:48.677595+0000","last_change":"2024-05-27T00:33:48.677669+0000","last_active":"2024-05-27T00:33:48.677595+0000","last_peered":"2024-05-27T00:33:48.677595+0000","last_clean":"2024-05-27T00:33:48.677595+0000","last_became_active":"2024-05-27T00:15:24.621184+0000","last_became_peered":"2024-05-27T00:15:24.621184+0000","last_unstale":"2024-05-27T00:33:48.677595+0000","last_undegraded":"2024-05-27T00:33:48.677595+0000","last_fullsized":"2024-05-27T00:33:48.677595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'818","last_scrub_stamp":"2024-05-27T00:33:48.677506+0000","last_deep_scrub":"48'818","last_deep_scrub_stamp":"2024-05-27T00:33:48.677506+0000","last_clean_scrub_stamp":"2024-05-27T00:33:48.677506+0000","objects_scrubbed":65,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:30:51.642281+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":292,"num_read_kb":141638,"num_write":933,"num_write_kb":64377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":986554,"num_omap_keys":1675,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"48'2192","reported_seq":4081,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:34:02.573339+0000","last_change":"2024-05-27T00:34:02.573376+0000","last_active":"2024-05-27T00:34:02.573339+0000","last_peered":"2024-05-27T00:34:02.573339+0000","last_clean":"2024-05-27T00:34:02.573339+0000","last_became_active":"2024-05-27T00:15:25.638693+0000","last_became_peered":"2024-05-27T00:15:25.638693+0000","last_unstale":"2024-05-27T00:34:02.573339+0000","last_undegraded":"2024-05-27T00:34:02.573339+0000","last_fullsized":"2024-05-27T00:34:02.573339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2192","last_scrub_stamp":"2024-05-27T00:34:02.573264+0000","last_deep_scrub":"48'2192","last_deep_scrub_stamp":"2024-05-27T00:34:02.573264+0000","last_clean_scrub_stamp":"2024-05-27T00:34:02.573264+0000","objects_scrubbed":620,"log_size":2192,"log_dups_size":0,"ondisk_log_size":2192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:33:31.167625+0000","scrub_duration":883,"objects_trimmed":0,"snaptrim_duration":0.028015244000000002,"stat_sum":{"num_bytes":15862998,"num_objects":620,"num_object_clones":289,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":200,"num_read":48,"num_read_kb":16478,"num_write":1412,"num_write_kb":22882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"48'119","reported_seq":242,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:50.691590+0000","last_change":"2024-05-27T00:33:50.691648+0000","last_active":"2024-05-27T00:33:50.691590+0000","last_peered":"2024-05-27T00:33:50.691590+0000","last_clean":"2024-05-27T00:33:50.691590+0000","last_became_active":"2024-05-27T00:15:14.502750+0000","last_became_peered":"2024-05-27T00:15:14.502750+0000","last_unstale":"2024-05-27T00:33:50.691590+0000","last_undegraded":"2024-05-27T00:33:50.691590+0000","last_fullsized":"2024-05-27T00:33:50.691590+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'119","last_scrub_stamp":"2024-05-27T00:33:50.691493+0000","last_deep_scrub":"48'119","last_deep_scrub_stamp":"2024-05-27T00:33:50.691493+0000","last_clean_scrub_stamp":"2024-05-27T00:33:50.691493+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:00:59.771033+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"48'923","reported_seq":1131,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:46.859929+0000","last_change":"2024-05-27T00:33:46.860040+0000","last_active":"2024-05-27T00:33:46.859929+0000","last_peered":"2024-05-27T00:33:46.859929+0000","last_clean":"2024-05-27T00:33:46.859929+0000","last_became_active":"2024-05-27T00:15:24.603030+0000","last_became_peered":"2024-05-27T00:15:24.603030+0000","last_unstale":"2024-05-27T00:33:46.859929+0000","last_undegraded":"2024-05-27T00:33:46.859929+0000","last_fullsized":"2024-05-27T00:33:46.859929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'923","last_scrub_stamp":"2024-05-27T00:33:46.859763+0000","last_deep_scrub":"48'923","last_deep_scrub_stamp":"2024-05-27T00:33:46.859763+0000","last_clean_scrub_stamp":"2024-05-27T00:33:46.859763+0000","objects_scrubbed":47,"log_size":923,"log_dups_size":0,"ondisk_log_size":923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:54:17.979175+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":219,"num_read_kb":131843,"num_write":971,"num_write_kb":67539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":694215,"num_omap_keys":1022,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"48'2253","reported_seq":4148,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.149662+0000","last_change":"2024-05-27T00:33:58.559618+0000","last_active":"2024-05-27T00:33:59.149662+0000","last_peered":"2024-05-27T00:33:59.149662+0000","last_clean":"2024-05-27T00:33:59.149662+0000","last_became_active":"2024-05-27T00:16:06.517816+0000","last_became_peered":"2024-05-27T00:16:06.517816+0000","last_unstale":"2024-05-27T00:33:59.149662+0000","last_undegraded":"2024-05-27T00:33:59.149662+0000","last_fullsized":"2024-05-27T00:33:59.149662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2251","last_scrub_stamp":"2024-05-27T00:33:58.559523+0000","last_deep_scrub":"48'2251","last_deep_scrub_stamp":"2024-05-27T00:33:58.559523+0000","last_clean_scrub_stamp":"2024-05-27T00:33:58.559523+0000","objects_scrubbed":639,"log_size":2253,"log_dups_size":0,"ondisk_log_size":2253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:45:17.838325+0000","scrub_duration":907,"objects_trimmed":0,"snaptrim_duration":0.0085828740000000008,"stat_sum":{"num_bytes":27761524,"num_objects":639,"num_object_clones":294,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":199,"num_read":54,"num_read_kb":34546,"num_write":1484,"num_write_kb":32905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"48'494","reported_seq":742,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:52.715417+0000","last_change":"2024-05-27T00:33:52.715491+0000","last_active":"2024-05-27T00:33:52.715417+0000","last_peered":"2024-05-27T00:33:52.715417+0000","last_clean":"2024-05-27T00:33:52.715417+0000","last_became_active":"2024-05-27T00:15:24.626276+0000","last_became_peered":"2024-05-27T00:15:24.626276+0000","last_unstale":"2024-05-27T00:33:52.715417+0000","last_undegraded":"2024-05-27T00:33:52.715417+0000","last_fullsized":"2024-05-27T00:33:52.715417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'494","last_scrub_stamp":"2024-05-27T00:33:52.715313+0000","last_deep_scrub":"48'494","last_deep_scrub_stamp":"2024-05-27T00:33:52.715313+0000","last_clean_scrub_stamp":"2024-05-27T00:33:52.715313+0000","objects_scrubbed":64,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:46:40.306275+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":299,"num_read_kb":174228,"num_write":559,"num_write_kb":63257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":693343,"num_omap_keys":1361,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"48'2085","reported_seq":3985,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:51.238726+0000","last_change":"2024-05-27T00:33:51.238726+0000","last_active":"2024-05-27T00:33:51.238726+0000","last_peered":"2024-05-27T00:33:51.238726+0000","last_clean":"2024-05-27T00:33:51.238726+0000","last_became_active":"2024-05-27T00:15:25.610322+0000","last_became_peered":"2024-05-27T00:15:25.610322+0000","last_unstale":"2024-05-27T00:33:51.238726+0000","last_undegraded":"2024-05-27T00:33:51.238726+0000","last_fullsized":"2024-05-27T00:33:51.238726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2084","last_scrub_stamp":"2024-05-27T00:33:51.238558+0000","last_deep_scrub":"48'2084","last_deep_scrub_stamp":"2024-05-27T00:33:51.238558+0000","last_clean_scrub_stamp":"2024-05-27T00:33:51.238558+0000","objects_scrubbed":569,"log_size":2085,"log_dups_size":0,"ondisk_log_size":2085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:28:00.544353+0000","scrub_duration":1243,"objects_trimmed":0,"snaptrim_duration":0.016778365999999999,"stat_sum":{"num_bytes":6084007,"num_objects":569,"num_object_clones":261,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":179,"num_read":34,"num_read_kb":53,"num_write":1407,"num_write_kb":7726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'1728","reported_seq":2626,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:45.565851+0000","last_change":"2024-05-27T00:33:45.565881+0000","last_active":"2024-05-27T00:33:45.565851+0000","last_peered":"2024-05-27T00:33:45.565851+0000","last_clean":"2024-05-27T00:33:45.565851+0000","last_became_active":"2024-05-27T00:15:24.597892+0000","last_became_peered":"2024-05-27T00:15:24.597892+0000","last_unstale":"2024-05-27T00:33:45.565851+0000","last_undegraded":"2024-05-27T00:33:45.565851+0000","last_fullsized":"2024-05-27T00:33:45.565851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1728","last_scrub_stamp":"2024-05-27T00:33:45.565774+0000","last_deep_scrub":"48'1728","last_deep_scrub_stamp":"2024-05-27T00:33:45.565774+0000","last_clean_scrub_stamp":"2024-05-27T00:33:45.565774+0000","objects_scrubbed":79,"log_size":1728,"log_dups_size":0,"ondisk_log_size":1728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T01:01:36.629433+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":79,"num_object_clones":0,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":1008,"num_read_kb":169683,"num_write":1833,"num_write_kb":85277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":73,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1183597,"num_omap_keys":2018,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"48'2238","reported_seq":4131,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:52.264473+0000","last_change":"2024-05-27T00:33:52.264561+0000","last_active":"2024-05-27T00:33:52.264473+0000","last_peered":"2024-05-27T00:33:52.264473+0000","last_clean":"2024-05-27T00:33:52.264473+0000","last_became_active":"2024-05-27T00:15:25.591174+0000","last_became_peered":"2024-05-27T00:15:25.591174+0000","last_unstale":"2024-05-27T00:33:52.264473+0000","last_undegraded":"2024-05-27T00:33:52.264473+0000","last_fullsized":"2024-05-27T00:33:52.264473+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2238","last_scrub_stamp":"2024-05-27T00:33:52.264350+0000","last_deep_scrub":"48'2238","last_deep_scrub_stamp":"2024-05-27T00:33:52.264350+0000","last_clean_scrub_stamp":"2024-05-27T00:33:52.264350+0000","objects_scrubbed":638,"log_size":2238,"log_dups_size":0,"ondisk_log_size":2238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:43:48.835432+0000","scrub_duration":1647,"objects_trimmed":0,"snaptrim_duration":0.016042048999999999,"stat_sum":{"num_bytes":11674242,"num_objects":638,"num_object_clones":289,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":200,"num_read":51,"num_read_kb":12401,"num_write":1481,"num_write_kb":14079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"48'775","reported_seq":1045,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:47.804905+0000","last_change":"2024-05-27T00:33:47.804905+0000","last_active":"2024-05-27T00:33:47.804905+0000","last_peered":"2024-05-27T00:33:47.804905+0000","last_clean":"2024-05-27T00:33:47.804905+0000","last_became_active":"2024-05-27T00:15:24.614321+0000","last_became_peered":"2024-05-27T00:15:24.614321+0000","last_unstale":"2024-05-27T00:33:47.804905+0000","last_undegraded":"2024-05-27T00:33:47.804905+0000","last_fullsized":"2024-05-27T00:33:47.804905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'774","last_scrub_stamp":"2024-05-27T00:33:47.804789+0000","last_deep_scrub":"48'774","last_deep_scrub_stamp":"2024-05-27T00:33:47.804789+0000","last_clean_scrub_stamp":"2024-05-27T00:33:47.804789+0000","objects_scrubbed":66,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:34:59.498814+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":344,"num_read_kb":168323,"num_write":933,"num_write_kb":68377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":59,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":954327,"num_omap_keys":1567,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"48'2237","reported_seq":4139,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:49.355448+0000","last_change":"2024-05-27T00:33:49.355504+0000","last_active":"2024-05-27T00:33:49.355448+0000","last_peered":"2024-05-27T00:33:49.355448+0000","last_clean":"2024-05-27T00:33:49.355448+0000","last_became_active":"2024-05-27T00:15:25.613312+0000","last_became_peered":"2024-05-27T00:15:25.613312+0000","last_unstale":"2024-05-27T00:33:49.355448+0000","last_undegraded":"2024-05-27T00:33:49.355448+0000","last_fullsized":"2024-05-27T00:33:49.355448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2237","last_scrub_stamp":"2024-05-27T00:33:49.355346+0000","last_deep_scrub":"48'2237","last_deep_scrub_stamp":"2024-05-27T00:33:49.355346+0000","last_clean_scrub_stamp":"2024-05-27T00:33:49.355346+0000","objects_scrubbed":605,"log_size":2237,"log_dups_size":0,"ondisk_log_size":2237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:30:22.450453+0000","scrub_duration":1335,"objects_trimmed":0,"snaptrim_duration":0.021175818999999999,"stat_sum":{"num_bytes":27400217,"num_objects":605,"num_object_clones":275,"num_object_copies":1210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":605,"num_whiteouts":181,"num_read":49,"num_read_kb":37771,"num_write":1530,"num_write_kb":29686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"48'878","reported_seq":1111,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:46.728010+0000","last_change":"2024-05-27T00:33:46.728074+0000","last_active":"2024-05-27T00:33:46.728010+0000","last_peered":"2024-05-27T00:33:46.728010+0000","last_clean":"2024-05-27T00:33:46.728010+0000","last_became_active":"2024-05-27T00:15:24.603168+0000","last_became_peered":"2024-05-27T00:15:24.603168+0000","last_unstale":"2024-05-27T00:33:46.728010+0000","last_undegraded":"2024-05-27T00:33:46.728010+0000","last_fullsized":"2024-05-27T00:33:46.728010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'878","last_scrub_stamp":"2024-05-27T00:33:46.727894+0000","last_deep_scrub":"48'878","last_deep_scrub_stamp":"2024-05-27T00:33:46.727894+0000","last_clean_scrub_stamp":"2024-05-27T00:33:46.727894+0000","objects_scrubbed":60,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:23:51.296241+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":262,"num_read_kb":165581,"num_write":945,"num_write_kb":83700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":655959,"num_omap_keys":1179,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"48'2199","reported_seq":4096,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:58.586390+0000","last_change":"2024-05-27T00:33:58.586390+0000","last_active":"2024-05-27T00:33:58.586390+0000","last_peered":"2024-05-27T00:33:58.586390+0000","last_clean":"2024-05-27T00:33:58.586390+0000","last_became_active":"2024-05-27T00:15:25.612401+0000","last_became_peered":"2024-05-27T00:15:25.612401+0000","last_unstale":"2024-05-27T00:33:58.586390+0000","last_undegraded":"2024-05-27T00:33:58.586390+0000","last_fullsized":"2024-05-27T00:33:58.586390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2198","last_scrub_stamp":"2024-05-27T00:33:58.586203+0000","last_deep_scrub":"48'2198","last_deep_scrub_stamp":"2024-05-27T00:33:58.586203+0000","last_clean_scrub_stamp":"2024-05-27T00:33:58.586203+0000","objects_scrubbed":606,"log_size":2199,"log_dups_size":0,"ondisk_log_size":2199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:57:17.766945+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0.034092800999999999,"stat_sum":{"num_bytes":22501150,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":211,"num_read":50,"num_read_kb":38442,"num_write":1426,"num_write_kb":28481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"48'1472","reported_seq":1723,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:56.711536+0000","last_change":"2024-05-27T00:33:56.711657+0000","last_active":"2024-05-27T00:33:56.711536+0000","last_peered":"2024-05-27T00:33:56.711536+0000","last_clean":"2024-05-27T00:33:56.711536+0000","last_became_active":"2024-05-27T00:15:24.634448+0000","last_became_peered":"2024-05-27T00:15:24.634448+0000","last_unstale":"2024-05-27T00:33:56.711536+0000","last_undegraded":"2024-05-27T00:33:56.711536+0000","last_fullsized":"2024-05-27T00:33:56.711536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1472","last_scrub_stamp":"2024-05-27T00:33:56.711201+0000","last_deep_scrub":"48'1472","last_deep_scrub_stamp":"2024-05-27T00:33:56.711201+0000","last_clean_scrub_stamp":"2024-05-27T00:33:56.711201+0000","objects_scrubbed":58,"log_size":1472,"log_dups_size":0,"ondisk_log_size":1472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:05:24.247140+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":299,"num_read_kb":161725,"num_write":1537,"num_write_kb":67545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":709255,"num_omap_keys":1403,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"48'2156","reported_seq":4068,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.150690+0000","last_change":"2024-05-27T00:33:54.344746+0000","last_active":"2024-05-27T00:33:59.150690+0000","last_peered":"2024-05-27T00:33:59.150690+0000","last_clean":"2024-05-27T00:33:59.150690+0000","last_became_active":"2024-05-27T00:15:25.592926+0000","last_became_peered":"2024-05-27T00:15:25.592926+0000","last_unstale":"2024-05-27T00:33:59.150690+0000","last_undegraded":"2024-05-27T00:33:59.150690+0000","last_fullsized":"2024-05-27T00:33:59.150690+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2154","last_scrub_stamp":"2024-05-27T00:33:54.344626+0000","last_deep_scrub":"48'2154","last_deep_scrub_stamp":"2024-05-27T00:33:54.344626+0000","last_clean_scrub_stamp":"2024-05-27T00:33:54.344626+0000","objects_scrubbed":586,"log_size":2156,"log_dups_size":0,"ondisk_log_size":2156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:19:48.105935+0000","scrub_duration":1679,"objects_trimmed":0,"snaptrim_duration":0.014512239,"stat_sum":{"num_bytes":23141178,"num_objects":586,"num_object_clones":267,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":185,"num_read":51,"num_read_kb":31023,"num_write":1460,"num_write_kb":28383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'893","reported_seq":1141,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:49.849060+0000","last_change":"2024-05-27T00:33:49.849060+0000","last_active":"2024-05-27T00:33:49.849060+0000","last_peered":"2024-05-27T00:33:49.849060+0000","last_clean":"2024-05-27T00:33:49.849060+0000","last_became_active":"2024-05-27T00:15:24.627080+0000","last_became_peered":"2024-05-27T00:15:24.627080+0000","last_unstale":"2024-05-27T00:33:49.849060+0000","last_undegraded":"2024-05-27T00:33:49.849060+0000","last_fullsized":"2024-05-27T00:33:49.849060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'892","last_scrub_stamp":"2024-05-27T00:33:49.848513+0000","last_deep_scrub":"48'892","last_deep_scrub_stamp":"2024-05-27T00:33:49.848513+0000","last_clean_scrub_stamp":"2024-05-27T00:33:49.848513+0000","objects_scrubbed":62,"log_size":893,"log_dups_size":0,"ondisk_log_size":893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:52:11.891059+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":312,"num_read_kb":117165,"num_write":1611,"num_write_kb":59353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":690121,"num_omap_keys":1336,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"48'2101","reported_seq":4009,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.151985+0000","last_change":"2024-05-27T00:33:57.413508+0000","last_active":"2024-05-27T00:33:59.151985+0000","last_peered":"2024-05-27T00:33:59.151985+0000","last_clean":"2024-05-27T00:33:59.151985+0000","last_became_active":"2024-05-27T00:15:25.631615+0000","last_became_peered":"2024-05-27T00:15:25.631615+0000","last_unstale":"2024-05-27T00:33:59.151985+0000","last_undegraded":"2024-05-27T00:33:59.151985+0000","last_fullsized":"2024-05-27T00:33:59.151985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2099","last_scrub_stamp":"2024-05-27T00:33:57.413425+0000","last_deep_scrub":"48'2099","last_deep_scrub_stamp":"2024-05-27T00:33:57.413425+0000","last_clean_scrub_stamp":"2024-05-27T00:33:57.413425+0000","objects_scrubbed":587,"log_size":2101,"log_dups_size":0,"ondisk_log_size":2101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:56:25.829680+0000","scrub_duration":755,"objects_trimmed":0,"snaptrim_duration":0.0089509419999999999,"stat_sum":{"num_bytes":13324494,"num_objects":587,"num_object_clones":270,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":181,"num_read":49,"num_read_kb":2453,"num_write":1391,"num_write_kb":19663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"48'722","reported_seq":985,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:58.533181+0000","last_change":"2024-05-27T00:33:51.952051+0000","last_active":"2024-05-27T00:33:58.533181+0000","last_peered":"2024-05-27T00:33:58.533181+0000","last_clean":"2024-05-27T00:33:58.533181+0000","last_became_active":"2024-05-27T00:15:24.627818+0000","last_became_peered":"2024-05-27T00:15:24.627818+0000","last_unstale":"2024-05-27T00:33:58.533181+0000","last_undegraded":"2024-05-27T00:33:58.533181+0000","last_fullsized":"2024-05-27T00:33:58.533181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'702","last_scrub_stamp":"2024-05-27T00:33:51.951893+0000","last_deep_scrub":"48'702","last_deep_scrub_stamp":"2024-05-27T00:33:51.951893+0000","last_clean_scrub_stamp":"2024-05-27T00:33:51.951893+0000","objects_scrubbed":54,"log_size":722,"log_dups_size":0,"ondisk_log_size":722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T00:47:50.026667+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9739985,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":352,"num_read_kb":95868,"num_write":837,"num_write_kb":48904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":875188,"num_omap_keys":1562,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"48'2135","reported_seq":4040,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:48.900437+0000","last_change":"2024-05-27T00:33:48.900506+0000","last_active":"2024-05-27T00:33:48.900437+0000","last_peered":"2024-05-27T00:33:48.900437+0000","last_clean":"2024-05-27T00:33:48.900437+0000","last_became_active":"2024-05-27T00:15:25.596511+0000","last_became_peered":"2024-05-27T00:15:25.596511+0000","last_unstale":"2024-05-27T00:33:48.900437+0000","last_undegraded":"2024-05-27T00:33:48.900437+0000","last_fullsized":"2024-05-27T00:33:48.900437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2135","last_scrub_stamp":"2024-05-27T00:33:48.900341+0000","last_deep_scrub":"48'2135","last_deep_scrub_stamp":"2024-05-27T00:33:48.900341+0000","last_clean_scrub_stamp":"2024-05-27T00:33:48.900341+0000","objects_scrubbed":583,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:27:09.151837+0000","scrub_duration":903,"objects_trimmed":0,"snaptrim_duration":0.018225775999999999,"stat_sum":{"num_bytes":9782754,"num_objects":583,"num_object_clones":270,"num_object_copies":1166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":186,"num_read":49,"num_read_kb":4155,"num_write":1427,"num_write_kb":16778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'547","reported_seq":780,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:54.954903+0000","last_change":"2024-05-27T00:33:54.954903+0000","last_active":"2024-05-27T00:33:54.954903+0000","last_peered":"2024-05-27T00:33:54.954903+0000","last_clean":"2024-05-27T00:33:54.954903+0000","last_became_active":"2024-05-27T00:15:24.627046+0000","last_became_peered":"2024-05-27T00:15:24.627046+0000","last_unstale":"2024-05-27T00:33:54.954903+0000","last_undegraded":"2024-05-27T00:33:54.954903+0000","last_fullsized":"2024-05-27T00:33:54.954903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'546","last_scrub_stamp":"2024-05-27T00:33:54.954699+0000","last_deep_scrub":"48'546","last_deep_scrub_stamp":"2024-05-27T00:33:54.954699+0000","last_clean_scrub_stamp":"2024-05-27T00:33:54.954699+0000","objects_scrubbed":78,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T02:48:58.995859+0000","scrub_duration":283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":297,"num_read_kb":59216,"num_write":641,"num_write_kb":42959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":72,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":998434,"num_omap_keys":1879,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"48'2226","reported_seq":4123,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:48.266492+0000","last_change":"2024-05-27T00:33:48.266492+0000","last_active":"2024-05-27T00:33:48.266492+0000","last_peered":"2024-05-27T00:33:48.266492+0000","last_clean":"2024-05-27T00:33:48.266492+0000","last_became_active":"2024-05-27T00:15:25.609831+0000","last_became_peered":"2024-05-27T00:15:25.609831+0000","last_unstale":"2024-05-27T00:33:48.266492+0000","last_undegraded":"2024-05-27T00:33:48.266492+0000","last_fullsized":"2024-05-27T00:33:48.266492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2225","last_scrub_stamp":"2024-05-27T00:33:48.266318+0000","last_deep_scrub":"48'2225","last_deep_scrub_stamp":"2024-05-27T00:33:48.266318+0000","last_clean_scrub_stamp":"2024-05-27T00:33:48.266318+0000","objects_scrubbed":622,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T04:10:45.063655+0000","scrub_duration":1319,"objects_trimmed":0,"snaptrim_duration":0.022802596000000001,"stat_sum":{"num_bytes":19605226,"num_objects":622,"num_object_clones":284,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":195,"num_read":52,"num_read_kb":30708,"num_write":1476,"num_write_kb":21111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"48'837","reported_seq":1018,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:47.776516+0000","last_change":"2024-05-27T00:33:47.776547+0000","last_active":"2024-05-27T00:33:47.776516+0000","last_peered":"2024-05-27T00:33:47.776516+0000","last_clean":"2024-05-27T00:33:47.776516+0000","last_became_active":"2024-05-27T00:15:24.609230+0000","last_became_peered":"2024-05-27T00:15:24.609230+0000","last_unstale":"2024-05-27T00:33:47.776516+0000","last_undegraded":"2024-05-27T00:33:47.776516+0000","last_fullsized":"2024-05-27T00:33:47.776516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'837","last_scrub_stamp":"2024-05-27T00:33:47.776458+0000","last_deep_scrub":"48'837","last_deep_scrub_stamp":"2024-05-27T00:33:47.776458+0000","last_clean_scrub_stamp":"2024-05-27T00:33:47.776458+0000","objects_scrubbed":57,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:24:31.323895+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":176,"num_read_kb":80391,"num_write":882,"num_write_kb":51073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":721727,"num_omap_keys":1319,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"48'2307","reported_seq":4202,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.150400+0000","last_change":"2024-05-27T00:33:51.151714+0000","last_active":"2024-05-27T00:33:59.150400+0000","last_peered":"2024-05-27T00:33:59.150400+0000","last_clean":"2024-05-27T00:33:59.150400+0000","last_became_active":"2024-05-27T00:15:25.596591+0000","last_became_peered":"2024-05-27T00:15:25.596591+0000","last_unstale":"2024-05-27T00:33:59.150400+0000","last_undegraded":"2024-05-27T00:33:59.150400+0000","last_fullsized":"2024-05-27T00:33:59.150400+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2305","last_scrub_stamp":"2024-05-27T00:33:51.151548+0000","last_deep_scrub":"48'2305","last_deep_scrub_stamp":"2024-05-27T00:33:51.151548+0000","last_clean_scrub_stamp":"2024-05-27T00:33:51.151548+0000","objects_scrubbed":645,"log_size":2307,"log_dups_size":0,"ondisk_log_size":2307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:28:22.737122+0000","scrub_duration":1543,"objects_trimmed":0,"snaptrim_duration":0.0084477519999999993,"stat_sum":{"num_bytes":19978404,"num_objects":645,"num_object_clones":297,"num_object_copies":1290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":206,"num_read":62,"num_read_kb":32956,"num_write":1523,"num_write_kb":26495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"48'1098","reported_seq":1333,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:44.664114+0000","last_change":"2024-05-27T00:33:44.664114+0000","last_active":"2024-05-27T00:33:44.664114+0000","last_peered":"2024-05-27T00:33:44.664114+0000","last_clean":"2024-05-27T00:33:44.664114+0000","last_became_active":"2024-05-27T00:15:24.597977+0000","last_became_peered":"2024-05-27T00:15:24.597977+0000","last_unstale":"2024-05-27T00:33:44.664114+0000","last_undegraded":"2024-05-27T00:33:44.664114+0000","last_fullsized":"2024-05-27T00:33:44.664114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1097","last_scrub_stamp":"2024-05-27T00:33:44.664017+0000","last_deep_scrub":"48'1097","last_deep_scrub_stamp":"2024-05-27T00:33:44.664017+0000","last_clean_scrub_stamp":"2024-05-27T00:33:44.664017+0000","objects_scrubbed":54,"log_size":1098,"log_dups_size":0,"ondisk_log_size":1098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:57:50.619399+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":269,"num_read_kb":153672,"num_write":1142,"num_write_kb":75700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":695727,"num_omap_keys":1211,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"48'2250","reported_seq":4126,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.153159+0000","last_change":"2024-05-27T00:33:52.050450+0000","last_active":"2024-05-27T00:33:59.153159+0000","last_peered":"2024-05-27T00:33:59.153159+0000","last_clean":"2024-05-27T00:33:59.153159+0000","last_became_active":"2024-05-27T00:15:25.600837+0000","last_became_peered":"2024-05-27T00:15:25.600837+0000","last_unstale":"2024-05-27T00:33:59.153159+0000","last_undegraded":"2024-05-27T00:33:59.153159+0000","last_fullsized":"2024-05-27T00:33:59.153159+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2246","last_scrub_stamp":"2024-05-27T00:33:52.050265+0000","last_deep_scrub":"48'2246","last_deep_scrub_stamp":"2024-05-27T00:33:52.050265+0000","last_clean_scrub_stamp":"2024-05-27T00:33:52.050265+0000","objects_scrubbed":631,"log_size":2250,"log_dups_size":0,"ondisk_log_size":2250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:48:01.029232+0000","scrub_duration":1443,"objects_trimmed":0,"snaptrim_duration":0.0091147599999999995,"stat_sum":{"num_bytes":15864263,"num_objects":631,"num_object_clones":289,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":195,"num_read":35,"num_read_kb":8228,"num_write":1477,"num_write_kb":20882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"48'765","reported_seq":1029,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:53.850590+0000","last_change":"2024-05-27T00:33:53.850667+0000","last_active":"2024-05-27T00:33:53.850590+0000","last_peered":"2024-05-27T00:33:53.850590+0000","last_clean":"2024-05-27T00:33:53.850590+0000","last_became_active":"2024-05-27T00:15:24.628206+0000","last_became_peered":"2024-05-27T00:15:24.628206+0000","last_unstale":"2024-05-27T00:33:53.850590+0000","last_undegraded":"2024-05-27T00:33:53.850590+0000","last_fullsized":"2024-05-27T00:33:53.850590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'765","last_scrub_stamp":"2024-05-27T00:33:53.850469+0000","last_deep_scrub":"48'765","last_deep_scrub_stamp":"2024-05-27T00:33:53.850469+0000","last_clean_scrub_stamp":"2024-05-27T00:33:53.850469+0000","objects_scrubbed":72,"log_size":765,"log_dups_size":0,"ondisk_log_size":765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:12:08.708503+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":72,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":331,"num_read_kb":161875,"num_write":847,"num_write_kb":84354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":63,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":652670,"num_omap_keys":1127,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"48'2174","reported_seq":4063,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:57.650006+0000","last_change":"2024-05-27T00:33:57.650006+0000","last_active":"2024-05-27T00:33:57.650006+0000","last_peered":"2024-05-27T00:33:57.650006+0000","last_clean":"2024-05-27T00:33:57.650006+0000","last_became_active":"2024-05-27T00:15:25.618939+0000","last_became_peered":"2024-05-27T00:15:25.618939+0000","last_unstale":"2024-05-27T00:33:57.650006+0000","last_undegraded":"2024-05-27T00:33:57.650006+0000","last_fullsized":"2024-05-27T00:33:57.650006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2173","last_scrub_stamp":"2024-05-27T00:33:57.649878+0000","last_deep_scrub":"48'2173","last_deep_scrub_stamp":"2024-05-27T00:33:57.649878+0000","last_clean_scrub_stamp":"2024-05-27T00:33:57.649878+0000","objects_scrubbed":584,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:17:08.552033+0000","scrub_duration":719,"objects_trimmed":0,"snaptrim_duration":0.026544363000000001,"stat_sum":{"num_bytes":11205938,"num_objects":584,"num_object_clones":274,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":172,"num_read":36,"num_read_kb":3645,"num_write":1480,"num_write_kb":12517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"48'2118","reported_seq":4015,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:55.071138+0000","last_change":"2024-05-27T00:33:55.071169+0000","last_active":"2024-05-27T00:33:55.071138+0000","last_peered":"2024-05-27T00:33:55.071138+0000","last_clean":"2024-05-27T00:33:55.071138+0000","last_became_active":"2024-05-27T00:16:06.811922+0000","last_became_peered":"2024-05-27T00:16:06.811922+0000","last_unstale":"2024-05-27T00:33:55.071138+0000","last_undegraded":"2024-05-27T00:33:55.071138+0000","last_fullsized":"2024-05-27T00:33:55.071138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2118","last_scrub_stamp":"2024-05-27T00:33:55.071025+0000","last_deep_scrub":"48'2118","last_deep_scrub_stamp":"2024-05-27T00:33:55.071025+0000","last_clean_scrub_stamp":"2024-05-27T00:33:55.071025+0000","objects_scrubbed":589,"log_size":2118,"log_dups_size":0,"ondisk_log_size":2118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:24:38.618462+0000","scrub_duration":1531,"objects_trimmed":0,"snaptrim_duration":0.011085174,"stat_sum":{"num_bytes":14355433,"num_objects":589,"num_object_clones":272,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":196,"num_read":48,"num_read_kb":4199,"num_write":1386,"num_write_kb":25841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"48'552","reported_seq":776,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:51.771380+0000","last_change":"2024-05-27T00:33:51.771475+0000","last_active":"2024-05-27T00:33:51.771380+0000","last_peered":"2024-05-27T00:33:51.771380+0000","last_clean":"2024-05-27T00:33:51.771380+0000","last_became_active":"2024-05-27T00:15:24.626881+0000","last_became_peered":"2024-05-27T00:15:24.626881+0000","last_unstale":"2024-05-27T00:33:51.771380+0000","last_undegraded":"2024-05-27T00:33:51.771380+0000","last_fullsized":"2024-05-27T00:33:51.771380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'552","last_scrub_stamp":"2024-05-27T00:33:51.771238+0000","last_deep_scrub":"48'552","last_deep_scrub_stamp":"2024-05-27T00:33:51.771238+0000","last_clean_scrub_stamp":"2024-05-27T00:33:51.771238+0000","objects_scrubbed":63,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:10:57.421841+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":267,"num_read_kb":124158,"num_write":637,"num_write_kb":63683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":56,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":899974,"num_omap_keys":1549,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"48'2312","reported_seq":4175,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:59.627849+0000","last_change":"2024-05-27T00:33:59.627953+0000","last_active":"2024-05-27T00:33:59.627849+0000","last_peered":"2024-05-27T00:33:59.627849+0000","last_clean":"2024-05-27T00:33:59.627849+0000","last_became_active":"2024-05-27T00:15:25.622132+0000","last_became_peered":"2024-05-27T00:15:25.622132+0000","last_unstale":"2024-05-27T00:33:59.627849+0000","last_undegraded":"2024-05-27T00:33:59.627849+0000","last_fullsized":"2024-05-27T00:33:59.627849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2312","last_scrub_stamp":"2024-05-27T00:33:59.627777+0000","last_deep_scrub":"48'2312","last_deep_scrub_stamp":"2024-05-27T00:33:59.627777+0000","last_clean_scrub_stamp":"2024-05-27T00:33:59.627777+0000","objects_scrubbed":648,"log_size":2312,"log_dups_size":0,"ondisk_log_size":2312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:27:37.527933+0000","scrub_duration":959,"objects_trimmed":0,"snaptrim_duration":0.034279472999999998,"stat_sum":{"num_bytes":10551410,"num_objects":648,"num_object_clones":309,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":224,"num_read":43,"num_read_kb":11171,"num_write":1487,"num_write_kb":16962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"48'921","reported_seq":1189,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:56.102396+0000","last_change":"2024-05-27T00:33:56.102443+0000","last_active":"2024-05-27T00:33:56.102396+0000","last_peered":"2024-05-27T00:33:56.102396+0000","last_clean":"2024-05-27T00:33:56.102396+0000","last_became_active":"2024-05-27T00:15:24.633765+0000","last_became_peered":"2024-05-27T00:15:24.633765+0000","last_unstale":"2024-05-27T00:33:56.102396+0000","last_undegraded":"2024-05-27T00:33:56.102396+0000","last_fullsized":"2024-05-27T00:33:56.102396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'921","last_scrub_stamp":"2024-05-27T00:33:56.102301+0000","last_deep_scrub":"48'921","last_deep_scrub_stamp":"2024-05-27T00:33:56.102301+0000","last_clean_scrub_stamp":"2024-05-27T00:33:56.102301+0000","objects_scrubbed":63,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:50:29.064028+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":337,"num_read_kb":206604,"num_write":987,"num_write_kb":108718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":645336,"num_omap_keys":1164,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"48'2150","reported_seq":4054,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:52.443111+0000","last_change":"2024-05-27T00:33:52.443192+0000","last_active":"2024-05-27T00:33:52.443111+0000","last_peered":"2024-05-27T00:33:52.443111+0000","last_clean":"2024-05-27T00:33:52.443111+0000","last_became_active":"2024-05-27T00:15:25.612675+0000","last_became_peered":"2024-05-27T00:15:25.612675+0000","last_unstale":"2024-05-27T00:33:52.443111+0000","last_undegraded":"2024-05-27T00:33:52.443111+0000","last_fullsized":"2024-05-27T00:33:52.443111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2150","last_scrub_stamp":"2024-05-27T00:33:52.443014+0000","last_deep_scrub":"48'2150","last_deep_scrub_stamp":"2024-05-27T00:33:52.443014+0000","last_clean_scrub_stamp":"2024-05-27T00:33:52.443014+0000","objects_scrubbed":609,"log_size":2150,"log_dups_size":0,"ondisk_log_size":2150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:01:01.522554+0000","scrub_duration":1391,"objects_trimmed":0,"snaptrim_duration":0.017325917,"stat_sum":{"num_bytes":23014283,"num_objects":609,"num_object_clones":284,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":183,"num_read":59,"num_read_kb":40021,"num_write":1420,"num_write_kb":24406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"48'1339","reported_seq":1607,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:45.718003+0000","last_change":"2024-05-27T00:33:45.718003+0000","last_active":"2024-05-27T00:33:45.718003+0000","last_peered":"2024-05-27T00:33:45.718003+0000","last_clean":"2024-05-27T00:33:45.718003+0000","last_became_active":"2024-05-27T00:15:24.600331+0000","last_became_peered":"2024-05-27T00:15:24.600331+0000","last_unstale":"2024-05-27T00:33:45.718003+0000","last_undegraded":"2024-05-27T00:33:45.718003+0000","last_fullsized":"2024-05-27T00:33:45.718003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'1338","last_scrub_stamp":"2024-05-27T00:33:45.717878+0000","last_deep_scrub":"48'1338","last_deep_scrub_stamp":"2024-05-27T00:33:45.717878+0000","last_clean_scrub_stamp":"2024-05-27T00:33:45.717878+0000","objects_scrubbed":67,"log_size":1339,"log_dups_size":0,"ondisk_log_size":1339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:49.255280+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":353,"num_read_kb":145683,"num_write":1468,"num_write_kb":63909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":983521,"num_omap_keys":1776,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"48'2142","reported_seq":4052,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:56.775699+0000","last_change":"2024-05-27T00:33:56.775699+0000","last_active":"2024-05-27T00:33:56.775699+0000","last_peered":"2024-05-27T00:33:56.775699+0000","last_clean":"2024-05-27T00:33:56.775699+0000","last_became_active":"2024-05-27T00:15:25.615362+0000","last_became_peered":"2024-05-27T00:15:25.615362+0000","last_unstale":"2024-05-27T00:33:56.775699+0000","last_undegraded":"2024-05-27T00:33:56.775699+0000","last_fullsized":"2024-05-27T00:33:56.775699+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'2141","last_scrub_stamp":"2024-05-27T00:33:56.775580+0000","last_deep_scrub":"48'2141","last_deep_scrub_stamp":"2024-05-27T00:33:56.775580+0000","last_clean_scrub_stamp":"2024-05-27T00:33:56.775580+0000","objects_scrubbed":590,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T03:46:04.203947+0000","scrub_duration":1115,"objects_trimmed":0,"snaptrim_duration":0.0089714449999999998,"stat_sum":{"num_bytes":23597398,"num_objects":590,"num_object_clones":273,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":185,"num_read":53,"num_read_kb":39809,"num_write":1432,"num_write_kb":24946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"48'590","reported_seq":841,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-05-27T00:33:45.035783+0000","last_change":"2024-05-27T00:33:45.035806+0000","last_active":"2024-05-27T00:33:45.035783+0000","last_peered":"2024-05-27T00:33:45.035783+0000","last_clean":"2024-05-27T00:33:45.035783+0000","last_became_active":"2024-05-27T00:15:24.614792+0000","last_became_peered":"2024-05-27T00:15:24.614792+0000","last_unstale":"2024-05-27T00:33:45.035783+0000","last_undegraded":"2024-05-27T00:33:45.035783+0000","last_fullsized":"2024-05-27T00:33:45.035783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'590","last_scrub_stamp":"2024-05-27T00:33:45.035706+0000","last_deep_scrub":"48'590","last_deep_scrub_stamp":"2024-05-27T00:33:45.035706+0000","last_clean_scrub_stamp":"2024-05-27T00:33:45.035706+0000","objects_scrubbed":75,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T05:35:08.289189+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":75,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":327,"num_read_kb":63931,"num_write":705,"num_write_kb":51511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":64,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":878537,"num_omap_keys":1434,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1180698353,"num_objects":39309,"num_object_clones":18317,"num_object_copies":78618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39309,"num_whiteouts":12400,"num_read":3379,"num_read_kb":1679737,"num_write":93168,"num_write_kb":1481682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2455085056,"data_stored":2363411938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141429,"ondisk_log_size":141429,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":555374061,"num_objects":2078,"num_object_clones":0,"num_object_copies":4156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":10398,"num_read_kb":4541572,"num_write":34324,"num_write_kb":2263108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1861,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":26114489,"num_omap_keys":45815,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1110802432,"data_stored":1110748122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30880,"ondisk_log_size":30880,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542374,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722892,"kb_used_data":557124,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93648948,"statfs":{"total":96636764160,"available":95896522752,"internally_reserved":0,"allocated":570494976,"data_stored":557282144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":169736650},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542375,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514108,"kb_used_data":392756,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":93857732,"statfs":{"total":96636764160,"available":96110317568,"internally_reserved":0,"allocated":402182144,"data_stored":393172585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542376,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707296,"kb_used_data":523480,"kb_used_omap":1,"kb_used_meta":183806,"kb_avail":93664544,"statfs":{"total":96636764160,"available":95912493056,"internally_reserved":0,"allocated":536043520,"data_stored":522333872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188217797},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542374,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644272,"kb_used_data":478504,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93727568,"statfs":{"total":96636764160,"available":95977029632,"internally_reserved":0,"allocated":489988096,"data_stored":475844387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169736645},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542375,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631908,"kb_used_data":474460,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93739932,"statfs":{"total":96636764160,"available":95989690368,"internally_reserved":0,"allocated":485847040,"data_stored":471784053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161216967},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542374,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387820,"kb_used_data":276196,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":93984020,"statfs":{"total":96636764160,"available":96239636480,"internally_reserved":0,"allocated":282824704,"data_stored":274404683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114293189},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542375,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508240,"kb_used_data":368840,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93863600,"statfs":{"total":96636764160,"available":96116326400,"internally_reserved":0,"allocated":377692160,"data_stored":369492720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142735815},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542374,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582136,"kb_used_data":416368,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93789704,"statfs":{"total":96636764160,"available":96040656896,"internally_reserved":0,"allocated":426360832,"data_stored":413011336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169736645},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100675584,"data_stored":100663934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163590144,"data_stored":163582517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62914560,"data_stored":62914560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":142614528,"data_stored":142607008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":117456896,"data_stored":117445139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":151367680,"data_stored":151363619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":169500672,"data_stored":169492794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":202682368,"data_stored":202678551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":324546560,"data_stored":311508911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213557248,"data_stored":205662080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219365376,"data_stored":211242000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":342687744,"data_stored":328928922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":371986432,"data_stored":358151125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":384131072,"data_stored":370722130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":232136704,"data_stored":223431668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":366673920,"data_stored":353765102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T00:34:08.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-27T00:34:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.850+0000 7fe9b7200640 1 Processor -- start 2024-05-27T00:34:08.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.850+0000 7fe9b7200640 1 -- start start 2024-05-27T00:34:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9b7200640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe9b0154de0 0x7fe9b015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9b7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9b00b4dd0 0x7fe9b00cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9b7200640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b0072dc0 con 0x7fe9b0071a90 2024-05-27T00:34:08.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9b7200640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe9b01551c0 con 0x7fe9b0154de0 2024-05-27T00:34:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9b7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe9b0071f60 con 0x7fe9b00b4dd0 2024-05-27T00:34:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9afe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe9b0154de0 0x7fe9b015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9afe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe9b0154de0 0x7fe9b015aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35280/0 (socket says 172.21.15.98:35280) 2024-05-27T00:34:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9afe00640 1 -- 172.21.15.98:0/4015740265 learned_addr learned my addr 172.21.15.98:0/4015740265 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9b5200640 1 --2- 172.21.15.98:0/4015740265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9b00b4dd0 0x7fe9b00cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9afe00640 1 -- 172.21.15.98:0/4015740265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9b00b4dd0 msgr2=0x7fe9b00cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9afe00640 1 --2- 172.21.15.98:0/4015740265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9b00b4dd0 0x7fe9b00cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9afe00640 1 -- 172.21.15.98:0/4015740265 >> v1:172.21.15.98:6789/0 conn(0x7fe9b0071a90 legacy=0x7fe9b00b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:08.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9afe00640 1 -- 172.21.15.98:0/4015740265 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b0072240 con 0x7fe9b0154de0 2024-05-27T00:34:08.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9afe00640 1 --2- 172.21.15.98:0/4015740265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 0x7fe9b015aa00 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0009870 tx=0x7fe9a001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f23ed81a2ffebb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.854+0000 7fe9af400640 1 -- 172.21.15.98:0/4015740265 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a00b5070 con 0x7fe9b0154de0 2024-05-27T00:34:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9af400640 1 -- 172.21.15.98:0/4015740265 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9a00a8a50 con 0x7fe9b0154de0 2024-05-27T00:34:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9af400640 1 -- 172.21.15.98:0/4015740265 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a00a8e20 con 0x7fe9b0154de0 2024-05-27T00:34:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 -- 172.21.15.98:0/4015740265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 msgr2=0x7fe9b015aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 --2- 172.21.15.98:0/4015740265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 0x7fe9b015aa00 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0009870 tx=0x7fe9a001ed50 comp rx=0 tx=0).stop 2024-05-27T00:34:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 -- 172.21.15.98:0/4015740265 shutdown_connections 2024-05-27T00:34:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 --2- 172.21.15.98:0/4015740265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9b00b4dd0 0x7fe9b00cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 --2- 172.21.15.98:0/4015740265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 0x7fe9b015aa00 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 -- 172.21.15.98:0/4015740265 >> 172.21.15.98:0/4015740265 conn(0x7fe9b00ccaf0 msgr2=0x7fe9b00ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 -- 172.21.15.98:0/4015740265 shutdown_connections 2024-05-27T00:34:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 -- 172.21.15.98:0/4015740265 wait complete. 2024-05-27T00:34:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 Processor -- start 2024-05-27T00:34:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 -- start start 2024-05-27T00:34:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9b0071a90 0x7fe9b0155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9b00b4dd0 0x7fe9b0153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 0x7fe9b0153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b0072ae0 con 0x7fe9b0154de0 2024-05-27T00:34:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b0072800 con 0x7fe9b0071a90 2024-05-27T00:34:08.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe9b00cd900 con 0x7fe9b00b4dd0 2024-05-27T00:34:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b5200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 0x7fe9b0153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b5200640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 0x7fe9b0153f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35290/0 (socket says 172.21.15.98:35290) 2024-05-27T00:34:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9b5200640 1 -- 172.21.15.98:0/3741025050 learned_addr learned my addr 172.21.15.98:0/3741025050 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9a7e00640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9b00b4dd0 0x7fe9b0153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.858+0000 7fe9afe00640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9b0071a90 0x7fe9b0155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9b5200640 1 -- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9b00b4dd0 msgr2=0x7fe9b0153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9b5200640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9b00b4dd0 0x7fe9b0153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9b5200640 1 -- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9b0071a90 msgr2=0x7fe9b0155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9b5200640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9b0071a90 0x7fe9b0155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9b5200640 1 -- 172.21.15.98:0/3741025050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b015ec50 con 0x7fe9b0154de0 2024-05-27T00:34:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9afe00640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9b0071a90 0x7fe9b0155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:34:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9b5200640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 0x7fe9b0153f80 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8007e60 tx=0x7fe9a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9acc00640 1 -- 172.21.15.98:0/3741025050 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8014420 con 0x7fe9b0154de0 2024-05-27T00:34:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9acc00640 1 -- 172.21.15.98:0/3741025050 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9a8019070 con 0x7fe9b0154de0 2024-05-27T00:34:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9acc00640 1 -- 172.21.15.98:0/3741025050 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8014820 con 0x7fe9b0154de0 2024-05-27T00:34:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9b7200640 1 -- 172.21.15.98:0/3741025050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b00c59d0 con 0x7fe9b0154de0 2024-05-27T00:34:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9b7200640 1 -- 172.21.15.98:0/3741025050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9b015e970 con 0x7fe9b0154de0 2024-05-27T00:34:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9b7200640 1 -- 172.21.15.98:0/3741025050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9b0075cb0 con 0x7fe9b0154de0 2024-05-27T00:34:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9acc00640 1 -- 172.21.15.98:0/3741025050 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe9a80149c0 con 0x7fe9b0154de0 2024-05-27T00:34:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9acc00640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe98c0678c0 0x7fe98c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9acc00640 1 -- 172.21.15.98:0/3741025050 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7fe9a8086850 con 0x7fe9b0154de0 2024-05-27T00:34:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.862+0000 7fe9afe00640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe98c0678c0 0x7fe98c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.866+0000 7fe9afe00640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe98c0678c0 0x7fe98c069d80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe9a00b2000 tx=0x7fe9a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:08.866+0000 7fe9acc00640 1 -- 172.21.15.98:0/3741025050 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7fe9a8056ef0 con 0x7fe9b0154de0 2024-05-27T00:34:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.026+0000 7fe9b7200640 1 -- 172.21.15.98:0/3741025050 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fe9b00c68a0 con 0x7fe9b0154de0 2024-05-27T00:34:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.030+0000 7fe9acc00640 1 -- 172.21.15.98:0/3741025050 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7fe9a805d0c0 con 0x7fe9b0154de0 2024-05-27T00:34:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.030+0000 7fe9acc00640 1 -- 172.21.15.98:0/3741025050 <== mon.0 v2:172.21.15.98:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe9a80567f0 con 0x7fe9b0154de0 2024-05-27T00:34:09.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 -- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe98c0678c0 msgr2=0x7fe98c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe98c0678c0 0x7fe98c069d80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe9a00b2000 tx=0x7fe9a0008040 comp rx=0 tx=0).stop 2024-05-27T00:34:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 -- 172.21.15.98:0/3741025050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 msgr2=0x7fe9b0153f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 0x7fe9b0153f80 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8007e60 tx=0x7fe9a800e4b0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 -- 172.21.15.98:0/3741025050 shutdown_connections 2024-05-27T00:34:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b0154de0 0x7fe9b0153f80 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9b00b4dd0 0x7fe9b0153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7fe98c0678c0 0x7fe98c069d80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 --2- 172.21.15.98:0/3741025050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9b0071a90 0x7fe9b0155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 -- 172.21.15.98:0/3741025050 >> 172.21.15.98:0/3741025050 conn(0x7fe9b00ccaf0 msgr2=0x7fe9b00ce940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 -- 172.21.15.98:0/3741025050 shutdown_connections 2024-05-27T00:34:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.034+0000 7fe9b7200640 1 -- 172.21.15.98:0/3741025050 wait complete. 2024-05-27T00:34:09.053 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-27T00:34:09.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:34:09.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6859800640 1 Processor -- start 2024-05-27T00:34:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6859800640 1 -- start start 2024-05-27T00:34:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6859800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f6854157cd0 0x7f68540c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6859800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68541580b0 0x7f685415e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6859800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68540c59d0 con 0x7f68540b3cb0 2024-05-27T00:34:09.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6859800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f68540c51e0 con 0x7f6854157cd0 2024-05-27T00:34:09.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6859800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f68540cabb0 con 0x7f68541580b0 2024-05-27T00:34:09.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6858e00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f68540b3cb0 0x7f68540c9a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57670/0 (socket says 172.21.15.98:57670) 2024-05-27T00:34:09.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6858e00640 1 -- 172.21.15.98:0/4028354034 learned_addr learned my addr 172.21.15.98:0/4028354034 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:34:09.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f684be00640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68541580b0 0x7f685415e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853e00640 1 --2- 172.21.15.98:0/4028354034 >> v2:172.21.15.204:3300/0 conn(0x7f6854157cd0 0x7f68540c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3096748028 0 0) 0x7f68540c59d0 con 0x7f68540b3cb0 2024-05-27T00:34:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6838003530 con 0x7f68540b3cb0 2024-05-27T00:34:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1660961 0 0) 0x7f6838003530 con 0x7f68540b3cb0 2024-05-27T00:34:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68541580b0 msgr2=0x7f685415e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68541580b0 0x7f685415e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 >> v2:172.21.15.204:3300/0 conn(0x7f6854157cd0 msgr2=0x7f68540c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 --2- 172.21.15.98:0/4028354034 >> v2:172.21.15.204:3300/0 conn(0x7f6854157cd0 0x7f68540c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68540cae50 con 0x7f68540b3cb0 2024-05-27T00:34:09.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f684c001a00 con 0x7f68540b3cb0 2024-05-27T00:34:09.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 >> v1:172.21.15.98:6789/0 conn(0x7f68540b3cb0 legacy=0x7f68540c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68380039c0 0x7f6838003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68541580b0 0x7f6838004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6838004d50 0x7f68380051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68540c51e0 con 0x7f6838004d50 2024-05-27T00:34:09.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f68540cabb0 con 0x7f68380039c0 2024-05-27T00:34:09.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f683800b9d0 con 0x7f68541580b0 2024-05-27T00:34:09.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6858e00640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6838004d50 0x7f68380051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853e00640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68380039c0 0x7f6838003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f684be00640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68541580b0 0x7f6838004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6858e00640 1 -- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68541580b0 msgr2=0x7f6838004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:09.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6858e00640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68541580b0 0x7f6838004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6858e00640 1 -- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68380039c0 msgr2=0x7f6838003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:09.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6858e00640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68380039c0 0x7f6838003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6858e00640 1 -- 172.21.15.98:0/4028354034 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6838003530 con 0x7f6838004d50 2024-05-27T00:34:09.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f684be00640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68541580b0 0x7f6838004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:34:09.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.194+0000 7f6853e00640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68380039c0 0x7f6838003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:34:09.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6858e00640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6838004d50 0x7f68380051d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f684c00a920 tx=0x7f684c01d3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39f9ce3196f4ea6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:09.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c00a4c0 con 0x7f6838004d50 2024-05-27T00:34:09.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f684c0072f0 con 0x7f6838004d50 2024-05-27T00:34:09.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6853400640 1 -- 172.21.15.98:0/4028354034 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c007700 con 0x7f6838004d50 2024-05-27T00:34:09.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6859800640 1 -- 172.21.15.98:0/4028354034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6838004d50 msgr2=0x7f68380051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6859800640 1 --2- 172.21.15.98:0/4028354034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6838004d50 0x7f68380051d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f684c00a920 tx=0x7f684c01d3e0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6858e00640 1 -- 172.21.15.98:0/4028354034 reap_dead start 2024-05-27T00:34:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6859800640 1 -- 172.21.15.98:0/4028354034 shutdown_connections 2024-05-27T00:34:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6859800640 1 -- 172.21.15.98:0/4028354034 >> 172.21.15.98:0/4028354034 conn(0x7f6854070b30 msgr2=0x7f6854070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6859800640 1 -- 172.21.15.98:0/4028354034 shutdown_connections 2024-05-27T00:34:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.198+0000 7f6859800640 1 -- 172.21.15.98:0/4028354034 wait complete. 2024-05-27T00:34:09.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6859800640 1 Processor -- start 2024-05-27T00:34:09.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6859800640 1 -- start start 2024-05-27T00:34:09.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6859800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6838004d50 0x7f68540c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6859800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68540c6ed0 0x7f6854156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6859800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f685414d450 0x7f6854143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6859800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f685415e120 con 0x7f68540c6ed0 2024-05-27T00:34:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6859800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f685415de40 con 0x7f6838004d50 2024-05-27T00:34:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6859800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f68540cb410 con 0x7f685414d450 2024-05-27T00:34:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f684be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68540c6ed0 0x7f6854156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f684be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68540c6ed0 0x7f6854156520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35308/0 (socket says 172.21.15.98:35308) 2024-05-27T00:34:09.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f684be00640 1 -- 172.21.15.98:0/3145147203 learned_addr learned my addr 172.21.15.98:0/3145147203 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6853e00640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6838004d50 0x7f68540c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6858e00640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f685414d450 0x7f6854143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f684be00640 1 -- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f685414d450 msgr2=0x7f6854143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f684be00640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f685414d450 0x7f6854143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f684be00640 1 -- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6838004d50 msgr2=0x7f68540c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f684be00640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6838004d50 0x7f68540c76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f684be00640 1 -- 172.21.15.98:0/3145147203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68540c7be0 con 0x7f68540c6ed0 2024-05-27T00:34:09.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f684be00640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68540c6ed0 0x7f6854156520 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f683c0116a0 tx=0x7f683c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:09.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6850c00640 1 -- 172.21.15.98:0/3145147203 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c004220 con 0x7f68540c6ed0 2024-05-27T00:34:09.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6850c00640 1 -- 172.21.15.98:0/3145147203 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f683c0043c0 con 0x7f68540c6ed0 2024-05-27T00:34:09.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6859800640 1 -- 172.21.15.98:0/3145147203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6854150d70 con 0x7f68540c6ed0 2024-05-27T00:34:09.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6850c00640 1 -- 172.21.15.98:0/3145147203 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c00f9a0 con 0x7f68540c6ed0 2024-05-27T00:34:09.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.202+0000 7f6859800640 1 -- 172.21.15.98:0/3145147203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f685415b990 con 0x7f68540c6ed0 2024-05-27T00:34:09.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.206+0000 7f6859800640 1 -- 172.21.15.98:0/3145147203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6854075c40 con 0x7f68540c6ed0 2024-05-27T00:34:09.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.206+0000 7f6850c00640 1 -- 172.21.15.98:0/3145147203 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f683c00fb40 con 0x7f68540c6ed0 2024-05-27T00:34:09.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.206+0000 7f6850c00640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68300678c0 0x7f6830069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.206+0000 7f6850c00640 1 -- 172.21.15.98:0/3145147203 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f683c086960 con 0x7f68540c6ed0 2024-05-27T00:34:09.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.206+0000 7f6853e00640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68300678c0 0x7f6830069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.206+0000 7f6853e00640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68300678c0 0x7f6830069d80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f68440099b0 tx=0x7f6844008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:09.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.210+0000 7f6850c00640 1 -- 172.21.15.98:0/3145147203 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f683c019030 con 0x7f68540c6ed0 2024-05-27T00:34:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.398+0000 7f6859800640 1 -- 172.21.15.98:0/3145147203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f685415abd0 con 0x7f68540c6ed0 2024-05-27T00:34:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.398+0000 7f6850c00640 1 -- 172.21.15.98:0/3145147203 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6839 (secure 0 0 0) 0x7f683c056ef0 con 0x7f68540c6ed0 2024-05-27T00:34:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:34:09.400 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3675441597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3675441597},{"type":"v1","addr":"172.21.15.98:6834","nonce":3675441597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:32.656788+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6396,"qdb_cluster":[6396]},"id":1}]} 2024-05-27T00:34:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 73 2024-05-27T00:34:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 -- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68300678c0 msgr2=0x7f6830069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68300678c0 0x7f6830069d80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f68440099b0 tx=0x7f6844008040 comp rx=0 tx=0).stop 2024-05-27T00:34:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 -- 172.21.15.98:0/3145147203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68540c6ed0 msgr2=0x7f6854156520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68540c6ed0 0x7f6854156520 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f683c0116a0 tx=0x7f683c011b60 comp rx=0 tx=0).stop 2024-05-27T00:34:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 -- 172.21.15.98:0/3145147203 shutdown_connections 2024-05-27T00:34:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f685414d450 0x7f6854143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68540c6ed0 0x7f6854156520 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f68300678c0 0x7f6830069d80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 --2- 172.21.15.98:0/3145147203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6838004d50 0x7f68540c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 -- 172.21.15.98:0/3145147203 >> 172.21.15.98:0/3145147203 conn(0x7f6854070b30 msgr2=0x7f6854150020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 -- 172.21.15.98:0/3145147203 shutdown_connections 2024-05-27T00:34:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.402+0000 7f6859800640 1 -- 172.21.15.98:0/3145147203 wait complete. 2024-05-27T00:34:09.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T00:34:09.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.554+0000 7f58c7800640 1 Processor -- start 2024-05-27T00:34:09.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.554+0000 7f58c7800640 1 -- start start 2024-05-27T00:34:09.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58c7800640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f58c00cc8f0 0x7f58c0070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58c7800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 0x7f58c00c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58c7800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c00c1000 con 0x7f58c01511d0 2024-05-27T00:34:09.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58c7800640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f58c00b1dd0 con 0x7f58c00cc8f0 2024-05-27T00:34:09.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58c7800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f58c00c1a20 con 0x7f58c00c8670 2024-05-27T00:34:09.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58bfe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f58c00cc8f0 0x7f58c0070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58bfe00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f58c00cc8f0 0x7f58c0070e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35312/0 (socket says 172.21.15.98:35312) 2024-05-27T00:34:09.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58bfe00640 1 -- 172.21.15.98:0/135956322 learned_addr learned my addr 172.21.15.98:0/135956322 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:09.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58c5800640 1 --2- 172.21.15.98:0/135956322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 0x7f58c00c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58bfe00640 1 -- 172.21.15.98:0/135956322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 msgr2=0x7f58c00c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:09.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58bfe00640 1 --2- 172.21.15.98:0/135956322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 0x7f58c00c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58bfe00640 1 -- 172.21.15.98:0/135956322 >> v1:172.21.15.98:6789/0 conn(0x7f58c01511d0 legacy=0x7f58c00c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58bfe00640 1 -- 172.21.15.98:0/135956322 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c01515b0 con 0x7f58c00cc8f0 2024-05-27T00:34:09.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58c5800640 1 --2- 172.21.15.98:0/135956322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 0x7f58c00c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:34:09.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.558+0000 7f58bfe00640 1 --2- 172.21.15.98:0/135956322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c00cc8f0 0x7f58c0070e10 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009870 tx=0x7f58ac01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da8cb9792fa126d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:09.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58bf400640 1 -- 172.21.15.98:0/135956322 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac0b0070 con 0x7f58c00cc8f0 2024-05-27T00:34:09.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58bf400640 1 -- 172.21.15.98:0/135956322 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f58ac0ab600 con 0x7f58c00cc8f0 2024-05-27T00:34:09.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58bf400640 1 -- 172.21.15.98:0/135956322 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ac0ab9f0 con 0x7f58c00cc8f0 2024-05-27T00:34:09.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 -- 172.21.15.98:0/135956322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c00cc8f0 msgr2=0x7f58c0070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 --2- 172.21.15.98:0/135956322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c00cc8f0 0x7f58c0070e10 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009870 tx=0x7f58ac01c4d0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 -- 172.21.15.98:0/135956322 shutdown_connections 2024-05-27T00:34:09.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 --2- 172.21.15.98:0/135956322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 0x7f58c00c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 --2- 172.21.15.98:0/135956322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c00cc8f0 0x7f58c0070e10 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 -- 172.21.15.98:0/135956322 >> 172.21.15.98:0/135956322 conn(0x7f58c0067530 msgr2=0x7f58c006ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:09.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 -- 172.21.15.98:0/135956322 shutdown_connections 2024-05-27T00:34:09.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 -- 172.21.15.98:0/135956322 wait complete. 2024-05-27T00:34:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 Processor -- start 2024-05-27T00:34:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 -- start start 2024-05-27T00:34:09.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 0x7f58c0143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58c00cc8f0 0x7f58c00c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c01511d0 0x7f58c00c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f58c00c2150 con 0x7f58c01511d0 2024-05-27T00:34:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f58c00b1a60 con 0x7f58c00cc8f0 2024-05-27T00:34:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c7800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f58c00c22d0 con 0x7f58c00c8670 2024-05-27T00:34:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c5800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c01511d0 0x7f58c00c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c5800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c01511d0 0x7f58c00c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35316/0 (socket says 172.21.15.98:35316) 2024-05-27T00:34:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c5800640 1 -- 172.21.15.98:0/3626699987 learned_addr learned my addr 172.21.15.98:0/3626699987 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58bfe00640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58c00cc8f0 0x7f58c00c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c4e00640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 0x7f58c0143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c5800640 1 -- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 msgr2=0x7f58c0143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c5800640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 0x7f58c0143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c5800640 1 -- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58c00cc8f0 msgr2=0x7f58c00c6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c5800640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58c00cc8f0 0x7f58c00c6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.562+0000 7f58c5800640 1 -- 172.21.15.98:0/3626699987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c00afe30 con 0x7f58c01511d0 2024-05-27T00:34:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58c5800640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c01511d0 0x7f58c00c6fc0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f58b8007f20 tx=0x7f58b800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58bcc00640 1 -- 172.21.15.98:0/3626699987 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b801a070 con 0x7f58c01511d0 2024-05-27T00:34:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58bcc00640 1 -- 172.21.15.98:0/3626699987 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f58b8003ba0 con 0x7f58c01511d0 2024-05-27T00:34:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58bcc00640 1 -- 172.21.15.98:0/3626699987 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8004030 con 0x7f58c01511d0 2024-05-27T00:34:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58c7800640 1 -- 172.21.15.98:0/3626699987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c00b0110 con 0x7f58c01511d0 2024-05-27T00:34:09.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58c7800640 1 -- 172.21.15.98:0/3626699987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c00b06d0 con 0x7f58c01511d0 2024-05-27T00:34:09.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58c7800640 1 -- 172.21.15.98:0/3626699987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c0075cb0 con 0x7f58c01511d0 2024-05-27T00:34:09.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58bcc00640 1 -- 172.21.15.98:0/3626699987 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f58b801d070 con 0x7f58c01511d0 2024-05-27T00:34:09.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58bcc00640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f58a00678c0 0x7f58a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58bcc00640 1 -- 172.21.15.98:0/3626699987 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f58b80865e0 con 0x7f58c01511d0 2024-05-27T00:34:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.566+0000 7f58c4e00640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f58a00678c0 0x7f58a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.570+0000 7f58c4e00640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f58a00678c0 0x7f58a0069d80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f58b4000f70 tx=0x7f58b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:09.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.570+0000 7f58bcc00640 1 -- 172.21.15.98:0/3626699987 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f58b8010070 con 0x7f58c01511d0 2024-05-27T00:34:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.758+0000 7f58c7800640 1 -- 172.21.15.98:0/3626699987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58c00713c0 con 0x7f58c01511d0 2024-05-27T00:34:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.758+0000 7f58bcc00640 1 -- 172.21.15.98:0/3626699987 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6839 (secure 0 0 0) 0x7f58b8056b70 con 0x7f58c01511d0 2024-05-27T00:34:09.762 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:34:09.762 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6838/3117088639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6835","nonce":3117088639},{"type":"v1","addr":"172.21.15.204:6838","nonce":3117088639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/4095022947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6833","nonce":4095022947},{"type":"v1","addr":"172.21.15.98:6835","nonce":4095022947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6836/1815065053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1815065053},{"type":"v1","addr":"172.21.15.204:6836","nonce":1815065053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2745908246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6837","nonce":2745908246},{"type":"v1","addr":"172.21.15.204:6839","nonce":2745908246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5864,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6834/3675441597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3675441597},{"type":"v1","addr":"172.21.15.98:6834","nonce":3675441597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T00:15:24.956429+0000","modified":"2024-05-27T00:32:32.656788+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":23,"addr":"172.21.15.98:6837/2042067068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2042067068},{"type":"v1","addr":"172.21.15.98:6837","nonce":2042067068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6396,"qdb_cluster":[6396]},"id":1}]} 2024-05-27T00:34:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 73 2024-05-27T00:34:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 -- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f58a00678c0 msgr2=0x7f58a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f58a00678c0 0x7f58a0069d80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f58b4000f70 tx=0x7f58b4009040 comp rx=0 tx=0).stop 2024-05-27T00:34:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 -- 172.21.15.98:0/3626699987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c01511d0 msgr2=0x7f58c00c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c01511d0 0x7f58c00c6fc0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f58b8007f20 tx=0x7f58b800d560 comp rx=0 tx=0).stop 2024-05-27T00:34:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 -- 172.21.15.98:0/3626699987 shutdown_connections 2024-05-27T00:34:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58c01511d0 0x7f58c00c6fc0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f58a00678c0 0x7f58a0069d80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58c00cc8f0 0x7f58c00c6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 --2- 172.21.15.98:0/3626699987 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58c00c8670 0x7f58c0143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 -- 172.21.15.98:0/3626699987 >> 172.21.15.98:0/3626699987 conn(0x7f58c0067530 msgr2=0x7f58c014f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 -- 172.21.15.98:0/3626699987 shutdown_connections 2024-05-27T00:34:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.762+0000 7f58c7800640 1 -- 172.21.15.98:0/3626699987 wait complete. 2024-05-27T00:34:09.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T00:34:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.914+0000 7f5273400640 1 Processor -- start 2024-05-27T00:34:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.914+0000 7f5273400640 1 -- start start 2024-05-27T00:34:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f5273400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f526c0731f0 0x7f526c0c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f5273400640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f526c0735d0 0x7f526c151430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f5273400640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526c0c17d0 con 0x7f526c0c79f0 2024-05-27T00:34:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f5273400640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f526c071990 con 0x7f526c0735d0 2024-05-27T00:34:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f5273400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f526c070e10 con 0x7f526c0731f0 2024-05-27T00:34:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f526be00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f526c0735d0 0x7f526c151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f526be00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f526c0735d0 0x7f526c151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35318/0 (socket says 172.21.15.98:35318) 2024-05-27T00:34:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f526be00640 1 -- 172.21.15.98:0/3535552897 learned_addr learned my addr 172.21.15.98:0/3535552897 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f5270a00640 1 --2- 172.21.15.98:0/3535552897 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f526c0731f0 0x7f526c0c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f526be00640 1 -- 172.21.15.98:0/3535552897 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f526c0731f0 msgr2=0x7f526c0c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f526be00640 1 --2- 172.21.15.98:0/3535552897 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f526c0731f0 0x7f526c0c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f526be00640 1 -- 172.21.15.98:0/3535552897 >> v1:172.21.15.204:6789/0 conn(0x7f526c0c79f0 legacy=0x7f526c157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.918+0000 7f526be00640 1 -- 172.21.15.98:0/3535552897 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526c0710f0 con 0x7f526c0735d0 2024-05-27T00:34:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f526be00640 1 --2- 172.21.15.98:0/3535552897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 0x7f526c151430 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f5258009870 tx=0x7f525801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471fa146bead016e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f526b400640 1 -- 172.21.15.98:0/3535552897 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52580b5070 con 0x7f526c0735d0 2024-05-27T00:34:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f526b400640 1 -- 172.21.15.98:0/3535552897 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f52580a8a50 con 0x7f526c0735d0 2024-05-27T00:34:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f526b400640 1 -- 172.21.15.98:0/3535552897 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52580a8e40 con 0x7f526c0735d0 2024-05-27T00:34:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f5273400640 1 -- 172.21.15.98:0/3535552897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 msgr2=0x7f526c151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f5273400640 1 --2- 172.21.15.98:0/3535552897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 0x7f526c151430 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f5258009870 tx=0x7f525801ed50 comp rx=0 tx=0).stop 2024-05-27T00:34:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f5273400640 1 -- 172.21.15.98:0/3535552897 shutdown_connections 2024-05-27T00:34:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f5273400640 1 --2- 172.21.15.98:0/3535552897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 0x7f526c151430 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f5273400640 1 --2- 172.21.15.98:0/3535552897 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f526c0731f0 0x7f526c0c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f5273400640 1 -- 172.21.15.98:0/3535552897 >> 172.21.15.98:0/3535552897 conn(0x7f526c067530 msgr2=0x7f526c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f5273400640 1 -- 172.21.15.98:0/3535552897 shutdown_connections 2024-05-27T00:34:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f5273400640 1 -- 172.21.15.98:0/3535552897 wait complete. 2024-05-27T00:34:09.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f5273400640 1 Processor -- start 2024-05-27T00:34:09.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.922+0000 7f5273400640 1 -- start start 2024-05-27T00:34:09.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5273400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f526c0731f0 0x7f526c145f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5273400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 0x7f526c1f20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5273400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f526c0c79f0 0x7f526c1e3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5273400640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f526c0c8400 con 0x7f526c0735d0 2024-05-27T00:34:09.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5273400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f526c0716b0 con 0x7f526c0731f0 2024-05-27T00:34:09.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5273400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f526c0c13f0 con 0x7f526c0c79f0 2024-05-27T00:34:09.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f526be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 0x7f526c1f20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f526be00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 0x7f526c1f20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35328/0 (socket says 172.21.15.98:35328) 2024-05-27T00:34:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f526be00640 1 -- 172.21.15.98:0/3978958290 learned_addr learned my addr 172.21.15.98:0/3978958290 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5270a00640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f526c0731f0 0x7f526c145f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5271400640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f526c0c79f0 0x7f526c1e3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f526be00640 1 -- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f526c0c79f0 msgr2=0x7f526c1e3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f526be00640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f526c0c79f0 0x7f526c1e3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f526be00640 1 -- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f526c0731f0 msgr2=0x7f526c145f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:09.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f526be00640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f526c0731f0 0x7f526c145f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:09.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f526be00640 1 -- 172.21.15.98:0/3978958290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526c149e10 con 0x7f526c0735d0 2024-05-27T00:34:09.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f526be00640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 0x7f526c1f20f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f52580b3040 tx=0x7f525801baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:09.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5268c00640 1 -- 172.21.15.98:0/3978958290 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52580c0070 con 0x7f526c0735d0 2024-05-27T00:34:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5268c00640 1 -- 172.21.15.98:0/3978958290 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f52580ae770 con 0x7f526c0735d0 2024-05-27T00:34:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5273400640 1 -- 172.21.15.98:0/3978958290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526c1ee4c0 con 0x7f526c0735d0 2024-05-27T00:34:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5268c00640 1 -- 172.21.15.98:0/3978958290 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52580b5070 con 0x7f526c0735d0 2024-05-27T00:34:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5273400640 1 -- 172.21.15.98:0/3978958290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f526c1eea80 con 0x7f526c0735d0 2024-05-27T00:34:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5273400640 1 -- 172.21.15.98:0/3978958290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f526c075cb0 con 0x7f526c0735d0 2024-05-27T00:34:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.926+0000 7f5268c00640 1 -- 172.21.15.98:0/3978958290 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f52580c3070 con 0x7f526c0735d0 2024-05-27T00:34:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.930+0000 7f5268c00640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f524c0678c0 0x7f524c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.930+0000 7f5268c00640 1 -- 172.21.15.98:0/3978958290 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f5258123be0 con 0x7f526c0735d0 2024-05-27T00:34:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.930+0000 7f5270a00640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f524c0678c0 0x7f524c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.930+0000 7f5270a00640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f524c0678c0 0x7f524c069d80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f525c007330 tx=0x7f525c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:09.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:09.934+0000 7f5268c00640 1 -- 172.21.15.98:0/3978958290 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f52580188b0 con 0x7f526c0735d0 2024-05-27T00:34:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.090+0000 7f5273400640 1 -- 172.21.15.98:0/3978958290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f526c1467b0 con 0x7f526c0735d0 2024-05-27T00:34:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5268c00640 1 -- 172.21.15.98:0/3978958290 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+18411 (secure 0 0 0) 0x7f52580f4170 con 0x7f526c0735d0 2024-05-27T00:34:10.095 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-05-27T00:34:10.095 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"fsid":"f6d0c69d-6040-4984-af90-0cab9df1ba47","created":"2024-05-27T00:15:00.391007+0000","modified":"2024-05-27T00:32:10.313136+0000","last_up_change":"2024-05-27T00:15:11.404218+0000","last_in_change":"2024-05-27T00:15:02.792649+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T00:15:13.211703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T00:15:23.224464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T00:15:23.950896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b68fe95c-cc54-40c9-a1ac-363d1ed74dea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6818","nonce":4085423032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6822","nonce":4085423032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6827","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6829","nonce":4085423032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":4085423032},{"type":"v1","addr":"172.21.15.98:6826","nonce":4085423032}]},"public_addr":"172.21.15.98:6818/4085423032","cluster_addr":"172.21.15.98:6822/4085423032","heartbeat_back_addr":"172.21.15.98:6829/4085423032","heartbeat_front_addr":"172.21.15.98:6826/4085423032","state":["exists","up"]},{"osd":1,"uuid":"3b76f535-44e3-4dcd-90be-4dd6e2b326b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6809","nonce":1799281170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6811","nonce":1799281170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6815","nonce":1799281170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1799281170},{"type":"v1","addr":"172.21.15.98:6813","nonce":1799281170}]},"public_addr":"172.21.15.98:6809/1799281170","cluster_addr":"172.21.15.98:6811/1799281170","heartbeat_back_addr":"172.21.15.98:6815/1799281170","heartbeat_front_addr":"172.21.15.98:6813/1799281170","state":["exists","up"]},{"osd":2,"uuid":"bac80420-b2e9-4fe6-8a5a-6502ca48305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6801","nonce":2594065140}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6803","nonce":2594065140}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6807","nonce":2594065140}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2594065140},{"type":"v1","addr":"172.21.15.98:6805","nonce":2594065140}]},"public_addr":"172.21.15.98:6801/2594065140","cluster_addr":"172.21.15.98:6803/2594065140","heartbeat_back_addr":"172.21.15.98:6807/2594065140","heartbeat_front_addr":"172.21.15.98:6805/2594065140","state":["exists","up"]},{"osd":3,"uuid":"320acdd6-155a-4661-8ecc-ece9d4b785eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6819","nonce":1805101132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6823","nonce":1805101132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6831","nonce":1805101132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":1805101132},{"type":"v1","addr":"172.21.15.98:6828","nonce":1805101132}]},"public_addr":"172.21.15.98:6819/1805101132","cluster_addr":"172.21.15.98:6823/1805101132","heartbeat_back_addr":"172.21.15.98:6831/1805101132","heartbeat_front_addr":"172.21.15.98:6828/1805101132","state":["exists","up"]},{"osd":4,"uuid":"6ef9f4cb-d026-46ad-b3f9-ddebfe50fb76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6801","nonce":4073643582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6803","nonce":4073643582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6807","nonce":4073643582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":4073643582},{"type":"v1","addr":"172.21.15.204:6805","nonce":4073643582}]},"public_addr":"172.21.15.204:6801/4073643582","cluster_addr":"172.21.15.204:6803/4073643582","heartbeat_back_addr":"172.21.15.204:6807/4073643582","heartbeat_front_addr":"172.21.15.204:6805/4073643582","state":["exists","up"]},{"osd":5,"uuid":"b1b6c7ed-78bc-4f71-9522-78446864221c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6809","nonce":858542033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6811","nonce":858542033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6815","nonce":858542033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":858542033},{"type":"v1","addr":"172.21.15.204:6813","nonce":858542033}]},"public_addr":"172.21.15.204:6809/858542033","cluster_addr":"172.21.15.204:6811/858542033","heartbeat_back_addr":"172.21.15.204:6815/858542033","heartbeat_front_addr":"172.21.15.204:6813/858542033","state":["exists","up"]},{"osd":6,"uuid":"aeda16ff-23e1-4f54-8053-b375e68393b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6825","nonce":469091443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6827","nonce":469091443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6831","nonce":469091443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":469091443},{"type":"v1","addr":"172.21.15.204:6829","nonce":469091443}]},"public_addr":"172.21.15.204:6825/469091443","cluster_addr":"172.21.15.204:6827/469091443","heartbeat_back_addr":"172.21.15.204:6831/469091443","heartbeat_front_addr":"172.21.15.204:6829/469091443","state":["exists","up"]},{"osd":7,"uuid":"5fcb4aee-a2ad-47b3-a75c-d7a531602e85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6817","nonce":874852314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6819","nonce":874852314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6823","nonce":874852314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":874852314},{"type":"v1","addr":"172.21.15.204:6821","nonce":874852314}]},"public_addr":"172.21.15.204:6817/874852314","cluster_addr":"172.21.15.204:6819/874852314","heartbeat_back_addr":"172.21.15.204:6823/874852314","heartbeat_front_addr":"172.21.15.204:6821/874852314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:10.428822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.396875+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.353119+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.532531+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.143164+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.007588+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.219402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T00:15:08.106225+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6834/3070334683":"2024-05-28T00:32:10.313051+0000","172.21.15.98:6836/1802902856":"2024-05-28T00:30:40.295945+0000","172.21.15.98:6834/2148124202":"2024-05-28T00:28:50.279743+0000","172.21.15.98:6832/2148124202":"2024-05-28T00:28:50.279743+0000","172.21.15.98:6837/2940211164":"2024-05-28T00:27:55.271086+0000","172.21.15.98:6836/2940211164":"2024-05-28T00:27:55.271086+0000","172.21.15.98:6834/1223646158":"2024-05-28T00:27:00.259848+0000","172.21.15.98:6837/518550803":"2024-05-28T00:24:50.239991+0000","172.21.15.98:6836/3824013488":"2024-05-28T00:16:25.159349+0000","172.21.15.98:6832/3070334683":"2024-05-28T00:32:10.313051+0000","172.21.15.98:6832/3247908011":"2024-05-28T00:21:25.205263+0000","172.21.15.98:6834/4290077910":"2024-05-28T00:23:40.226716+0000","172.21.15.98:6832/4290077910":"2024-05-28T00:23:40.226716+0000","172.21.15.98:6837/1802902856":"2024-05-28T00:30:40.295945+0000","172.21.15.98:6832/3903574515":"2024-05-28T00:18:25.179026+0000","172.21.15.98:6837/3824013488":"2024-05-28T00:16:25.159349+0000","172.21.15.98:6836/1781395818":"2024-05-28T00:19:35.189766+0000","172.21.15.98:6834/3903574515":"2024-05-28T00:18:25.179026+0000","172.21.15.98:6837/1781395818":"2024-05-28T00:19:35.189766+0000","172.21.15.98:6834/3247908011":"2024-05-28T00:21:25.205263+0000","172.21.15.98:6832/1223646158":"2024-05-28T00:27:00.259848+0000","172.21.15.98:6836/1698253956":"2024-05-28T00:22:20.213891+0000","172.21.15.98:6836/518550803":"2024-05-28T00:24:50.239991+0000","172.21.15.98:6837/1698253956":"2024-05-28T00:22:20.213891+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T00:34:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 -- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f524c0678c0 msgr2=0x7f524c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f524c0678c0 0x7f524c069d80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f525c007330 tx=0x7f525c00b040 comp rx=0 tx=0).stop 2024-05-27T00:34:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 -- 172.21.15.98:0/3978958290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 msgr2=0x7f526c1f20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 0x7f526c1f20f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f52580b3040 tx=0x7f525801baf0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 -- 172.21.15.98:0/3978958290 shutdown_connections 2024-05-27T00:34:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f526c0c79f0 0x7f526c1e3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f526c0735d0 0x7f526c1f20f0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f524c0678c0 0x7f524c069d80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 --2- 172.21.15.98:0/3978958290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f526c0731f0 0x7f526c145f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 -- 172.21.15.98:0/3978958290 >> 172.21.15.98:0/3978958290 conn(0x7f526c067530 msgr2=0x7f526c0c9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 -- 172.21.15.98:0/3978958290 shutdown_connections 2024-05-27T00:34:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.094+0000 7f5273400640 1 -- 172.21.15.98:0/3978958290 wait complete. 2024-05-27T00:34:10.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-27T00:34:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.250+0000 7f7115800640 1 Processor -- start 2024-05-27T00:34:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.250+0000 7f7115800640 1 -- start start 2024-05-27T00:34:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f7115800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7110070c00 0x7f71100c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f7115800640 1 --2- >> v2:172.21.15.204:3300/0 conn(0x7f7110070fe0 0x7f7110154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f7115800640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f711015a950 con 0x7f71100c8f40 2024-05-27T00:34:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f7115800640 1 -- --> v2:172.21.15.204:3300/0 -- mon_getmap magic: 0 -- 0x7f711015e060 con 0x7f7110070fe0 2024-05-27T00:34:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f7115800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f711015e340 con 0x7f7110070c00 2024-05-27T00:34:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f710f400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f71100c8f40 0x7f711015ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57684/0 (socket says 172.21.15.98:57684) 2024-05-27T00:34:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f710f400640 1 -- 172.21.15.98:0/1538315324 learned_addr learned my addr 172.21.15.98:0/1538315324 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:34:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f710ea00640 1 --2- 172.21.15.98:0/1538315324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7110070c00 0x7f71100c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f7106000640 1 --2- 172.21.15.98:0/1538315324 >> v2:172.21.15.204:3300/0 conn(0x7f7110070fe0 0x7f7110154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f710e000640 1 -- 172.21.15.98:0/1538315324 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827361053 0 0) 0x7f711015a950 con 0x7f71100c8f40 2024-05-27T00:34:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f710e000640 1 -- 172.21.15.98:0/1538315324 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70f4003530 con 0x7f71100c8f40 2024-05-27T00:34:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f710ea00640 1 -- 172.21.15.98:0/1538315324 >> v2:172.21.15.204:3300/0 conn(0x7f7110070fe0 msgr2=0x7f7110154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f710ea00640 1 --2- 172.21.15.98:0/1538315324 >> v2:172.21.15.204:3300/0 conn(0x7f7110070fe0 0x7f7110154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f710ea00640 1 -- 172.21.15.98:0/1538315324 >> v1:172.21.15.98:6789/0 conn(0x7f71100c8f40 legacy=0x7f711015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f710e000640 1 -- 172.21.15.98:0/1538315324 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3635571544 0 0) 0x7f70f4003530 con 0x7f71100c8f40 2024-05-27T00:34:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f710ea00640 1 -- 172.21.15.98:0/1538315324 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711015e620 con 0x7f7110070c00 2024-05-27T00:34:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.254+0000 7f7106000640 1 --2- 172.21.15.98:0/1538315324 >> v2:172.21.15.204:3300/0 conn(0x7f7110070fe0 0x7f7110154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T00:34:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f710ea00640 1 --2- 172.21.15.98:0/1538315324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7110070c00 0x7f71100c8a00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f71000097c0 tx=0x7f710001ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eab67f733c02591 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f710e000640 1 -- 172.21.15.98:0/1538315324 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71000b5070 con 0x7f7110070c00 2024-05-27T00:34:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f710e000640 1 -- 172.21.15.98:0/1538315324 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f71000a85f0 con 0x7f7110070c00 2024-05-27T00:34:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f710e000640 1 -- 172.21.15.98:0/1538315324 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7108001a00 con 0x7f7110070c00 2024-05-27T00:34:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f7115800640 1 -- 172.21.15.98:0/1538315324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7110070c00 msgr2=0x7f71100c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f7115800640 1 --2- 172.21.15.98:0/1538315324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7110070c00 0x7f71100c8a00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f71000097c0 tx=0x7f710001ebb0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f7115800640 1 -- 172.21.15.98:0/1538315324 shutdown_connections 2024-05-27T00:34:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f7115800640 1 --2- 172.21.15.98:0/1538315324 >> v2:172.21.15.204:3300/0 conn(0x7f7110070fe0 0x7f7110154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f7115800640 1 --2- 172.21.15.98:0/1538315324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7110070c00 0x7f71100c8a00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f7115800640 1 -- 172.21.15.98:0/1538315324 >> 172.21.15.98:0/1538315324 conn(0x7f7110072f90 msgr2=0x7f71100733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f7115800640 1 -- 172.21.15.98:0/1538315324 shutdown_connections 2024-05-27T00:34:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f7115800640 1 -- 172.21.15.98:0/1538315324 wait complete. 2024-05-27T00:34:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f7115800640 1 Processor -- start 2024-05-27T00:34:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.258+0000 7f7115800640 1 -- start start 2024-05-27T00:34:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7115800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7110070c00 0x7f711006d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7115800640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7110070fe0 0x7f711014e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7115800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71100c8f40 0x7f71101f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7115800640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f711015b560 con 0x7f7110070fe0 2024-05-27T00:34:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7115800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f711015ebe0 con 0x7f7110070c00 2024-05-27T00:34:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7115800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f711015b6e0 con 0x7f71100c8f40 2024-05-27T00:34:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7106000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7110070fe0 0x7f711014e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7106000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7110070fe0 0x7f711014e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35342/0 (socket says 172.21.15.98:35342) 2024-05-27T00:34:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7106000640 1 -- 172.21.15.98:0/1940291804 learned_addr learned my addr 172.21.15.98:0/1940291804 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f710ea00640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7110070c00 0x7f711006d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f710f400640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71100c8f40 0x7f71101f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7106000640 1 -- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71100c8f40 msgr2=0x7f71101f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7106000640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71100c8f40 0x7f71101f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7106000640 1 -- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7110070c00 msgr2=0x7f711006d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7106000640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7110070c00 0x7f711006d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7106000640 1 -- 172.21.15.98:0/1940291804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7110151c10 con 0x7f7110070fe0 2024-05-27T00:34:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f710ea00640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7110070c00 0x7f711006d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:34:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7106000640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7110070fe0 0x7f711014e040 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f70f800ea00 tx=0x7f70f800eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7107400640 1 -- 172.21.15.98:0/1940291804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f8017070 con 0x7f7110070fe0 2024-05-27T00:34:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7107400640 1 -- 172.21.15.98:0/1940291804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f70f800ca60 con 0x7f7110070fe0 2024-05-27T00:34:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7115800640 1 -- 172.21.15.98:0/1940291804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711014efd0 con 0x7f7110070fe0 2024-05-27T00:34:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7107400640 1 -- 172.21.15.98:0/1940291804 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f800ce70 con 0x7f7110070fe0 2024-05-27T00:34:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7115800640 1 -- 172.21.15.98:0/1940291804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7110158e30 con 0x7f7110070fe0 2024-05-27T00:34:10.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7115800640 1 -- 172.21.15.98:0/1940291804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7110075cb0 con 0x7f7110070fe0 2024-05-27T00:34:10.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.262+0000 7f7107400640 1 -- 172.21.15.98:0/1940291804 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f70f801a070 con 0x7f7110070fe0 2024-05-27T00:34:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.266+0000 7f7107400640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70ec0678f0 0x7f70ec069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.266+0000 7f7107400640 1 -- 172.21.15.98:0/1940291804 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6857+0+0 (secure 0 0 0) 0x7f70f8085ec0 con 0x7f7110070fe0 2024-05-27T00:34:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.266+0000 7f710ea00640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70ec0678f0 0x7f70ec069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.266+0000 7f710ea00640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70ec0678f0 0x7f70ec069db0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f71000b2000 tx=0x7f71000aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.270+0000 7f7107400640 1 -- 172.21.15.98:0/1940291804 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f70f8056450 con 0x7f7110070fe0 2024-05-27T00:34:10.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.458+0000 7f7115800640 1 -- 172.21.15.98:0/1940291804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7110073830 con 0x7f7110070fe0 2024-05-27T00:34:10.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7107400640 1 -- 172.21.15.98:0/1940291804 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) ==== 161+0+0 (secure 0 0 0) 0x7f70f8055d50 con 0x7f7110070fe0 2024-05-27T00:34:10.462 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-27T00:34:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 -- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70ec0678f0 msgr2=0x7f70ec069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70ec0678f0 0x7f70ec069db0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f71000b2000 tx=0x7f71000aa070 comp rx=0 tx=0).stop 2024-05-27T00:34:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 -- 172.21.15.98:0/1940291804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7110070fe0 msgr2=0x7f711014e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7110070fe0 0x7f711014e040 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f70f800ea00 tx=0x7f70f800eec0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 -- 172.21.15.98:0/1940291804 shutdown_connections 2024-05-27T00:34:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71100c8f40 0x7f71101f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7110070fe0 0x7f711014e040 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f70ec0678f0 0x7f70ec069db0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 --2- 172.21.15.98:0/1940291804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7110070c00 0x7f711006d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 -- 172.21.15.98:0/1940291804 >> 172.21.15.98:0/1940291804 conn(0x7f7110072f90 msgr2=0x7f7110159620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 -- 172.21.15.98:0/1940291804 shutdown_connections 2024-05-27T00:34:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.462+0000 7f7115800640 1 -- 172.21.15.98:0/1940291804 wait complete. 2024-05-27T00:34:10.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-27T00:34:10.559 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2024-05-27T00:34:10.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.626+0000 7f37be000640 1 Processor -- start 2024-05-27T00:34:10.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37be000640 1 -- start start 2024-05-27T00:34:10.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37be000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80b1e10 0x7f37b806e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37be000640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37b80c2b10 0x7f37b806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37be000640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37b80c8420 con 0x7f37b81516b0 2024-05-27T00:34:10.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37be000640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f37b80c1df0 con 0x7f37b80c2b10 2024-05-27T00:34:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37be000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f37b80c3000 con 0x7f37b80b1e10 2024-05-27T00:34:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37b7e00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37b80c2b10 0x7f37b806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37b7400640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f37b81516b0 0x7f37b80c1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57698/0 (socket says 172.21.15.98:57698) 2024-05-27T00:34:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37b7400640 1 -- 172.21.15.98:0/3950691652 learned_addr learned my addr 172.21.15.98:0/3950691652 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:34:10.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37aea00640 1 --2- 172.21.15.98:0/3950691652 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80b1e10 0x7f37b806e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37b7e00640 1 -- 172.21.15.98:0/3950691652 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80b1e10 msgr2=0x7f37b806e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37b7e00640 1 --2- 172.21.15.98:0/3950691652 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80b1e10 0x7f37b806e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37b7e00640 1 -- 172.21.15.98:0/3950691652 >> v1:172.21.15.98:6789/0 conn(0x7f37b81516b0 legacy=0x7f37b80c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.630+0000 7f37b7e00640 1 -- 172.21.15.98:0/3950691652 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b80c7e60 con 0x7f37b80c2b10 2024-05-27T00:34:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37b7e00640 1 --2- 172.21.15.98:0/3950691652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b80c2b10 0x7f37b806ecc0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f37b00097f0 tx=0x7f37b001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c519ceba2e424a6e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:10.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37b6a00640 1 -- 172.21.15.98:0/3950691652 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b00b2070 con 0x7f37b80c2b10 2024-05-27T00:34:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37b6a00640 1 -- 172.21.15.98:0/3950691652 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f37b00ad640 con 0x7f37b80c2b10 2024-05-27T00:34:10.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37b6a00640 1 -- 172.21.15.98:0/3950691652 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b00ada30 con 0x7f37b80c2b10 2024-05-27T00:34:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37be000640 1 -- 172.21.15.98:0/3950691652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b80c2b10 msgr2=0x7f37b806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37be000640 1 --2- 172.21.15.98:0/3950691652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b80c2b10 0x7f37b806ecc0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f37b00097f0 tx=0x7f37b001c4a0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37be000640 1 -- 172.21.15.98:0/3950691652 shutdown_connections 2024-05-27T00:34:10.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37be000640 1 --2- 172.21.15.98:0/3950691652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b80c2b10 0x7f37b806ecc0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37be000640 1 --2- 172.21.15.98:0/3950691652 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80b1e10 0x7f37b806e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37be000640 1 -- 172.21.15.98:0/3950691652 >> 172.21.15.98:0/3950691652 conn(0x7f37b80cbf00 msgr2=0x7f37b80cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37be000640 1 -- 172.21.15.98:0/3950691652 shutdown_connections 2024-05-27T00:34:10.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.634+0000 7f37be000640 1 -- 172.21.15.98:0/3950691652 wait complete. 2024-05-27T00:34:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37be000640 1 Processor -- start 2024-05-27T00:34:10.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37be000640 1 -- start start 2024-05-27T00:34:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37be000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37b80b1e10 0x7f37b81f7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37be000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80c2b10 0x7f37b80cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37be000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b81516b0 0x7f37b81496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37be000640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37b8151c10 con 0x7f37b81516b0 2024-05-27T00:34:10.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37be000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f37b8151a90 con 0x7f37b80b1e10 2024-05-27T00:34:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37be000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f37b8164540 con 0x7f37b80c2b10 2024-05-27T00:34:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b81516b0 0x7f37b81496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7e00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b81516b0 0x7f37b81496a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35358/0 (socket says 172.21.15.98:35358) 2024-05-27T00:34:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7e00640 1 -- 172.21.15.98:0/721279891 learned_addr learned my addr 172.21.15.98:0/721279891 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37aea00640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80c2b10 0x7f37b80cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7400640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37b80b1e10 0x7f37b81f7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7e00640 1 -- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80c2b10 msgr2=0x7f37b80cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7e00640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80c2b10 0x7f37b80cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7e00640 1 -- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37b80b1e10 msgr2=0x7f37b81f7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:10.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7e00640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37b80b1e10 0x7f37b81f7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:10.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7e00640 1 -- 172.21.15.98:0/721279891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b80af670 con 0x7f37b81516b0 2024-05-27T00:34:10.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37aea00640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80c2b10 0x7f37b80cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:34:10.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7400640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37b80b1e10 0x7f37b81f7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T00:34:10.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37b7e00640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b81516b0 0x7f37b81496a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f37b00b0040 tx=0x7f37b0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:10.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37afe00640 1 -- 172.21.15.98:0/721279891 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b00bc070 con 0x7f37b81516b0 2024-05-27T00:34:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37afe00640 1 -- 172.21.15.98:0/721279891 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f37b00b6610 con 0x7f37b81516b0 2024-05-27T00:34:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37be000640 1 -- 172.21.15.98:0/721279891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37b80af950 con 0x7f37b81516b0 2024-05-27T00:34:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37afe00640 1 -- 172.21.15.98:0/721279891 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b00b2040 con 0x7f37b81516b0 2024-05-27T00:34:10.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.638+0000 7f37be000640 1 -- 172.21.15.98:0/721279891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37b80aff10 con 0x7f37b81516b0 2024-05-27T00:34:10.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.642+0000 7f37be000640 1 -- 172.21.15.98:0/721279891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37b8075cb0 con 0x7f37b81516b0 2024-05-27T00:34:10.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.642+0000 7f37afe00640 1 -- 172.21.15.98:0/721279891 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f37b00bf070 con 0x7f37b81516b0 2024-05-27T00:34:10.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.642+0000 7f37afe00640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f37940678c0 0x7f3794069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:10.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.642+0000 7f37afe00640 1 -- 172.21.15.98:0/721279891 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6943+0+0 (secure 0 0 0) 0x7f37b011f830 con 0x7f37b81516b0 2024-05-27T00:34:10.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.642+0000 7f37b7400640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f37940678c0 0x7f3794069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:10.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.642+0000 7f37b7400640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f37940678c0 0x7f3794069d80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f37a80012f0 tx=0x7f37a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:10.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.646+0000 7f37afe00640 1 -- 172.21.15.98:0/721279891 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f37b0016090 con 0x7f37b81516b0 2024-05-27T00:34:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:10.834+0000 7f37be000640 1 -- 172.21.15.98:0/721279891 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f37b806ebd0 con 0x7f37b81516b0 2024-05-27T00:34:11.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.462+0000 7f37afe00640 1 -- 172.21.15.98:0/721279891 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) ==== 104+0+0 (secure 0 0 0) 0x7f37b00efd70 con 0x7f37b81516b0 2024-05-27T00:34:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 -- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f37940678c0 msgr2=0x7f3794069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f37940678c0 0x7f3794069d80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f37a80012f0 tx=0x7f37a8009040 comp rx=0 tx=0).stop 2024-05-27T00:34:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 -- 172.21.15.98:0/721279891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b81516b0 msgr2=0x7f37b81496a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b81516b0 0x7f37b81496a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f37b00b0040 tx=0x7f37b0019240 comp rx=0 tx=0).stop 2024-05-27T00:34:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 -- 172.21.15.98:0/721279891 shutdown_connections 2024-05-27T00:34:11.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37b81516b0 0x7f37b81496a0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f37b80c2b10 0x7f37b80cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f37940678c0 0x7f3794069d80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 --2- 172.21.15.98:0/721279891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37b80b1e10 0x7f37b81f7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 -- 172.21.15.98:0/721279891 >> 172.21.15.98:0/721279891 conn(0x7f37b80cbf00 msgr2=0x7f37b8147e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 -- 172.21.15.98:0/721279891 shutdown_connections 2024-05-27T00:34:11.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.466+0000 7f37be000640 1 -- 172.21.15.98:0/721279891 wait complete. 2024-05-27T00:34:11.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-27T00:34:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.626+0000 7f014fe00640 1 Processor -- start 2024-05-27T00:34:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.626+0000 7f014fe00640 1 -- start start 2024-05-27T00:34:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148158640 0x7f0148158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014fe00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0148159010 con 0x7f01480b1c20 2024-05-27T00:34:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014fe00640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f014815bc40 con 0x7f0148152990 2024-05-27T00:34:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f014815c850 con 0x7f0148158640 2024-05-27T00:34:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014ca00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f01480b1c20 0x7f01480c7f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57706/0 (socket says 172.21.15.98:57706) 2024-05-27T00:34:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014ca00640 1 -- 172.21.15.98:0/2706211085 learned_addr learned my addr 172.21.15.98:0/2706211085 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-05-27T00:34:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014de00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148158640 0x7f0148158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832582230 0 0) 0x7f0148159010 con 0x7f01480b1c20 2024-05-27T00:34:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0130003530 con 0x7f01480b1c20 2024-05-27T00:34:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1407995934 0 0) 0x7f0130003530 con 0x7f01480b1c20 2024-05-27T00:34:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148158640 msgr2=0x7f0148158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148158640 0x7f0148158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 >> v1:172.21.15.204:6789/0 conn(0x7f0148152990 legacy=0x7f014806e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:11.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014815bf20 con 0x7f01480b1c20 2024-05-27T00:34:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 722810630 0 0) 0x7f0134001680 con 0x7f01480b1c20 2024-05-27T00:34:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 >> v1:172.21.15.98:6789/0 conn(0x7f01480b1c20 legacy=0x7f01480c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01300039f0 0x7f0130003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 0x7f0130004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148158640 0x7f01300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f014815c850 con 0x7f0130004810 2024-05-27T00:34:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f013000bab0 con 0x7f01300039f0 2024-05-27T00:34:11.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f013000bcb0 con 0x7f0148158640 2024-05-27T00:34:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014ca00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 0x7f0130004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014d400640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01300039f0 0x7f0130003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014de00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148158640 0x7f01300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014ca00640 1 -- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148158640 msgr2=0x7f01300051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014ca00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148158640 0x7f01300051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014ca00640 1 -- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01300039f0 msgr2=0x7f0130003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014ca00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01300039f0 0x7f0130003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:11.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.630+0000 7f014ca00640 1 -- 172.21.15.98:0/2706211085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0130003530 con 0x7f0130004810 2024-05-27T00:34:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f014ca00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 0x7f0130004c90 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f013400c210 tx=0x7f013401f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f95aafbb7c2c2e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134007d20 con 0x7f0130004810 2024-05-27T00:34:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0134005ae0 con 0x7f0130004810 2024-05-27T00:34:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f0147e00640 1 -- 172.21.15.98:0/2706211085 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134007070 con 0x7f0130004810 2024-05-27T00:34:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f014fe00640 1 -- 172.21.15.98:0/2706211085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 msgr2=0x7f0130004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f014fe00640 1 --2- 172.21.15.98:0/2706211085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 0x7f0130004c90 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f013400c210 tx=0x7f013401f390 comp rx=0 tx=0).stop 2024-05-27T00:34:11.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f014de00640 1 -- 172.21.15.98:0/2706211085 reap_dead start 2024-05-27T00:34:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f014fe00640 1 -- 172.21.15.98:0/2706211085 shutdown_connections 2024-05-27T00:34:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f014fe00640 1 -- 172.21.15.98:0/2706211085 >> 172.21.15.98:0/2706211085 conn(0x7f01480cbf00 msgr2=0x7f01480cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f014fe00640 1 -- 172.21.15.98:0/2706211085 shutdown_connections 2024-05-27T00:34:11.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.634+0000 7f014fe00640 1 -- 172.21.15.98:0/2706211085 wait complete. 2024-05-27T00:34:11.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 Processor -- start 2024-05-27T00:34:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 -- start start 2024-05-27T00:34:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 0x7f014815ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01480c28a0 0x7f0148148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148148e70 0x7f01481f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f014806ee90 con 0x7f0130004810 2024-05-27T00:34:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f014815c4e0 con 0x7f01480c28a0 2024-05-27T00:34:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f014806f010 con 0x7f0148148e70 2024-05-27T00:34:11.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 0x7f014815ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014d400640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 0x7f014815ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35382/0 (socket says 172.21.15.98:35382) 2024-05-27T00:34:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014d400640 1 -- 172.21.15.98:0/2642953100 learned_addr learned my addr 172.21.15.98:0/2642953100 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014ca00640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01480c28a0 0x7f0148148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014de00640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148148e70 0x7f01481f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014d400640 1 -- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148148e70 msgr2=0x7f01481f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014d400640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148148e70 0x7f01481f3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014d400640 1 -- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01480c28a0 msgr2=0x7f0148148930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:11.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014d400640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01480c28a0 0x7f0148148930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014d400640 1 -- 172.21.15.98:0/2642953100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014815b6b0 con 0x7f0130004810 2024-05-27T00:34:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014d400640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 0x7f014815ad10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f013c00f9b0 tx=0x7f013c00fe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f0145600640 1 -- 172.21.15.98:0/2642953100 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c003b90 con 0x7f0130004810 2024-05-27T00:34:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f0145600640 1 -- 172.21.15.98:0/2642953100 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f013c018070 con 0x7f0130004810 2024-05-27T00:34:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 -- 172.21.15.98:0/2642953100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014815a090 con 0x7f0130004810 2024-05-27T00:34:11.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f0145600640 1 -- 172.21.15.98:0/2642953100 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c00d7f0 con 0x7f0130004810 2024-05-27T00:34:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 -- 172.21.15.98:0/2642953100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f014815a650 con 0x7f0130004810 2024-05-27T00:34:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.638+0000 7f014fe00640 1 -- 172.21.15.98:0/2642953100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0148149250 con 0x7f0130004810 2024-05-27T00:34:11.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.642+0000 7f0145600640 1 -- 172.21.15.98:0/2642953100 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f013c00d990 con 0x7f0130004810 2024-05-27T00:34:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.642+0000 7f0145600640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f01280678f0 0x7f0128069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.642+0000 7f0145600640 1 -- 172.21.15.98:0/2642953100 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6943+0+0 (secure 0 0 0) 0x7f013c086340 con 0x7f0130004810 2024-05-27T00:34:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.642+0000 7f014ca00640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f01280678f0 0x7f0128069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:11.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.642+0000 7f014ca00640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f01280678f0 0x7f0128069db0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f01340af000 tx=0x7f01340aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:11.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.646+0000 7f0145600640 1 -- 172.21.15.98:0/2642953100 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f013c01a310 con 0x7f0130004810 2024-05-27T00:34:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:11.802+0000 7f014fe00640 1 -- 172.21.15.98:0/2642953100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0148075cb0 con 0x7f0130004810 2024-05-27T00:34:12.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.470+0000 7f0145600640 1 -- 172.21.15.98:0/2642953100 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v50) ==== 181+0+0 (secure 0 0 0) 0x7f013c056910 con 0x7f0130004810 2024-05-27T00:34:12.472 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2024-05-27T00:34:12.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 -- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f01280678f0 msgr2=0x7f0128069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:12.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f01280678f0 0x7f0128069db0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f01340af000 tx=0x7f01340aa000 comp rx=0 tx=0).stop 2024-05-27T00:34:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 -- 172.21.15.98:0/2642953100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 msgr2=0x7f014815ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 0x7f014815ad10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f013c00f9b0 tx=0x7f013c00fe70 comp rx=0 tx=0).stop 2024-05-27T00:34:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 -- 172.21.15.98:0/2642953100 shutdown_connections 2024-05-27T00:34:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0148148e70 0x7f01481f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f01280678f0 0x7f0128069db0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01480c28a0 0x7f0148148930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 --2- 172.21.15.98:0/2642953100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0130004810 0x7f014815ad10 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:12.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 -- 172.21.15.98:0/2642953100 >> 172.21.15.98:0/2642953100 conn(0x7f01480cbf00 msgr2=0x7f01480c6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 -- 172.21.15.98:0/2642953100 shutdown_connections 2024-05-27T00:34:12.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.474+0000 7f014fe00640 1 -- 172.21.15.98:0/2642953100 wait complete. 2024-05-27T00:34:12.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-27T00:34:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.682+0000 7f13dd600640 1 Processor -- start 2024-05-27T00:34:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.682+0000 7f13dd600640 1 -- start start 2024-05-27T00:34:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.686+0000 7f13dd600640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13d8154870 0x7f13d80c8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.686+0000 7f13dd600640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d815a780 con 0x7f13d815b500 2024-05-27T00:34:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.686+0000 7f13dd600640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d815e870 con 0x7f13d8154c50 2024-05-27T00:34:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.686+0000 7f13dd600640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f13d815eb50 con 0x7f13d8154870 2024-05-27T00:34:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.686+0000 7f13d6a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13d8154870 0x7f13d80c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.686+0000 7f13d6a00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13d8154870 0x7f13d80c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35384/0 (socket says 172.21.15.98:35384) 2024-05-27T00:34:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.686+0000 7f13d6a00640 1 -- 172.21.15.98:0/2980767719 learned_addr learned my addr 172.21.15.98:0/2980767719 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.686+0000 7f13d6a00640 1 -- 172.21.15.98:0/2980767719 >> v1:172.21.15.204:6789/0 conn(0x7f13d8154c50 legacy=0x7f13d80c1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.686+0000 7f13d6a00640 1 -- 172.21.15.98:0/2980767719 >> v1:172.21.15.98:6789/0 conn(0x7f13d815b500 legacy=0x7f13d815dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T00:34:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.686+0000 7f13d6a00640 1 -- 172.21.15.98:0/2980767719 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d815aa40 con 0x7f13d8154870 2024-05-27T00:34:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.690+0000 7f13d6a00640 1 --2- 172.21.15.98:0/2980767719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154870 0x7f13d80c8360 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009870 tx=0x7f13c001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d818a6d115f1e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.690+0000 7f13d5600640 1 -- 172.21.15.98:0/2980767719 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c00ab660 con 0x7f13d8154870 2024-05-27T00:34:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.694+0000 7f13d5600640 1 -- 172.21.15.98:0/2980767719 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f13c00b0070 con 0x7f13d8154870 2024-05-27T00:34:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.694+0000 7f13d5600640 1 -- 172.21.15.98:0/2980767719 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c00aba40 con 0x7f13d8154870 2024-05-27T00:34:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.694+0000 7f13dd600640 1 -- 172.21.15.98:0/2980767719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154870 msgr2=0x7f13d80c8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.694+0000 7f13dd600640 1 --2- 172.21.15.98:0/2980767719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154870 0x7f13d80c8360 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009870 tx=0x7f13c001c460 comp rx=0 tx=0).stop 2024-05-27T00:34:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.694+0000 7f13dd600640 1 -- 172.21.15.98:0/2980767719 shutdown_connections 2024-05-27T00:34:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.694+0000 7f13dd600640 1 --2- 172.21.15.98:0/2980767719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154870 0x7f13d80c8360 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.694+0000 7f13dd600640 1 -- 172.21.15.98:0/2980767719 >> 172.21.15.98:0/2980767719 conn(0x7f13d80cbf00 msgr2=0x7f13d80cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.694+0000 7f13dd600640 1 -- 172.21.15.98:0/2980767719 shutdown_connections 2024-05-27T00:34:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.698+0000 7f13dd600640 1 -- 172.21.15.98:0/2980767719 wait complete. 2024-05-27T00:34:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.698+0000 7f13dd600640 1 Processor -- start 2024-05-27T00:34:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13dd600640 1 -- start start 2024-05-27T00:34:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13dd600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8154870 0x7f13d81475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13dd600640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154c50 0x7f13d81f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13dd600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13d815b500 0x7f13d815d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13dd600640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13d81682a0 con 0x7f13d8154c50 2024-05-27T00:34:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13dd600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8168120 con 0x7f13d8154870 2024-05-27T00:34:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13dd600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f13d8168420 con 0x7f13d815b500 2024-05-27T00:34:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13d6000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154c50 0x7f13d81f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13d6000640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154c50 0x7f13d81f3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35400/0 (socket says 172.21.15.98:35400) 2024-05-27T00:34:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13d6000640 1 -- 172.21.15.98:0/3610194842 learned_addr learned my addr 172.21.15.98:0/3610194842 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-05-27T00:34:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13d6000640 1 -- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13d815b500 msgr2=0x7f13d815d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13d6000640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13d815b500 0x7f13d815d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13d6000640 1 -- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8154870 msgr2=0x7f13d81475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13d6000640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8154870 0x7f13d81475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.702+0000 7f13d6000640 1 -- 172.21.15.98:0/3610194842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d81f6c30 con 0x7f13d8154c50 2024-05-27T00:34:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.706+0000 7f13d6000640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154c50 0x7f13d81f3620 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f13c800c420 tx=0x7f13c800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.706+0000 7f13cea00640 1 -- 172.21.15.98:0/3610194842 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c80050d0 con 0x7f13d8154c50 2024-05-27T00:34:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.706+0000 7f13cea00640 1 -- 172.21.15.98:0/3610194842 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f13c8005270 con 0x7f13d8154c50 2024-05-27T00:34:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.706+0000 7f13cea00640 1 -- 172.21.15.98:0/3610194842 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c80050d0 con 0x7f13d8154c50 2024-05-27T00:34:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.706+0000 7f13dd600640 1 -- 172.21.15.98:0/3610194842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d81f6f10 con 0x7f13d8154c50 2024-05-27T00:34:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.710+0000 7f13cea00640 1 -- 172.21.15.98:0/3610194842 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f13c8007020 con 0x7f13d8154c50 2024-05-27T00:34:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.710+0000 7f13dd600640 1 -- 172.21.15.98:0/3610194842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d8147ae0 con 0x7f13d8154c50 2024-05-27T00:34:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.710+0000 7f13cea00640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13b4067ac0 0x7f13b4069f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T00:34:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.710+0000 7f13cea00640 1 -- 172.21.15.98:0/3610194842 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6505+0+0 (secure 0 0 0) 0x7f13c808cab0 con 0x7f13d8154c50 2024-05-27T00:34:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.710+0000 7f13d6a00640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13b4067ac0 0x7f13b4069f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T00:34:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.710+0000 7f13dd600640 1 -- 172.21.15.98:0/3610194842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d8075cb0 con 0x7f13d8154c50 2024-05-27T00:34:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.714+0000 7f13d6a00640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13b4067ac0 0x7f13b4069f80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f13c00af000 tx=0x7f13c0004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T00:34:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.718+0000 7f13cea00640 1 -- 172.21.15.98:0/3610194842 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177847 (secure 0 0 0) 0x7f13c8090070 con 0x7f13d8154c50 2024-05-27T00:34:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:12.886+0000 7f13dd600640 1 -- 172.21.15.98:0/3610194842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f13d814e530 con 0x7f13d8154c50 2024-05-27T00:34:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.478+0000 7f13cea00640 1 -- 172.21.15.98:0/3610194842 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v51) ==== 169+0+0 (secure 0 0 0) 0x7f13c805d2e0 con 0x7f13d8154c50 2024-05-27T00:34:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2024-05-27T00:34:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.482+0000 7f13dd600640 1 -- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13b4067ac0 msgr2=0x7f13b4069f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.482+0000 7f13dd600640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13b4067ac0 0x7f13b4069f80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f13c00af000 tx=0x7f13c0004740 comp rx=0 tx=0).stop 2024-05-27T00:34:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.482+0000 7f13dd600640 1 -- 172.21.15.98:0/3610194842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154c50 msgr2=0x7f13d81f3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T00:34:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.482+0000 7f13dd600640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154c50 0x7f13d81f3620 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f13c800c420 tx=0x7f13c800c8e0 comp rx=0 tx=0).stop 2024-05-27T00:34:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.486+0000 7f13dd600640 1 -- 172.21.15.98:0/3610194842 shutdown_connections 2024-05-27T00:34:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.486+0000 7f13dd600640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13d815b500 0x7f13d815d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.486+0000 7f13dd600640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13d8154c50 0x7f13d81f3620 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.486+0000 7f13dd600640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:6832/3774298058,v1:172.21.15.204:6833/3774298058] conn(0x7f13b4067ac0 0x7f13b4069f80 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.486+0000 7f13dd600640 1 --2- 172.21.15.98:0/3610194842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8154870 0x7f13d81475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T00:34:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.486+0000 7f13dd600640 1 -- 172.21.15.98:0/3610194842 >> 172.21.15.98:0/3610194842 conn(0x7f13d80cbf00 msgr2=0x7f13d80c1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T00:34:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.486+0000 7f13dd600640 1 -- 172.21.15.98:0/3610194842 shutdown_connections 2024-05-27T00:34:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-05-27T00:34:13.486+0000 7f13dd600640 1 -- 172.21.15.98:0/3610194842 wait complete. 2024-05-27T00:34:13.525 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-27T00:34:13.526 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-27T00:34:13.526 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:13.555 INFO:tasks.ceph.mds.a:Stopped 2024-05-27T00:34:13.555 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-27T00:34:13.555 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:13.626 INFO:tasks.ceph.mds.c:Stopped 2024-05-27T00:34:13.626 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-05-27T00:34:13.626 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:13.696 INFO:tasks.ceph.mds.e:Stopped 2024-05-27T00:34:13.697 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-27T00:34:13.697 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:13.714 INFO:tasks.ceph.mds.b:Stopped 2024-05-27T00:34:13.715 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-27T00:34:13.715 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:13.769 INFO:tasks.ceph.mds.d:Stopped 2024-05-27T00:34:13.770 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-05-27T00:34:13.770 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:13.827 INFO:tasks.ceph.mds.f:Stopped 2024-05-27T00:34:13.827 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-27T00:34:13.827 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-27T00:34:13.828 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:19.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-05-27T00:34:19.929 INFO:tasks.ceph.osd.0:Stopped 2024-05-27T00:34:19.930 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-27T00:34:19.930 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:20.016 INFO:tasks.ceph.osd.1:Stopped 2024-05-27T00:34:20.017 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-27T00:34:20.017 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:25.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-05-27T00:34:25.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-05-27T00:34:25.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:34:25.614 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:34:26.119 INFO:tasks.ceph.osd.2:Stopped 2024-05-27T00:34:26.120 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-27T00:34:26.120 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:31.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-05-27T00:34:31.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-05-27T00:34:32.222 INFO:tasks.ceph.osd.3:Stopped 2024-05-27T00:34:32.223 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-27T00:34:32.223 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:32.305 INFO:tasks.ceph.osd.4:Stopped 2024-05-27T00:34:32.306 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-27T00:34:32.306 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:36.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-05-27T00:34:36.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-05-27T00:34:38.408 INFO:tasks.ceph.osd.5:Stopped 2024-05-27T00:34:38.408 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-27T00:34:38.408 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:42.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-05-27T00:34:42.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-05-27T00:34:44.510 INFO:tasks.ceph.osd.6:Stopped 2024-05-27T00:34:44.510 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-27T00:34:44.511 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:48.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-05-27T00:34:48.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-05-27T00:34:50.612 INFO:tasks.ceph.osd.7:Stopped 2024-05-27T00:34:50.613 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-27T00:34:50.613 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-27T00:34:50.614 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:50.670 INFO:tasks.ceph.mgr.y:Stopped 2024-05-27T00:34:50.671 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-27T00:34:50.671 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:50.700 INFO:tasks.ceph.mgr.x:Stopped 2024-05-27T00:34:50.701 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-27T00:34:50.701 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-27T00:34:50.701 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:50.722 INFO:tasks.ceph.mon.a:Stopped 2024-05-27T00:34:50.722 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-27T00:34:50.723 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:50.760 INFO:tasks.ceph.mon.b:Stopped 2024-05-27T00:34:50.761 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-27T00:34:50.761 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T00:34:50.819 INFO:tasks.ceph.mon.c:Stopped 2024-05-27T00:34:50.819 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-27T00:34:50.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'Replacing daemon mds' | head -n 1 2024-05-27T00:34:50.863 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:34:50.864 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-27T00:34:51.122 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:34:51.123 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-27T00:34:51.399 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:34:51.399 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-27T00:34:51.630 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:34:51.630 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-27T00:34:51.821 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:34:51.821 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-27T00:34:52.037 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:34:52.037 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-27T00:34:52.257 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:34:52.258 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-27T00:34:52.422 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:34:52.423 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-27T00:34:52.608 INFO:tasks.ceph:Archiving mon data... 2024-05-27T00:34:52.610 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174/data/mon.a.tgz 2024-05-27T00:34:52.610 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-05-27T00:34:52.619 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.pkvz8BW9n9 2024-05-27T00:34:52.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.pkvz8BW9n9 2024-05-27T00:34:52.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.pkvz8BW9n9 2024-05-27T00:34:52.821 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.pkvz8BW9n9 is 374KB 2024-05-27T00:34:52.891 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.pkvz8BW9n9 2024-05-27T00:34:52.899 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174/data/mon.b.tgz 2024-05-27T00:34:52.900 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-05-27T00:34:52.906 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.BEmSOnM3wU 2024-05-27T00:34:52.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.BEmSOnM3wU 2024-05-27T00:34:53.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.BEmSOnM3wU 2024-05-27T00:34:53.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-05-27T00:34:53.103 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.BEmSOnM3wU is 379KB 2024-05-27T00:34:53.162 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.BEmSOnM3wU 2024-05-27T00:34:53.167 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174/data/mon.c.tgz 2024-05-27T00:34:53.168 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-05-27T00:34:53.216 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.uesAEN5FzT 2024-05-27T00:34:53.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.uesAEN5FzT 2024-05-27T00:34:53.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.uesAEN5FzT 2024-05-27T00:34:53.409 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.uesAEN5FzT is 379KB 2024-05-27T00:34:53.470 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.uesAEN5FzT 2024-05-27T00:34:53.476 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-27T00:34:53.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-27T00:34:53.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-27T00:34:53.549 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-27T00:34:53.580 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-27T00:34:53.580 INFO:tasks.ceph:Archiving crash dumps... 2024-05-27T00:34:53.582 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174/remote/smithi098/crash 2024-05-27T00:34:53.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-27T00:34:53.611 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174/remote/smithi204/crash 2024-05-27T00:34:53.612 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-27T00:34:53.635 INFO:tasks.ceph:Shutting down logrotate 2024-05-27T00:34:53.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:34:53.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T00:34:53.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-27T00:34:53.735 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-27T00:34:53.772 INFO:tasks.ceph:Compressing logs... 2024-05-27T00:34:53.772 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T00:34:53.780 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T00:34:53.795 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44209.log 2024-05-27T00:34:53.796 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35240.log 2024-05-27T00:34:53.796 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35939.log 2024-05-27T00:34:53.797 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44209.log.gz 2024-05-27T00:34:53.797 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42901.log 2024-05-27T00:34:53.798 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35240.log.gz 2024-05-27T00:34:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42875.log 2024-05-27T00:34:53.800 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35939.log.gz 2024-05-27T00:34:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44026.log 2024-05-27T00:34:53.800 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42901.log.gz 2024-05-27T00:34:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42063.log 2024-05-27T00:34:53.801 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42875.log.gz 2024-05-27T00:34:53.801 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33900.log 2024-05-27T00:34:53.802 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44026.log.gz 2024-05-27T00:34:53.802 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41878.log 2024-05-27T00:34:53.802 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42063.log.gz 2024-05-27T00:34:53.802 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43901.log 2024-05-27T00:34:53.802 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33900.log.gz 2024-05-27T00:34:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43199.log 2024-05-27T00:34:53.803 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41878.log.gz 2024-05-27T00:34:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44376.log 2024-05-27T00:34:53.804 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43901.log.gz 2024-05-27T00:34:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34148.log 2024-05-27T00:34:53.805 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43199.log.gz 2024-05-27T00:34:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44376.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-27T00:34:53.805 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44376.log.gz 2024-05-27T00:34:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34139.log 2024-05-27T00:34:53.806 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34148.log.gz 2024-05-27T00:34:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42927.log 2024-05-27T00:34:53.807 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34788.log 2024-05-27T00:34:53.807 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34139.log.gz 2024-05-27T00:34:53.808 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40112.log 2024-05-27T00:34:53.808 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42927.log.gz 2024-05-27T00:34:53.808 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39125.log 2024-05-27T00:34:53.810 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34788.log.gz 2024-05-27T00:34:53.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41319.log 2024-05-27T00:34:53.810 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40112.log.gz 2024-05-27T00:34:53.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41823.log 2024-05-27T00:34:53.811 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39125.log.gz 2024-05-27T00:34:53.811 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35188.log 2024-05-27T00:34:53.811 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41319.log.gz 2024-05-27T00:34:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34346.log 2024-05-27T00:34:53.813 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41823.log.gz 2024-05-27T00:34:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41161.log 2024-05-27T00:34:53.813 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35188.log.gz 2024-05-27T00:34:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40991.log 2024-05-27T00:34:53.814 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34346.log.gz 2024-05-27T00:34:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43383.log 2024-05-27T00:34:53.815 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41161.log.gz 2024-05-27T00:34:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45978.log 2024-05-27T00:34:53.816 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40991.log.gz 2024-05-27T00:34:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38555.log 2024-05-27T00:34:53.816 INFO:teuthology.orchestra.run.smithi098.stderr: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-27T00:34:53.816 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43383.log.gz 2024-05-27T00:34:53.818 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35422.log 2024-05-27T00:34:53.818 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45978.log.gz 2024-05-27T00:34:53.818 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43147.log 2024-05-27T00:34:53.818 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38555.log.gz 2024-05-27T00:34:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45614.log 2024-05-27T00:34:53.821 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35422.log.gz 2024-05-27T00:34:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40672.log 2024-05-27T00:34:53.821 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43147.log.gz 2024-05-27T00:34:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42166.log 2024-05-27T00:34:53.822 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45614.log.gz 2024-05-27T00:34:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40672.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-27T00:34:53.822 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40672.log.gz 2024-05-27T00:34:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45796.log 2024-05-27T00:34:53.824 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42166.log.gz 2024-05-27T00:34:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35913.log 2024-05-27T00:34:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42037.log 2024-05-27T00:34:53.824 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45796.log.gz 2024-05-27T00:34:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39540.log 2024-05-27T00:34:53.825 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35913.log.gz 2024-05-27T00:34:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35694.log 2024-05-27T00:34:53.825 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42037.log.gz 2024-05-27T00:34:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44954.log 2024-05-27T00:34:53.827 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39540.log.gz 2024-05-27T00:34:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45061.log 2024-05-27T00:34:53.827 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35694.log.gz 2024-05-27T00:34:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44079.log 2024-05-27T00:34:53.828 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44954.log.gz 2024-05-27T00:34:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42953.log 2024-05-27T00:34:53.829 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45061.log.gz 2024-05-27T00:34:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38962.log 2024-05-27T00:34:53.830 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44079.log.gz 2024-05-27T00:34:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38995.log 2024-05-27T00:34:53.831 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42953.log.gz 2024-05-27T00:34:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42522.log 2024-05-27T00:34:53.831 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38962.log.gz 2024-05-27T00:34:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-05-27T00:34:53.832 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38995.log.gz 2024-05-27T00:34:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38474.log 2024-05-27T00:34:53.832 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42522.log.gz 2024-05-27T00:34:53.833 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46033.log 2024-05-27T00:34:53.834 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-05-27T00:34:53.834 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41654.log 2024-05-27T00:34:53.834 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38474.log.gz 2024-05-27T00:34:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33939.log 2024-05-27T00:34:53.835 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46033.log.gz 2024-05-27T00:34:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39866.log 2024-05-27T00:34:53.837 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41654.log.gz 2024-05-27T00:34:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39458.log 2024-05-27T00:34:53.837 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33939.log.gz 2024-05-27T00:34:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2024-05-27T00:34:53.839 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39866.log.gz 2024-05-27T00:34:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33691.log 2024-05-27T00:34:53.840 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39458.log.gz 2024-05-27T00:34:53.840 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39672.log 2024-05-27T00:34:53.840 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2024-05-27T00:34:53.841 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43303.log 2024-05-27T00:34:53.842 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33691.log.gz 2024-05-27T00:34:53.842 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2024-05-27T00:34:53.842 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39672.log.gz 2024-05-27T00:34:53.842 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38801.log 2024-05-27T00:34:53.842 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43303.log.gz 2024-05-27T00:34:53.844 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46057.log 2024-05-27T00:34:53.844 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2024-05-27T00:34:53.844 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41906.log 2024-05-27T00:34:53.844 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38801.log.gz 2024-05-27T00:34:53.845 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45141.log 2024-05-27T00:34:53.845 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46057.log.gz 2024-05-27T00:34:53.845 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45900.log 2024-05-27T00:34:53.848 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41906.log.gz 2024-05-27T00:34:53.848 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43718.log 2024-05-27T00:34:53.849 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45141.log.gz 2024-05-27T00:34:53.849 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39185.log 2024-05-27T00:34:53.849 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45900.log.gz 2024-05-27T00:34:53.849 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42849.log 2024-05-27T00:34:53.849 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43718.log.gz 2024-05-27T00:34:53.849 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39892.log 2024-05-27T00:34:53.851 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28625.log 2024-05-27T00:34:53.851 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-27T00:34:53.851 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.28625.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-27T00:34:53.852 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.7.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28625.log.gz 2024-05-27T00:34:53.852 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28673.log 2024-05-27T00:34:53.852 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28601.log 2024-05-27T00:34:53.852 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.28673.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-27T00:34:53.852 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28673.log.gz 2024-05-27T00:34:53.853 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.28601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28577.log 2024-05-27T00:34:53.853 INFO:teuthology.orchestra.run.smithi204.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.28601.log.gz 2024-05-27T00:34:53.853 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28697.log 2024-05-27T00:34:53.853 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.28577.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-27T00:34:53.854 INFO:teuthology.orchestra.run.smithi204.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28577.log.gz 2024-05-27T00:34:53.854 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/ceph-client.admin.28697.log: -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-05-27T00:34:53.854 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-27T00:34:53.854 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mgr.x.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28697.log.gz 2024-05-27T00:34:53.854 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28526.log 2024-05-27T00:34:53.854 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose --/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.c.log 2024-05-27T00:34:53.854 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.28526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28649.log 2024-05-27T00:34:53.865 INFO:teuthology.orchestra.run.smithi204.stderr: 87.0%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.28526.log.gz 2024-05-27T00:34:53.865 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-27T00:34:53.865 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.28649.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28649.log.gz 2024-05-27T00:34:53.865 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28553.log 2024-05-27T00:34:53.865 INFO:teuthology.orchestra.run.smithi204.stderr: 94.3%/var/log/ceph/ceph.audit.log: -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-05-27T00:34:53.866 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-05-27T00:34:53.866 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.28553.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.28553.log.gz 93.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-27T00:34:53.866 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-05-27T00:34:53.866 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-05-27T00:34:53.866 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-27T00:34:53.866 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mds.b.log: 94.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-27T00:34:53.867 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-27T00:34:53.867 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mon.b.log: 94.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-05-27T00:34:53.867 INFO:teuthology.orchestra.run.smithi204.stderr: 94.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz/var/log/ceph/ceph-osd.4.log: 2024-05-27T00:34:53.879 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39185.log.gz 2024-05-27T00:34:53.879 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45009.log 2024-05-27T00:34:53.879 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42849.log.gz 2024-05-27T00:34:53.879 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41985.log 2024-05-27T00:34:53.879 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39892.log.gz 2024-05-27T00:34:53.880 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40294.log 2024-05-27T00:34:53.880 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45009.log.gz 2024-05-27T00:34:53.880 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38694.log 2024-05-27T00:34:53.880 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41985.log.gz 2024-05-27T00:34:53.880 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40860.log 2024-05-27T00:34:53.880 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40294.log.gz 2024-05-27T00:34:53.881 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39346.log 2024-05-27T00:34:53.881 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38694.log.gz 2024-05-27T00:34:53.881 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44295.log 2024-05-27T00:34:53.881 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40860.log.gz 2024-05-27T00:34:53.881 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43875.log 2024-05-27T00:34:53.881 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39346.log.gz 2024-05-27T00:34:53.882 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38422.log 2024-05-27T00:34:53.882 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44295.log.gz 2024-05-27T00:34:53.882 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34550.log 2024-05-27T00:34:53.882 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38422.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43875.log.gz 2024-05-27T00:34:53.882 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-27T00:34:53.883 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38422.log.gz 2024-05-27T00:34:53.883 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43689.log 2024-05-27T00:34:53.883 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34550.log.gz 2024-05-27T00:34:53.883 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34684.log 2024-05-27T00:34:53.883 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34762.log 2024-05-27T00:34:53.883 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43689.log.gz 2024-05-27T00:34:53.884 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42091.log 2024-05-27T00:34:53.884 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34684.log.gz 2024-05-27T00:34:53.884 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44709.log 2024-05-27T00:34:53.884 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34762.log.gz 2024-05-27T00:34:53.884 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46232.log 2024-05-27T00:34:53.884 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42091.log.gz 2024-05-27T00:34:53.885 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35578.log 2024-05-27T00:34:53.885 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44709.log.gz 2024-05-27T00:34:53.885 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41742.log 2024-05-27T00:34:53.885 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46232.log.gz 2024-05-27T00:34:53.885 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45640.log 2024-05-27T00:34:53.885 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35578.log.gz 2024-05-27T00:34:53.886 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39099.log 2024-05-27T00:34:53.886 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41742.log.gz 2024-05-27T00:34:53.886 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33551.log 2024-05-27T00:34:53.886 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45640.log.gz 2024-05-27T00:34:53.886 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39266.log 2024-05-27T00:34:53.887 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39099.log.gz 2024-05-27T00:34:53.887 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33953.log 2024-05-27T00:34:53.887 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33551.log.gz 2024-05-27T00:34:53.887 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39047.log 2024-05-27T00:34:53.887 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39266.log.gz 2024-05-27T00:34:53.887 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35994.log 2024-05-27T00:34:53.888 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33953.log.gz 2024-05-27T00:34:53.888 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.35136.log 2024-05-27T00:34:53.888 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39047.log.gz 2024-05-27T00:34:53.888 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40268.log 2024-05-27T00:34:53.888 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35994.log.gz 2024-05-27T00:34:53.888 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.0.35136.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-27T00:34:53.889 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.35136.log.gz 2024-05-27T00:34:53.889 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45337.log 2024-05-27T00:34:53.889 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40268.log.gz 2024-05-27T00:34:53.889 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46135.log 2024-05-27T00:34:53.889 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43551.log 2024-05-27T00:34:53.889 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45337.log.gz 2024-05-27T00:34:53.890 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35396.log 2024-05-27T00:34:53.890 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46135.log.gz 2024-05-27T00:34:53.890 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36024.log 2024-05-27T00:34:53.890 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43551.log.gz 2024-05-27T00:34:53.899 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33497.log 2024-05-27T00:34:53.900 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35396.log.gz 2024-05-27T00:34:53.900 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34162.log 2024-05-27T00:34:53.900 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36024.log.gz 2024-05-27T00:34:53.900 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43468.log 2024-05-27T00:34:53.900 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34162.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33497.log.gz 2024-05-27T00:34:53.901 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39918.log 2024-05-27T00:34:53.901 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34162.log.gz 2024-05-27T00:34:53.901 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43468.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43468.log.gz -5 2024-05-27T00:34:53.901 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39237.log 2024-05-27T00:34:53.901 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45770.log 2024-05-27T00:34:53.901 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39918.log.gz 2024-05-27T00:34:53.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39237.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39237.log.gz 2024-05-27T00:34:53.902 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44763.log 2024-05-27T00:34:53.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45770.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45770.log.gz 2024-05-27T00:34:53.902 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42979.log 2024-05-27T00:34:53.902 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.44763.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39073.log 2024-05-27T00:34:53.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42979.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44763.log.gz 2024-05-27T00:34:53.903 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45744.log 2024-05-27T00:34:53.903 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42979.log.gz 2024-05-27T00:34:53.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38749.log 2024-05-27T00:34:53.903 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39073.log.gz 2024-05-27T00:34:53.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41680.log 2024-05-27T00:34:53.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38749.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45744.log.gz 2024-05-27T00:34:53.904 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39620.log 2024-05-27T00:34:53.904 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38749.log.gz 2024-05-27T00:34:53.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36076.log 2024-05-27T00:34:53.904 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41680.log.gz 2024-05-27T00:34:53.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42629.log 2024-05-27T00:34:53.904 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39620.log.gz 2024-05-27T00:34:53.905 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44980.log 2024-05-27T00:34:53.905 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36076.log.gz 2024-05-27T00:34:53.905 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40568.log 2024-05-27T00:34:53.905 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42629.log.gz 2024-05-27T00:34:53.905 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35526.log 2024-05-27T00:34:53.905 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44980.log: /var/log/ceph/ceph-client.admin.40568.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44980.log.gz 2024-05-27T00:34:53.906 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46083.log 2024-05-27T00:34:53.906 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40568.log.gz 2024-05-27T00:34:53.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35526.log: gzip -5 88.1% --verbose -- replaced with /var/log/ceph/ceph-client.admin.35526.log.gz -- 2024-05-27T00:34:53.906 INFO:teuthology.orchestra.run.smithi098.stderr: /var/log/ceph/ceph-client.admin.35800.log 2024-05-27T00:34:53.911 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41550.log 2024-05-27T00:34:53.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35800.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46083.log.gz 2024-05-27T00:34:53.912 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45087.log 2024-05-27T00:34:53.912 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35800.log.gz 2024-05-27T00:34:53.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40646.log 2024-05-27T00:34:53.912 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41550.log.gz 2024-05-27T00:34:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.35118.log 2024-05-27T00:34:53.913 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45087.log.gz 2024-05-27T00:34:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41602.log 2024-05-27T00:34:53.913 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40646.log.gz 2024-05-27T00:34:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.0.35118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.35118.log.gz 2024-05-27T00:34:53.913 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42577.log 2024-05-27T00:34:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44737.log 2024-05-27T00:34:53.914 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41602.log.gz 2024-05-27T00:34:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45482.log 2024-05-27T00:34:53.914 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42577.log.gz 2024-05-27T00:34:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41398.log 2024-05-27T00:34:53.914 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44737.log.gz 2024-05-27T00:34:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43120.log 2024-05-27T00:34:53.915 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45482.log.gz 2024-05-27T00:34:53.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42823.log 2024-05-27T00:34:53.915 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41398.log.gz 2024-05-27T00:34:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35318.log 2024-05-27T00:34:53.919 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43120.log.gz 2024-05-27T00:34:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42288.log 2024-05-27T00:34:53.920 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42823.log.gz 2024-05-27T00:34:53.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43332.log 2024-05-27T00:34:53.920 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35318.log.gz 2024-05-27T00:34:53.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39372.log 2024-05-27T00:34:53.920 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42288.log.gz 2024-05-27T00:34:53.920 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.43332.log: -- /var/log/ceph/ceph-client.admin.39758.log 2024-05-27T00:34:53.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39372.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43332.log.gz 2024-05-27T00:34:53.921 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43770.log 2024-05-27T00:34:53.921 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz 2024-05-27T00:34:53.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33419.log 2024-05-27T00:34:53.921 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39758.log.gz 2024-05-27T00:34:53.921 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.43770.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44105.log 2024-05-27T00:34:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33419.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43770.log.gz 2024-05-27T00:34:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44157.log 2024-05-27T00:34:53.924 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33419.log.gz 2024-05-27T00:34:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45588.log 2024-05-27T00:34:53.925 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44105.log.gz 2024-05-27T00:34:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41267.log 2024-05-27T00:34:53.925 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44157.log.gz 2024-05-27T00:34:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40542.log 2024-05-27T00:34:53.925 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45588.log.gz 2024-05-27T00:34:53.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39970.log 2024-05-27T00:34:53.928 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41267.log.gz 2024-05-27T00:34:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42551.log 2024-05-27T00:34:53.928 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5%/var/log/ceph/ceph-client.admin.39970.log: -- replaced with /var/log/ceph/ceph-client.admin.40542.log.gz 2024-05-27T00:34:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35853.log 2024-05-27T00:34:53.928 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39970.log.gz 2024-05-27T00:34:53.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45822.log 2024-05-27T00:34:53.929 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42551.log.gz 2024-05-27T00:34:53.929 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.35853.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46109.log 2024-05-27T00:34:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45822.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35853.log.gz 2024-05-27T00:34:53.930 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33848.log 2024-05-27T00:34:53.930 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45822.log.gz 2024-05-27T00:34:53.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39840.log 2024-05-27T00:34:53.931 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46109.log.gz 2024-05-27T00:34:53.939 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45926.log 2024-05-27T00:34:53.940 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33848.log.gz 2024-05-27T00:34:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41346.log 2024-05-27T00:34:53.940 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39840.log.gz 2024-05-27T00:34:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34364.log 2024-05-27T00:34:53.940 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45926.log.gz 2024-05-27T00:34:53.940 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46161.log 2024-05-27T00:34:53.940 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41346.log.gz 2024-05-27T00:34:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.34364.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43065.log 2024-05-27T00:34:53.941 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7%/var/log/ceph/ceph-client.admin.46161.log: -- replaced with /var/log/ceph/ceph-client.admin.34364.log.gz 2024-05-27T00:34:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40965.log 2024-05-27T00:34:53.941 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46161.log.gz 2024-05-27T00:34:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33769.log 2024-05-27T00:34:53.941 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43065.log.gz 2024-05-27T00:34:53.941 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40050.log 2024-05-27T00:34:53.941 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40965.log.gz 2024-05-27T00:34:53.942 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35774.log 2024-05-27T00:34:53.943 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33769.log.gz 2024-05-27T00:34:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40050.log: gzip 89.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40050.log.gz --verbose 2024-05-27T00:34:53.943 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/ceph-client.admin.38607.log 2024-05-27T00:34:53.943 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35774.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-27T00:34:53.943 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35774.log.gz 2024-05-27T00:34:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38667.log 2024-05-27T00:34:53.944 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-05-27T00:34:53.944 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45560.log 2024-05-27T00:34:53.945 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39292.log 2024-05-27T00:34:53.946 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38667.log.gz 2024-05-27T00:34:53.946 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34999.log 2024-05-27T00:34:53.947 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45560.log.gz 2024-05-27T00:34:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43091.log 2024-05-27T00:34:53.948 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39292.log.gz 2024-05-27T00:34:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33281.log 2024-05-27T00:34:53.949 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34999.log.gz 2024-05-27T00:34:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33613.log 2024-05-27T00:34:53.950 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43091.log.gz 2024-05-27T00:34:53.951 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35604.log 2024-05-27T00:34:53.951 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33281.log.gz 2024-05-27T00:34:53.952 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44841.log 2024-05-27T00:34:53.952 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33613.log.gz 2024-05-27T00:34:53.953 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33665.log 2024-05-27T00:34:53.954 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35604.log.gz 2024-05-27T00:34:53.954 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41959.log 2024-05-27T00:34:53.955 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44841.log.gz 2024-05-27T00:34:53.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45193.log 2024-05-27T00:34:53.956 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33665.log.gz 2024-05-27T00:34:53.957 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40024.log 2024-05-27T00:34:53.957 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41959.log.gz 2024-05-27T00:34:53.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39944.log 2024-05-27T00:34:53.958 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45193.log.gz 2024-05-27T00:34:53.959 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43416.log 2024-05-27T00:34:53.959 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40024.log.gz 2024-05-27T00:34:53.960 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44430.log 2024-05-27T00:34:53.960 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39944.log.gz 2024-05-27T00:34:53.961 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39424.log 2024-05-27T00:34:53.962 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43416.log.gz 2024-05-27T00:34:53.962 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34654.log 2024-05-27T00:34:53.963 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44430.log.gz 2024-05-27T00:34:53.963 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42469.log 2024-05-27T00:34:53.964 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39424.log.gz 2024-05-27T00:34:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43225.log 2024-05-27T00:34:53.965 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34654.log.gz 2024-05-27T00:34:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35162.log 2024-05-27T00:34:53.966 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42469.log.gz 2024-05-27T00:34:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41132.log 2024-05-27T00:34:53.967 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43225.log.gz 2024-05-27T00:34:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35162.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.23131.log 2024-05-27T00:34:53.968 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35162.log.gz 2024-05-27T00:34:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2024-05-27T00:34:53.969 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41132.log.gz 2024-05-27T00:34:53.970 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.tmp-client.admin.23131.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-27T00:34:53.970 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.23131.log.gz 2024-05-27T00:34:53.971 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45666.log 2024-05-27T00:34:53.971 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz 2024-05-27T00:34:53.972 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33187.log 2024-05-27T00:34:53.974 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34840.log 2024-05-27T00:34:53.974 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45666.log.gz 2024-05-27T00:34:53.975 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33187.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33187.log.gz 2024-05-27T00:34:53.975 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40516.log 2024-05-27T00:34:53.976 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34840.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34840.log.gz 2024-05-27T00:34:53.976 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40348.log 2024-05-27T00:34:53.977 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40516.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40516.log.gz 2024-05-27T00:34:53.978 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41106.log 2024-05-27T00:34:53.979 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43442.log 2024-05-27T00:34:53.979 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40348.log.gz 2024-05-27T00:34:53.980 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41106.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41106.log.gz 2024-05-27T00:34:53.981 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38883.log 2024-05-27T00:34:53.981 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43442.log.gz 2024-05-27T00:34:53.982 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45224.log 2024-05-27T00:34:53.983 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41576.log 2024-05-27T00:34:53.984 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38883.log.gz 2024-05-27T00:34:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45224.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45224.log.gz 2024-05-27T00:34:53.985 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41576.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41576.log.gz 2024-05-27T00:34:53.986 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44616.log 2024-05-27T00:34:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35500.log 2024-05-27T00:34:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46258.log 2024-05-27T00:34:53.989 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44616.log.gz 2024-05-27T00:34:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43637.log 2024-05-27T00:34:53.990 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35500.log.gz 2024-05-27T00:34:53.991 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46258.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46258.log.gz 2024-05-27T00:34:53.991 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39398.log 2024-05-27T00:34:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43637.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43637.log.gz 2024-05-27T00:34:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35370.log 2024-05-27T00:34:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45115.log 2024-05-27T00:34:53.994 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39398.log.gz 2024-05-27T00:34:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35370.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35370.log.gz 2024-05-27T00:34:53.996 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45115.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45115.log.gz 2024-05-27T00:34:53.996 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39211.log 2024-05-27T00:34:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39784.log 2024-05-27T00:34:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40216.log 2024-05-27T00:34:53.999 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39211.log.gz 2024-05-27T00:34:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39784.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39784.log.gz 2024-05-27T00:34:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40216.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40216.log.gz 2024-05-27T00:34:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34154.log 2024-05-27T00:34:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39646.log 2024-05-27T00:34:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34918.log 2024-05-27T00:34:54.005 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34154.log.gz 2024-05-27T00:34:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39646.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39646.log.gz 2024-05-27T00:34:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34918.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34918.log.gz 2024-05-27T00:34:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44243.log 2024-05-27T00:34:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34151.log 2024-05-27T00:34:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34866.log 2024-05-27T00:34:54.010 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44243.log.gz 2024-05-27T00:34:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34151.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34151.log.gz 2024-05-27T00:34:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34866.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34866.log.gz 2024-05-27T00:34:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41933.log 2024-05-27T00:34:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45415.log 2024-05-27T00:34:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38855.log 2024-05-27T00:34:54.015 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41933.log.gz 2024-05-27T00:34:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44928.log 2024-05-27T00:34:54.016 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45415.log.gz 2024-05-27T00:34:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38855.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38855.log.gz 2024-05-27T00:34:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44324.log 2024-05-27T00:34:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44928.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44928.log.gz 2024-05-27T00:34:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44324.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44324.log.gz 2024-05-27T00:34:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42120.log 2024-05-27T00:34:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41849.log 2024-05-27T00:34:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34814.log 2024-05-27T00:34:54.023 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42120.log.gz 2024-05-27T00:34:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41768.log 2024-05-27T00:34:54.024 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41849.log.gz 2024-05-27T00:34:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34814.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34814.log.gz 2024-05-27T00:34:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42769.log 2024-05-27T00:34:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41768.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41768.log.gz 2024-05-27T00:34:54.027 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28093.log 2024-05-27T00:34:54.028 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46004.log 2024-05-27T00:34:54.028 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42769.log.gz 2024-05-27T00:34:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28093.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28093.log.gz 2024-05-27T00:34:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46004.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46004.log.gz 2024-05-27T00:34:54.031 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40489.log 2024-05-27T00:34:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34370.log 2024-05-27T00:34:54.034 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40886.log 2024-05-27T00:34:54.034 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40489.log.gz 2024-05-27T00:34:54.035 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34370.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34370.log.gz 2024-05-27T00:34:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40886.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40886.log.gz 2024-05-27T00:34:54.036 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33245.log 2024-05-27T00:34:54.037 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34358.log 2024-05-27T00:34:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41480.log 2024-05-27T00:34:54.039 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33245.log.gz 2024-05-27T00:34:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34358.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34358.log.gz 2024-05-27T00:34:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41480.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41480.log.gz 2024-05-27T00:34:54.041 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43251.log 2024-05-27T00:34:54.042 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-27T00:34:54.044 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33471.log 2024-05-27T00:34:54.044 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43251.log.gz 2024-05-27T00:34:54.045 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44508.log 2024-05-27T00:34:54.046 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33471.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33471.log.gz 2024-05-27T00:34:54.047 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34160.log 2024-05-27T00:34:54.047 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44508.log.gz 2024-05-27T00:34:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40620.log 2024-05-27T00:34:54.049 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-27T00:34:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34160.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34160.log.gz 2024-05-27T00:34:54.050 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33874.log 2024-05-27T00:34:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40620.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40620.log.gz 2024-05-27T00:34:54.051 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41189.log 2024-05-27T00:34:54.052 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40164.log 2024-05-27T00:34:54.053 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33874.log.gz 2024-05-27T00:34:54.053 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41189.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41189.log.gz 2024-05-27T00:34:54.054 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40164.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40164.log.gz 2024-05-27T00:34:54.055 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45692.log 2024-05-27T00:34:54.056 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41017.log 2024-05-27T00:34:54.057 INFO:teuthology.orchestra.run.smithi204.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-27T00:34:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35266.log 2024-05-27T00:34:54.058 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45692.log.gz 2024-05-27T00:34:54.058 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41017.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41017.log.gz 2024-05-27T00:34:54.060 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35266.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35266.log.gz 2024-05-27T00:34:54.060 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40190.log 2024-05-27T00:34:54.061 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28220.log 2024-05-27T00:34:54.063 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42397.log 2024-05-27T00:34:54.063 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40190.log.gz 2024-05-27T00:34:54.063 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28220.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28220.log.gz 2024-05-27T00:34:54.065 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42397.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42397.log.gz 2024-05-27T00:34:54.065 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34602.log 2024-05-27T00:34:54.066 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45272.log 2024-05-27T00:34:54.068 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40463.log 2024-05-27T00:34:54.068 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34602.log.gz 2024-05-27T00:34:54.069 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45272.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45272.log.gz 2024-05-27T00:34:54.069 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39996.log 2024-05-27T00:34:54.070 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40437.log 2024-05-27T00:34:54.070 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40463.log.gz 2024-05-27T00:34:54.072 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44587.log 2024-05-27T00:34:54.072 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39996.log.gz 2024-05-27T00:34:54.073 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40437.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40437.log.gz 2024-05-27T00:34:54.074 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44587.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44587.log.gz 2024-05-27T00:34:54.074 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45952.log 2024-05-27T00:34:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-05-27T00:34:54.077 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39021.log 2024-05-27T00:34:54.077 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45952.log.gz 2024-05-27T00:34:54.078 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40408.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-05-27T00:34:54.079 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39021.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39021.log.gz 2024-05-27T00:34:54.079 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43849.log 2024-05-27T00:34:54.080 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44402.log 2024-05-27T00:34:54.082 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34145.log 2024-05-27T00:34:54.083 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43849.log.gz 2024-05-27T00:34:54.083 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44402.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44402.log.gz 2024-05-27T00:34:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34145.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34145.log.gz 2024-05-27T00:34:54.084 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45848.log 2024-05-27T00:34:54.085 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33391.log 2024-05-27T00:34:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44269.log 2024-05-27T00:34:54.087 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45848.log.gz 2024-05-27T00:34:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33391.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33391.log.gz 2024-05-27T00:34:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44269.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44269.log.gz 2024-05-27T00:34:54.090 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34153.log 2024-05-27T00:34:54.091 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34736.log 2024-05-27T00:34:54.092 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38916.log 2024-05-27T00:34:54.092 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34153.log.gz 2024-05-27T00:34:54.093 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35292.log 2024-05-27T00:34:54.093 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34736.log.gz 2024-05-27T00:34:54.094 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38916.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38916.log.gz 2024-05-27T00:34:54.095 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43936.log 2024-05-27T00:34:54.095 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35292.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35292.log.gz 2024-05-27T00:34:54.097 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44535.log 2024-05-27T00:34:54.097 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43936.log.gz 2024-05-27T00:34:54.098 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44183.log 2024-05-27T00:34:54.100 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42371.log 2024-05-27T00:34:54.100 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44535.log.gz 2024-05-27T00:34:54.101 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43039.log 2024-05-27T00:34:54.101 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44183.log.gz 2024-05-27T00:34:54.102 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42371.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42371.log.gz 2024-05-27T00:34:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39814.log 2024-05-27T00:34:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43039.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43039.log.gz 2024-05-27T00:34:54.103 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34944.log 2024-05-27T00:34:54.105 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38500.log 2024-05-27T00:34:54.105 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39814.log.gz 2024-05-27T00:34:54.107 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34944.log: /var/log/ceph/ceph-client.admin.38500.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.34944.log.gz 2024-05-27T00:34:54.107 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38500.log.gz 2024-05-27T00:34:54.108 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44561.log 2024-05-27T00:34:54.109 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45718.log 2024-05-27T00:34:54.110 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33943.log 2024-05-27T00:34:54.111 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44561.log.gz 2024-05-27T00:34:54.111 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45718.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45718.log.gz 2024-05-27T00:34:54.113 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33943.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33943.log.gz 2024-05-27T00:34:54.113 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33935.log 2024-05-27T00:34:54.114 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33365.log 2024-05-27T00:34:54.116 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35748.log 2024-05-27T00:34:54.116 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33935.log.gz 2024-05-27T00:34:54.117 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44053.log 2024-05-27T00:34:54.117 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33365.log.gz 2024-05-27T00:34:54.118 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35748.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35748.log.gz 2024-05-27T00:34:54.118 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45508.log 2024-05-27T00:34:54.119 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44053.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44053.log.gz 2024-05-27T00:34:54.119 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43277.log 2024-05-27T00:34:54.121 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33822.log 2024-05-27T00:34:54.121 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45508.log.gz 2024-05-27T00:34:54.122 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43277.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43277.log.gz 2024-05-27T00:34:54.123 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33822.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33822.log.gz 2024-05-27T00:34:54.123 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44894.log 2024-05-27T00:34:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36111.log 2024-05-27T00:34:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27981.log 2024-05-27T00:34:54.126 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44894.log.gz 2024-05-27T00:34:54.127 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36111.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36111.log.gz 2024-05-27T00:34:54.128 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27981.log: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.27981.log.gz 2024-05-27T00:34:54.129 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44131.log 2024-05-27T00:34:54.130 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42655.log 2024-05-27T00:34:54.132 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34369.log 2024-05-27T00:34:54.132 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44131.log.gz 2024-05-27T00:34:54.132 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44789.log 2024-05-27T00:34:54.133 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42655.log.gz 2024-05-27T00:34:54.134 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34369.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34369.log.gz 2024-05-27T00:34:54.134 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44789.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-27T00:34:54.135 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44789.log.gz 2024-05-27T00:34:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-05-27T00:34:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43603.log 2024-05-27T00:34:54.139 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.43603.log: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-27T00:34:54.140 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43603.log.gz 2024-05-27T00:34:54.140 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41794.log 2024-05-27T00:34:54.141 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35642.log 2024-05-27T00:34:54.142 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41794.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41794.log.gz 2024-05-27T00:34:54.143 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35642.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35642.log.gz 2024-05-27T00:34:54.144 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33445.log 2024-05-27T00:34:54.145 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42345.log 2024-05-27T00:34:54.146 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33445.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33445.log.gz 2024-05-27T00:34:54.147 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42345.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42345.log.gz 2024-05-27T00:34:54.148 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33795.log 2024-05-27T00:34:54.149 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40912.log 2024-05-27T00:34:54.151 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33795.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33795.log.gz 2024-05-27T00:34:54.151 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40912.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40912.log.gz 2024-05-27T00:34:54.152 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34342.log 2024-05-27T00:34:54.153 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-05-27T00:34:54.155 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34342.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34342.log.gz 2024-05-27T00:34:54.157 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45534.log 2024-05-27T00:34:54.160 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38829.log 2024-05-27T00:34:54.160 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35668.log 2024-05-27T00:34:54.160 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45534.log.gz 2024-05-27T00:34:54.161 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38581.log 2024-05-27T00:34:54.163 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35668.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35668.log.gz 2024-05-27T00:34:54.164 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42733.log 2024-05-27T00:34:54.165 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38829.log.gz 2024-05-27T00:34:54.167 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42733.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42733.log.gz 2024-05-27T00:34:54.168 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38581.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38581.log.gz 2024-05-27T00:34:54.169 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40755.log 2024-05-27T00:34:54.170 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41524.log 2024-05-27T00:34:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40755.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40755.log.gz 2024-05-27T00:34:54.174 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44815.log 2024-05-27T00:34:54.178 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41524.log: /var/log/ceph/ceph-client.admin.44815.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41524.log.gz 2024-05-27T00:34:54.178 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45389.log 2024-05-27T00:34:54.179 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44815.log.gz 2024-05-27T00:34:54.181 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35827.log 2024-05-27T00:34:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45389.log: /var/log/ceph/ceph-client.admin.35827.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35827.log.gz 2024-05-27T00:34:54.186 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45389.log.gz 2024-05-27T00:34:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39320.log 2024-05-27T00:34:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39320.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39320.log.gz 2024-05-27T00:34:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39151.log 2024-05-27T00:34:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39151.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39151.log.gz 2024-05-27T00:34:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34892.log 2024-05-27T00:34:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34892.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34892.log.gz 2024-05-27T00:34:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42262.log 2024-05-27T00:34:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34366.log 2024-05-27T00:34:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42262.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42262.log.gz 2024-05-27T00:34:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34366.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34366.log.gz 2024-05-27T00:34:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33717.log 2024-05-27T00:34:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33717.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33717.log.gz 2024-05-27T00:34:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40076.log 2024-05-27T00:34:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35474.log 2024-05-27T00:34:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40076.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40076.log.gz 2024-05-27T00:34:54.210 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35474.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35474.log.gz 2024-05-27T00:34:54.210 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35720.log 2024-05-27T00:34:54.213 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35720.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35720.log.gz 2024-05-27T00:34:54.213 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40726.log 2024-05-27T00:34:54.216 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40726.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40726.log.gz 2024-05-27T00:34:54.216 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41215.log 2024-05-27T00:34:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41432.log 2024-05-27T00:34:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41215.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41215.log.gz 2024-05-27T00:34:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41432.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41432.log.gz 2024-05-27T00:34:54.223 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36050.log 2024-05-27T00:34:54.226 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36050.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36050.log.gz 2024-05-27T00:34:54.226 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43744.log 2024-05-27T00:34:54.231 INFO:teuthology.orchestra.run.smithi204.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-27T00:34:54.232 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43744.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43744.log.gz 2024-05-27T00:34:54.233 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44868.log 2024-05-27T00:34:54.233 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44868.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44868.log.gz 2024-05-27T00:34:54.233 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35344.log 2024-05-27T00:34:54.235 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35344.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35344.log.gz 2024-05-27T00:34:54.236 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33523.log 2024-05-27T00:34:54.236 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42011.log 2024-05-27T00:34:54.238 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33523.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33523.log.gz 2024-05-27T00:34:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42011.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42011.log.gz 2024-05-27T00:34:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43173.log 2024-05-27T00:34:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43173.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43173.log.gz 2024-05-27T00:34:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39512.log 2024-05-27T00:34:54.247 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39512.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39512.log.gz 2024-05-27T00:34:54.247 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44350.log 2024-05-27T00:34:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40322.log 2024-05-27T00:34:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44350.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44350.log.gz 2024-05-27T00:34:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35552.log 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40322.log.gz 2024-05-27T00:34:54.251 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-05-27T00:34:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35552.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35552.log.gz 2024-05-27T00:34:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38775.log 2024-05-27T00:34:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38775.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38775.log.gz 2024-05-27T00:34:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42423.log 2024-05-27T00:34:54.260 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42423.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42423.log.gz 2024-05-27T00:34:54.260 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34368.log 2024-05-27T00:34:54.265 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35887.log 2024-05-27T00:34:54.265 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34368.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34368.log.gz 2024-05-27T00:34:54.265 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35448.log 2024-05-27T00:34:54.266 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35887.log.gz 2024-05-27T00:34:54.269 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35448.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35448.log.gz 2024-05-27T00:34:54.269 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43497.log 2024-05-27T00:34:54.269 INFO:teuthology.orchestra.run.smithi204.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-27T00:34:54.271 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39732.log 2024-05-27T00:34:54.272 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43497.log.gz 2024-05-27T00:34:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42603.log 2024-05-27T00:34:54.276 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39732.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39732.log.gz 2024-05-27T00:34:54.278 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33954.log 2024-05-27T00:34:54.279 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42603.log.gz 2024-05-27T00:34:54.282 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45874.log 2024-05-27T00:34:54.283 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33954.log.gz 2024-05-27T00:34:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43005.log 2024-05-27T00:34:54.289 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45874.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45874.log.gz 2024-05-27T00:34:54.290 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43005.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43005.log.gz 2024-05-27T00:34:54.290 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33639.log 2024-05-27T00:34:54.293 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33639.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33639.log.gz 2024-05-27T00:34:54.293 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34628.log 2024-05-27T00:34:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34628.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34628.log.gz 2024-05-27T00:34:54.296 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40700.log 2024-05-27T00:34:54.299 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40700.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40700.log.gz 2024-05-27T00:34:54.299 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41241.log 2024-05-27T00:34:54.302 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43525.log 2024-05-27T00:34:54.302 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41241.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41241.log.gz 2024-05-27T00:34:54.305 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43525.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43525.log.gz 2024-05-27T00:34:54.305 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44456.log 2024-05-27T00:34:54.308 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44456.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44456.log.gz 2024-05-27T00:34:54.308 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41628.log 2024-05-27T00:34:54.311 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41628.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41628.log.gz 2024-05-27T00:34:54.311 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39698.log 2024-05-27T00:34:54.315 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36157.log 2024-05-27T00:34:54.315 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39698.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39698.log.gz 2024-05-27T00:34:54.315 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43577.log 2024-05-27T00:34:54.315 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36157.log.gz 2024-05-27T00:34:54.317 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2024-05-27T00:34:54.318 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43577.log.gz 2024-05-27T00:34:54.319 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40938.log 2024-05-27T00:34:54.328 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41372.log: /var/log/ceph/ceph-client.admin.40938.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gz 2024-05-27T00:34:54.328 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45035.log 2024-05-27T00:34:54.328 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40938.log.gz 2024-05-27T00:34:54.328 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45035.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45035.log.gz 2024-05-27T00:34:54.328 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35968.log 2024-05-27T00:34:54.329 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35968.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35968.log.gz 2024-05-27T00:34:54.329 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40382.log 2024-05-27T00:34:54.331 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40382.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40382.log.gz 2024-05-27T00:34:54.331 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44482.log 2024-05-27T00:34:54.333 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33947.log 2024-05-27T00:34:54.333 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.44482.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44482.log.gz 2024-05-27T00:34:54.336 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33947.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33947.log.gz 2024-05-27T00:34:54.337 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40242.log 2024-05-27T00:34:54.341 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40242.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40242.log.gz 2024-05-27T00:34:54.342 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38723.log 2024-05-27T00:34:54.343 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38723.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38723.log.gz 2024-05-27T00:34:54.343 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38448.log 2024-05-27T00:34:54.346 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39566.log 2024-05-27T00:34:54.346 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38448.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38448.log.gz 2024-05-27T00:34:54.348 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39566.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39566.log.gz 2024-05-27T00:34:54.348 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41708.log 2024-05-27T00:34:54.351 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41708.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41708.log.gz 2024-05-27T00:34:54.351 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35214.log 2024-05-27T00:34:54.354 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35214.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35214.log.gz 2024-05-27T00:34:54.354 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38641.log 2024-05-27T00:34:54.357 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38641.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38641.log.gz 2024-05-27T00:34:54.357 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-05-27T00:34:54.361 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34576.log 2024-05-27T00:34:54.364 INFO:teuthology.orchestra.run.smithi098.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-05-27T00:34:54.366 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45363.log 2024-05-27T00:34:54.367 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34576.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34576.log.gz 2024-05-27T00:34:54.369 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43797.log 2024-05-27T00:34:54.371 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45363.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45363.log.gz 2024-05-27T00:34:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34710.log 2024-05-27T00:34:54.375 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43797.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43797.log.gz 2024-05-27T00:34:54.377 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.34710.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42681.log 2024-05-27T00:34:54.378 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34710.log.gz 2024-05-27T00:34:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.42681.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43823.log 2024-05-27T00:34:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43823.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43823.log.gz 2024-05-27T00:34:54.385 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42681.log.gz 2024-05-27T00:34:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33946.log 2024-05-27T00:34:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33946.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33946.log.gz 2024-05-27T00:34:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42495.log 2024-05-27T00:34:54.391 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42495.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42495.log.gz 2024-05-27T00:34:54.391 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33579.log 2024-05-27T00:34:54.394 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33579.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33579.log.gz 2024-05-27T00:34:54.394 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42314.log 2024-05-27T00:34:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42314.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42314.log.gz 2024-05-27T00:34:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43663.log 2024-05-27T00:34:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.43663.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43663.log.gz 2024-05-27T00:34:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41293.log 2024-05-27T00:34:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41293.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41293.log.gz 2024-05-27T00:34:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33950.log 2024-05-27T00:34:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33743.log 2024-05-27T00:34:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33950.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33950.log.gz 2024-05-27T00:34:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33743.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33743.log.gz 2024-05-27T00:34:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45441.log 2024-05-27T00:34:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45441.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45441.log.gz 2024-05-27T00:34:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42707.log 2024-05-27T00:34:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.42707.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42707.log.gz 2024-05-27T00:34:54.416 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38528.log 2024-05-27T00:34:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38528.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38528.log.gz 2024-05-27T00:34:54.419 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40833.log 2024-05-27T00:34:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40833.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40833.log.gz 2024-05-27T00:34:54.423 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45167.log 2024-05-27T00:34:54.424 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46187.log 2024-05-27T00:34:54.425 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.45167.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45167.log.gz 2024-05-27T00:34:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.46187.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46187.log.gz 2024-05-27T00:34:54.734 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-27T00:34:59.181 INFO:teuthology.orchestra.run.smithi098.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-27T00:34:59.724 INFO:teuthology.orchestra.run.smithi204.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-27T00:34:59.767 INFO:teuthology.orchestra.run.smithi204.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-27T00:35:00.248 INFO:teuthology.orchestra.run.smithi098.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-27T00:35:02.000 INFO:teuthology.orchestra.run.smithi204.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-27T00:35:02.491 INFO:teuthology.orchestra.run.smithi204.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-27T00:35:02.492 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-05-27T00:35:02.492 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m8.672s 2024-05-27T00:35:02.492 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m28.683s 2024-05-27T00:35:02.492 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m0.955s 2024-05-27T00:35:03.444 INFO:teuthology.orchestra.run.smithi098.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-27T00:35:04.428 INFO:teuthology.orchestra.run.smithi098.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-27T00:35:22.778 INFO:teuthology.orchestra.run.smithi098.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-05-27T00:36:10.579 INFO:teuthology.orchestra.run.smithi098.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-05-27T00:36:10.581 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-05-27T00:36:10.581 INFO:teuthology.orchestra.run.smithi098.stderr:real 1m16.798s 2024-05-27T00:36:10.581 INFO:teuthology.orchestra.run.smithi098.stderr:user 2m8.217s 2024-05-27T00:36:10.581 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m10.196s 2024-05-27T00:36:10.583 INFO:tasks.ceph:Archiving logs... 2024-05-27T00:36:10.583 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174/remote/smithi098/log 2024-05-27T00:36:10.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T00:36:37.317 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174/remote/smithi204/log 2024-05-27T00:36:37.318 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T00:36:40.587 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-27T00:36:40.599 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-27T00:36:40.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T00:36:40.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T00:36:40.684 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-27T00:36:40.693 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-27T00:36:40.693 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T00:36:40.697 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T00:36:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2024-05-27T00:36:40.710 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2024-05-27T00:36:40.710 INFO:teuthology.orchestra.run.smithi098.stdout:*hv01.front.sepi 67.205.162.81 3 u 11 64 377 0.070 -1.538 0.160 2024-05-27T00:36:40.710 INFO:teuthology.orchestra.run.smithi098.stdout:+hv02.front.sepi 63.231.80.2 3 u 16 64 377 0.093 -0.105 0.177 2024-05-27T00:36:40.710 INFO:teuthology.orchestra.run.smithi098.stdout:+hv03.front.sepi 74.6.168.72 3 u 24 64 377 0.088 -1.433 0.169 2024-05-27T00:36:40.710 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-05-27T00:36:40.711 INFO:teuthology.orchestra.run.smithi204.stdout: remote refid st t when poll reach delay offset jitter 2024-05-27T00:36:40.711 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================== 2024-05-27T00:36:40.711 INFO:teuthology.orchestra.run.smithi204.stdout:*hv01.front.sepi 67.205.162.81 3 u 1 64 377 0.114 -2.830 0.427 2024-05-27T00:36:40.711 INFO:teuthology.orchestra.run.smithi204.stdout:+hv02.front.sepi 63.231.80.2 3 u 1 64 377 0.086 -0.885 0.520 2024-05-27T00:36:40.711 INFO:teuthology.orchestra.run.smithi204.stdout:+hv03.front.sepi 74.6.168.72 3 u 13 64 377 0.071 -2.807 0.445 2024-05-27T00:36:40.711 INFO:teuthology.orchestra.run.smithi204.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-05-27T00:36:40.713 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-27T00:36:40.723 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-27T00:36:40.723 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-27T00:36:40.733 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-27T00:36:40.743 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-27T00:36:40.773 INFO:teuthology.task.internal:Duration was 1981.919694 seconds 2024-05-27T00:36:40.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-27T00:36:40.821 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-27T00:36:40.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T00:36:40.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T00:36:40.932 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-27T00:36:40.932 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-05-27T00:36:40.932 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T00:36:40.953 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-05-27T00:36:40.953 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T00:36:40.964 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-27T00:36:40.964 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T00:36:41.004 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T00:36:41.041 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-27T00:36:41.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T00:36:41.088 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T00:36:41.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-27T00:36:41.174 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-27T00:36:41.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T00:36:41.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T00:36:41.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-27T00:36:41.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T00:36:41.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T00:36:41.254 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-05-27T00:36:41.262 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-05-27T00:36:41.278 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T00:36:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:36:41.320 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T00:36:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T00:36:41.324 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-27T00:36:41.335 INFO:teuthology.task.internal:Transferring archived files... 2024-05-27T00:36:41.336 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174/remote/smithi098 2024-05-27T00:36:41.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T00:36:41.387 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-26_21:24:02-fs-squid-distro-default-smithi/7727174/remote/smithi204 2024-05-27T00:36:41.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T00:36:41.408 INFO:teuthology.task.internal:Removing archive directory... 2024-05-27T00:36:41.408 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T00:36:41.427 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-27T00:36:41.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-27T00:36:41.469 INFO:teuthology.task.internal:Not uploading archives. 2024-05-27T00:36:41.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-27T00:36:41.592 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-27T00:36:41.592 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T00:36:41.595 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T00:36:41.601 INFO:teuthology.orchestra.run.smithi098.stdout: 655428 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 00:36 /home/ubuntu/cephtest 2024-05-27T00:36:41.601 INFO:teuthology.orchestra.run.smithi204.stdout: 524291 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 00:36 /home/ubuntu/cephtest 2024-05-27T00:36:41.603 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-27T00:36:41.643 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-27T00:36:41.759 INFO:teuthology.run:Summary data: client.0-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 1981.919694185257 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi098 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d7cb32d1d130135a98121a564d709025c2154b6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ac664771b3b6445687dcbc93a64d8036 status: fail success: false 2024-05-27T00:36:41.760 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T00:36:41.854 INFO:teuthology.run:FAIL